* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] <20210222230519.73f3e239@sf> @ 2021-02-22 23:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-22 23:34 UTC (permalink / raw) To: Sergei Trofimovich, linux-ia64, linux-kernel On 2/22/21 4:05 PM, Sergei Trofimovich wrote: > Hia Jens! > > Tried 5.11 on rx3600 box and noticed it has > a problem handling init (5.10 booted fine): > > INIT: version 2.98 booting > > OpenRC 0.42.1 is starting up Gentoo Linux (ia64) > > mkdir `/run/openrc': Read-only file system > mkdir `/run/openrc/starting': No such file or directory > mkdir `/run/openrc/started': No such file or directory > mkdir `/run/openrc/stopping': No such file or directory > mkdir `/run/openrc/inactive': No such file or directory > mkdir `/run/openrc/wasinactive': No such file or directory > mkdir `/run/openrc/failed': No such file or directory > mkdir `/run/openrc/hotplugged': No such file or directory > mkdir `/run/openrc/daemons': No such file or directory > mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > I suspect we build bad signal stack frame for userspace. > > With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: > > [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 > [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 > [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 > [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 > [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 > [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > Bisect points at: > > commit b269c229b0e89aedb7943c06673b56b6052cf5e5 > Author: Jens Axboe <axboe@kernel.dk> > Date: Fri Oct 9 14:49:43 2020 -0600 > > ia64: add support for TIF_NOTIFY_SIGNAL > > Wire up TIF_NOTIFY_SIGNAL handling for ia64. > > Cc: linux-ia64@vger.kernel.org > [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] > Signed-off-by: Jens Axboe <axboe@kernel.dk> > > diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h > index 64a1011f6812..51d20cb37706 100644 > --- a/arch/ia64/include/asm/thread_info.h > +++ b/arch/ia64/include/asm/thread_info.h > @@ -103,6 +103,7 @@ struct thread_info { > #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ > #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ > #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ > +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ > #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ > #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ > #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ > @@ -115,6 +116,7 @@ struct thread_info { > #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) > #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) > #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) > +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) > #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) > @@ -124,7 +126,7 @@ struct thread_info { > > /* "work to do on user-return" bits */ > #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ > - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) > + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) > /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ > #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) > > diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c > index 6b61a703bcf5..8d4e1cab9190 100644 > --- a/arch/ia64/kernel/process.c > +++ b/arch/ia64/kernel/process.c > @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) > } > > /* deal with pending signal delivery */ > - if (test_thread_flag(TIF_SIGPENDING)) { > + if (test_thread_flag(TIF_SIGPENDING) || > + test_thread_flag(TIF_NOTIFY_SIGNAL)) { > local_irq_enable(); /* force interrupt enable */ > ia64_do_signal(scr, in_syscall); > > which looks benign, but it enables a bit of conditional > TIF_NOTIFY_SIGNAL handling I don't understand. > > Can you help me get what is the interaction between > TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for > simple processes without io_uring use case? > > I wonder if it's ia64_do_signal()' generates a signal > delivery when it should not. Can you test: https://marc.info/?l=linux-ia64&m=161187407609443&w=1 with the addition mentioned here: https://marc.info/?l=linux-ia64&m=161187470709706&w=1 if needed? -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-22 23:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-22 23:34 UTC (permalink / raw) To: Sergei Trofimovich, linux-ia64, linux-kernel On 2/22/21 4:05 PM, Sergei Trofimovich wrote: > Hia Jens! > > Tried 5.11 on rx3600 box and noticed it has > a problem handling init (5.10 booted fine): > > INIT: version 2.98 booting > > OpenRC 0.42.1 is starting up Gentoo Linux (ia64) > > mkdir `/run/openrc': Read-only file system > mkdir `/run/openrc/starting': No such file or directory > mkdir `/run/openrc/started': No such file or directory > mkdir `/run/openrc/stopping': No such file or directory > mkdir `/run/openrc/inactive': No such file or directory > mkdir `/run/openrc/wasinactive': No such file or directory > mkdir `/run/openrc/failed': No such file or directory > mkdir `/run/openrc/hotplugged': No such file or directory > mkdir `/run/openrc/daemons': No such file or directory > mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > I suspect we build bad signal stack frame for userspace. > > With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: > > [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 > [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 > [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 > [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 > [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 > [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > Bisect points at: > > commit b269c229b0e89aedb7943c06673b56b6052cf5e5 > Author: Jens Axboe <axboe@kernel.dk> > Date: Fri Oct 9 14:49:43 2020 -0600 > > ia64: add support for TIF_NOTIFY_SIGNAL > > Wire up TIF_NOTIFY_SIGNAL handling for ia64. > > Cc: linux-ia64@vger.kernel.org > [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] > Signed-off-by: Jens Axboe <axboe@kernel.dk> > > diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h > index 64a1011f6812..51d20cb37706 100644 > --- a/arch/ia64/include/asm/thread_info.h > +++ b/arch/ia64/include/asm/thread_info.h > @@ -103,6 +103,7 @@ struct thread_info { > #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ > #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ > #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ > +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ > #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ > #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ > #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ > @@ -115,6 +116,7 @@ struct thread_info { > #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) > #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) > #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) > +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) > #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) > @@ -124,7 +126,7 @@ struct thread_info { > > /* "work to do on user-return" bits */ > #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ > - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) > + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) > /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ > #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) > > diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c > index 6b61a703bcf5..8d4e1cab9190 100644 > --- a/arch/ia64/kernel/process.c > +++ b/arch/ia64/kernel/process.c > @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) > } > > /* deal with pending signal delivery */ > - if (test_thread_flag(TIF_SIGPENDING)) { > + if (test_thread_flag(TIF_SIGPENDING) || > + test_thread_flag(TIF_NOTIFY_SIGNAL)) { > local_irq_enable(); /* force interrupt enable */ > ia64_do_signal(scr, in_syscall); > > which looks benign, but it enables a bit of conditional > TIF_NOTIFY_SIGNAL handling I don't understand. > > Can you help me get what is the interaction between > TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for > simple processes without io_uring use case? > > I wonder if it's ia64_do_signal()' generates a signal > delivery when it should not. Can you test: https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 with the addition mentioned here: https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 if needed? -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-02-22 23:34 ` Jens Axboe @ 2021-02-22 23:55 ` John Paul Adrian Glaubitz -1 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-02-22 23:55 UTC (permalink / raw) To: Jens Axboe, Sergei Trofimovich, linux-ia64, linux-kernel Hi Sergei! On 2/23/21 12:34 AM, Jens Axboe wrote: > On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >> Hia Jens! >> >> Tried 5.11 on rx3600 box and noticed it has >> a problem handling init (5.10 booted fine): >> >> INIT: version 2.98 booting >> >> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >> >> mkdir `/run/openrc': Read-only file system >> mkdir `/run/openrc/starting': No such file or directory >> mkdir `/run/openrc/started': No such file or directory >> mkdir `/run/openrc/stopping': No such file or directory >> mkdir `/run/openrc/inactive': No such file or directory >> mkdir `/run/openrc/wasinactive': No such file or directory >> mkdir `/run/openrc/failed': No such file or directory >> mkdir `/run/openrc/hotplugged': No such file or directory >> mkdir `/run/openrc/daemons': No such file or directory >> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >> >> I suspect we build bad signal stack frame for userspace. For me, Debian also stopped booting on ia64 with systemd and I bisected it to this change. Glad to see I'm not the only one. The changes Jens suggested back then unfortunately didn't help :(. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-22 23:55 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-02-22 23:55 UTC (permalink / raw) To: Jens Axboe, Sergei Trofimovich, linux-ia64, linux-kernel Hi Sergei! On 2/23/21 12:34 AM, Jens Axboe wrote: > On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >> Hia Jens! >> >> Tried 5.11 on rx3600 box and noticed it has >> a problem handling init (5.10 booted fine): >> >> INIT: version 2.98 booting >> >> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >> >> mkdir `/run/openrc': Read-only file system >> mkdir `/run/openrc/starting': No such file or directory >> mkdir `/run/openrc/started': No such file or directory >> mkdir `/run/openrc/stopping': No such file or directory >> mkdir `/run/openrc/inactive': No such file or directory >> mkdir `/run/openrc/wasinactive': No such file or directory >> mkdir `/run/openrc/failed': No such file or directory >> mkdir `/run/openrc/hotplugged': No such file or directory >> mkdir `/run/openrc/daemons': No such file or directory >> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >> >> I suspect we build bad signal stack frame for userspace. For me, Debian also stopped booting on ia64 with systemd and I bisected it to this change. Glad to see I'm not the only one. The changes Jens suggested back then unfortunately didn't help :(. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
[parent not found: <20210223083507.43b5a6dd@sf>]
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210223083507.43b5a6dd@sf> @ 2021-02-23 9:13 ` John Paul Adrian Glaubitz 2021-02-23 12:36 ` John Paul Adrian Glaubitz 0 siblings, 1 reply; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-02-23 9:13 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel On 2/23/21 9:35 AM, Sergei Trofimovich wrote:> On Tue, 23 Feb 2021 00:55:50 +0100 > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > >> For me, Debian also stopped booting on ia64 with systemd and I bisected it to this >> change. Glad to see I'm not the only one. >> >> The changes Jens suggested back then unfortunately didn't help :(. > > I wonder if systemd is less tolerant to kernel that throws signal 0 at it. > > Looks like https://marc.info/?l=linux-ia64&m=161404320617004&w=2 > fixed all the issues for me. Just gave it a try and it still doesn't work. And something broke CPU enumeration: Booting a command list Loading Linux 5.11.0+ ... [ 0.000000] Linux version 5.11.0+ (glaubitz@epyc) (ia64-linux-gcc (GCC) 10.1.0, GNU ld (GNU Binutils) 2.34) #191 SMP Tue Feb 23 09:54:44 CET 2021 [ 0.000000] efi: EFI v2.00 by HP [ 0.000000] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fde4000 ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fde2000 [ 0.000000] PCDP: v3 at 0x3fde2000 [ 0.000000] earlycon: uart8250 at MMIO 0x0000000088033000 (options '115200n8') [ 0.000000] printk: bootconsole [uart8250] enabled [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x000000003FDE4000 000028 (v02 HP ) [ 0.000000] ACPI: XSDT 0x000000003FDE402C 0000A4 (v01 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: FACP 0x000000003FDF6A08 0000F4 (v03 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: DSDT 0x000000003FDE41C8 00E566 (v01 HP rx2660 00000007 INTL 20050309) [ 0.000000] ACPI: FACS 0x000000003FDF6B00 000040 [ 0.000000] ACPI: SPCR 0x000000003FDF6B40 000050 (v01 HP 00000000 HP 00000000) [ 0.000000] ACPI: DBGP 0x000000003FDF6B90 000034 (v01 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: APIC 0x000000003FDF6FB0 0000C8 (v01 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: SPMI 0x000000003FDF6BC8 000050 (v04 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: CPEP 0x000000003FDF6E80 000034 (v01 HP rx2660 00000000 HP 00000000) [ 0.000000] ACPI: SSDT 0x000000003FDF2738 0004B3 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF2BF8 000456 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF3058 000EB8 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF3F18 000EB8 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF4DD8 000866 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF5648 000EB8 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF6508 000138 (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF6648 00013C (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF6788 00013C (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: SSDT 0x000000003FDF68C8 00013C (v01 HP rx2660 00000006 INTL 20050309) [ 0.000000] ACPI: Local APIC address (____ptrval____) [ 0.000000] 4 CPUs available, 4 CPUs total [ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] SAL 3.20: HP version 4.4 [ 0.000000] SAL Platform features: [ 0.000000] None [ 0.000000] SAL: AP wakeup using external interrupt vector 0xff [ 0.000000] MCA related initialization done [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000001007fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000001000000-0x000000003e67ffff] [ 0.000000] node 0: [mem 0x000000003eaec000-0x000000003ee77fff] [ 0.000000] node 0: [mem 0x000000003fc00000-0x000000003fd77fff] [ 0.000000] node 0: [mem 0x000000003fddc000-0x000000003fddffff] [ 0.000000] node 0: [mem 0x0000010040000000-0x000001007f1fbfff] [ 0.000000] node 0: [mem 0x000001007f200000-0x000001007fffffff] [ 0.000000] Zeroed struct page in unavailable ranges: 2335 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000001000000-0x000001007fffffff] [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 128234 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.11.0+ root=/dev/sda2 ro module_blacklist=radeon nomodeset hardened_usercopy=off [ 0.000000] You have booted with nomodeset. This means your GPU drivers are DISABLED [ 0.000000] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly [ 0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it [ 0.000000] Dentry cache hash table entries: 262144 (order: 7, 2097152 bytes, linear) [ 0.000000] Inode-cache hash table entries: 131072 (order: 6, 1048576 bytes, linear) [ 0.000000] Sorting __ex_table... [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] software IO TLB: mapped [mem 0x000000003a680000-0x000000003e680000] (64MB) [ 0.000000] Memory: 1949200K/2059792K available (17148K kernel code, 1420K rwdata, 1992K rodata, 992K init, 702K bss, 110592K reserved, 0K cma-reserved) [ 0.000000] random: get_random_u64 called from __kmem_cache_create+0x60/0x1060 with crng_init=0 [ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=4, Nodes=256 [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 1024 [ 0.000000] ACPI: Local APIC address (____ptrval____) [ 0.000000] GSI 25 (level, low) -> CPU 0 (0x0000) vector 49 [ 0.000000] +/-150ppm [ 0.000000] clocksource: itc: mask: 0xffffffffffffffff max_cycles: 0x5c0f696de1, max_idle_ns: 440795208283 ns [ 0.004000] Console: colour VGA+ 80x25 [ 0.004000] ACPI: Core revision 20210105 [ 0.004000] Calibrating delay loop... 3182.59 BogoMIPS (lpj=6365184) [ 0.020000] pid_max: default: 32768 minimum: 301 [ 0.020000] LSM: Security Framework initializing [ 0.020000] Yama: becoming mindful. [ 0.020000] AppArmor: AppArmor initialized [ 0.020000] TOMOYO Linux initialized [ 0.020000] Mount-cache hash table entries: 4096 (order: 1, 32768 bytes, linear) [ 0.024000] Mountpoint-cache hash table entries: 4096 (order: 1, 32768 bytes, linear) [ 0.024000] Boot processor id 0x0/0x0 [ 0.024000] rcu: Hierarchical SRCU implementation. [ 0.024000] smp: Bringing up secondary CPUs ... [ 0.004000] Fixed BSP b0 value from CPU 1 [ 0.004000] CPU 1: synchronized ITC with CPU 0 (last diff 0 cycles, maxerr 40 cycles) [ 0.004000] +/-150ppm [ 0.004000] CPU 2: synchronized ITC with CPU 0 (last diff -5 cycles, maxerr 167 cycles) [ 0.004000] +/-150ppm [ 0.004000] CPU 3: synchronized ITC with CPU 0 (last diff -5 cycles, maxerr 167 cycles) [ 0.004000] +/-150ppm [ 0.032000] smp: Brought up 1 node, 4 CPUs [ 0.032000] Total of 4 processors activated (12730.36 BogoMIPS). [ 0.036000] ERROR: Invalid distance value range [ 0.036000] [ 0.036000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.048000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.064000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.076000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.088000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.104000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.116000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.128000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.140000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.156000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.168000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.184000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.196000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.208000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.220000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.232000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.248000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.260000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.272000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.288000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.300000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.312000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.324000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.340000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.352000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.364000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.376000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.392000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.404000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.420000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.432000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.444000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.456000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.468000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.480000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.496000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.508000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.524000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.536000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.536000] random: fast init done [ 0.548000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.560000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.576000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.588000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.600000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.616000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.628000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.640000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.656000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.668000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.680000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.692000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.708000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.720000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.732000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.748000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.764000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.776000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.792000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.808000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.820000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.836000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.852000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.868000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.880000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.892000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.908000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.920000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.932000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.948000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.960000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.976000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.988000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.000000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.012000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.028000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.040000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.052000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.068000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.080000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.092000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.108000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.120000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.136000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.148000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.160000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.172000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.188000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.200000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.216000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.228000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.240000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.252000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.268000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.280000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.292000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.308000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.320000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.332000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.344000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.356000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.372000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.384000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.396000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.412000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.424000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.436000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.448000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.460000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.476000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.488000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.504000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.516000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.528000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.540000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.552000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.568000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.580000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.596000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.608000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.620000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.632000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.644000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.660000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.672000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.684000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.700000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.712000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.724000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.740000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.752000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.764000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.780000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.792000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.804000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.816000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.832000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.844000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.856000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.872000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.884000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.896000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.908000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.924000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.936000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.952000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.964000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.976000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1.988000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.004000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.016000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.028000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.044000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.056000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.068000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.080000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.096000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.108000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.124000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.136000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.148000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.164000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.176000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.192000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.204000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.216000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.228000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.244000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.256000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.268000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.284000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.296000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.308000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.324000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.336000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.348000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.364000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.376000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.388000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.400000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.416000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.428000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.440000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.456000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.468000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.480000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.492000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.504000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.520000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.532000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.544000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.560000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.572000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.584000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.596000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.612000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.624000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.640000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.656000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.668000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.680000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.692000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.704000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.720000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.732000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.748000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.760000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.772000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.784000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.800000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.812000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.828000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.840000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.852000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.868000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.880000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.892000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.908000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.920000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.936000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.948000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.960000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.972000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2.984000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.000000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.012000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.028000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.040000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.052000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.064000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.080000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.092000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.108000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.120000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.132000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.144000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.160000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.172000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.188000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.200000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.212000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.224000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.240000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.252000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.268000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.280000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.292000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.304000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.320000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.332000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.344000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.360000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.372000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.384000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.400000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.412000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.424000] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 3.440000] [ 3.440000] devtmpfs: initialized [ 3.440000] SMBIOS 2.4 present. [ 3.440000] DMI: hp server rx2660 , BIOS 04.04 07/15/2008 [ 3.444000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 3.444000] futex hash table entries: 1024 (order: 3, 131072 bytes, linear) [ 3.448000] NET: Registered protocol family 16 [ 3.448000] audit: initializing netlink subsys (disabled) [ 3.448000] audit: type=2000 audit(1614071467.448:1): state=initialized audit_enabled=0 res=1 [ 3.448000] thermal_sys: Registered thermal governor 'fair_share' [ 3.448000] thermal_sys: Registered thermal governor 'step_wise' [ 3.452000] ACPI: bus type PCI registered [ 3.456000] HugeTLB registered 256 MiB page size, pre-allocated 0 pages [ 4.912000] ACPI: Added _OSI(Module Device) [ 4.912000] ACPI: Added _OSI(Processor Device) [ 4.912000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 4.912000] ACPI: Added _OSI(Processor Aggregator Device) [ 4.912000] ACPI: Added _OSI(Linux-Dell-Video) [ 4.916000] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) [ 4.916000] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics) [ 4.920000] ACPI: 11 ACPI AML tables successfully acquired and loaded [ 4.980000] ACPI: Interpreter enabled [ 4.980000] ACPI: (supports S0 S5) [ 4.980000] ACPI: Using IOSAPIC for interrupt routing [ 5.008000] ACPI: Enabled 2 GPEs in block 10 to 1F [ 5.340000] ACPI: PCI Root Bridge [L000] (domain 0000 [bus 00]) [ 5.340000] acpi HWP0002:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.376000] PCI host bridge to bus 0000:00 [ 5.376000] pci_bus 0000:00: Unknown NUMA node; performance will be reduced [ 5.376000] pci_bus 0000:00: root bus resource [io 0x1000000-0x100ffff window] (bus address [0x0000-0xffff]) [ 5.376000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000fffff window] [ 5.376000] pci_bus 0000:00: root bus resource [mem 0x80000000-0x8fffffff window] [ 5.380000] pci_bus 0000:00: root bus resource [mem 0x80004000000-0x800ffffffff window] [ 5.380000] pci_bus 0000:00: root bus resource [bus 00] [ 5.384000] pci 0000:00:01.0: [103c:1303] type 00 class 0xff0000 [ 5.388000] pci 0000:00:01.1: [103c:1302] type 00 class 0x078000 [ 5.392000] pci 0000:00:01.1: reg 0x14: [mem 0x88034000-0x88034fff 64bit] [ 5.392000] pci 0000:00:01.1: reg 0x1c: [mem 0x80080000000-0x8008001ffff 64bit pref] [ 5.392000] pci 0000:00:01.2: [103c:1048] type 00 class 0x070002 [ 5.396000] pci 0000:00:01.2: reg 0x14: [mem 0x88033000-0x88033fff 64bit] [ 5.400000] pci 0000:00:02.0: [1033:0035] type 00 class 0x0c0310 [ 5.400000] pci 0000:00:02.0: reg 0x10: [mem 0x88032000-0x88032fff] [ 5.400000] pci 0000:00:02.0: supports D1 D2 [ 5.400000] pci 0000:00:02.0: PME# supported from D0 D1 D2 D3hot [ 5.404000] pci 0000:00:02.1: [1033:0035] type 00 class 0x0c0310 [ 5.404000] pci 0000:00:02.1: reg 0x10: [mem 0x88031000-0x88031fff] [ 5.408000] pci 0000:00:02.1: supports D1 D2 [ 5.408000] pci 0000:00:02.1: PME# supported from D0 D1 D2 D3hot [ 5.408000] pci 0000:00:02.2: [1033:00e0] type 00 class 0x0c0320 [ 5.408000] pci 0000:00:02.2: reg 0x10: [mem 0x88030000-0x880300ff] [ 5.412000] pci 0000:00:02.2: supports D1 D2 [ 5.412000] pci 0000:00:02.2: PME# supported from D0 D1 D2 D3hot [ 5.412000] pci 0000:00:03.0: [1002:515e] type 00 class 0x030000 [ 5.412000] pci 0000:00:03.0: reg 0x10: [mem 0x80000000-0x87ffffff pref] [ 5.416000] pci 0000:00:03.0: reg 0x14: [io 0x1001000-0x10010ff] [ 5.416000] pci 0000:00:03.0: reg 0x18: [mem 0x88020000-0x8802ffff] [ 5.416000] pci 0000:00:03.0: reg 0x30: [mem 0x88000000-0x8801ffff pref] [ 5.420000] pci 0000:00:03.0: supports D1 D2 [ 5.448000] ACPI: PCI Root Bridge [L002] (domain 0000 [bus 01]) [ 5.448000] acpi HWP0002:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.476000] PCI host bridge to bus 0000:01 [ 5.476000] pci_bus 0000:01: Unknown NUMA node; performance will be reduced [ 5.476000] pci_bus 0000:01: root bus resource [io 0x2000000-0x200ffff window] (bus address [0x0000-0xffff]) [ 5.476000] pci_bus 0000:01: root bus resource [mem 0xa0000000-0xafffffff window] [ 5.476000] pci_bus 0000:01: root bus resource [mem 0x80204000000-0x802ffffffff window] [ 5.480000] pci_bus 0000:01: root bus resource [bus 01] [ 5.484000] pci 0000:01:01.0: [1000:0054] type 00 class 0x010000 [ 5.484000] pci 0000:01:01.0: reg 0x10: [io 0x2001000-0x20010ff] [ 5.484000] pci 0000:01:01.0: reg 0x14: [mem 0xa0470000-0xa0473fff 64bit] [ 5.488000] pci 0000:01:01.0: reg 0x1c: [mem 0xa0460000-0xa046ffff 64bit] [ 5.488000] pci 0000:01:01.0: reg 0x30: [mem 0xa0000000-0xa03fffff pref] [ 5.488000] pci 0000:01:01.0: supports D1 D2 [ 5.492000] pci 0000:01:02.0: [14e4:1648] type 00 class 0x020000 [ 5.492000] pci 0000:01:02.0: reg 0x10: [mem 0xa0450000-0xa045ffff 64bit] [ 5.496000] pci 0000:01:02.0: reg 0x30: [mem 0xa0420000-0xa043ffff pref] [ 5.496000] pci 0000:01:02.0: PME# supported from D3hot D3cold [ 5.496000] pci 0000:01:02.1: [14e4:1648] type 00 class 0x020000 [ 5.500000] pci 0000:01:02.1: reg 0x10: [mem 0xa0440000-0xa044ffff 64bit] [ 5.500000] pci 0000:01:02.1: reg 0x30: [mem 0xa0400000-0xa041ffff pref] [ 5.504000] pci 0000:01:02.1: PME# supported from D3hot D3cold [ 5.532000] ACPI: PCI Root Bridge [L003] (domain 0000 [bus 02-03]) [ 5.532000] acpi HPQ0002:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.536000] acpi HPQ0002:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR] [ 5.564000] PCI host bridge to bus 0000:02 [ 5.564000] pci_bus 0000:02: Unknown NUMA node; performance will be reduced [ 5.564000] pci_bus 0000:02: root bus resource [io 0x3000000-0x300ffff window] (bus address [0x0000-0xffff]) [ 5.564000] pci_bus 0000:02: root bus resource [mem 0xb0000000-0xbfffffff window] [ 5.564000] pci_bus 0000:02: root bus resource [mem 0x80304000000-0x803ffffffff window] [ 5.568000] pci_bus 0000:02: root bus resource [bus 02-03] [ 5.568000] pci 0000:02:00.0: [103c:403b] type 01 class 0x060400 [ 5.576000] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 5.600000] pci 0000:02:00.0: PCI bridge to [bus 03] [ 5.600000] pci 0000:02:00.0: bridge window [io 0x3001000-0x300ffff] [ 5.600000] pci 0000:02:00.0: bridge window [mem 0xb0000000-0xbfffffff] [ 5.604000] pci 0000:02:00.0: bridge window [mem 0x80380000000-0x803ffffffff 64bit pref] [ 5.644000] ACPI: PCI Root Bridge [L004] (domain 0000 [bus 04-05]) [ 5.644000] acpi HPQ0002:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.652000] acpi HPQ0002:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR] [ 5.676000] PCI host bridge to bus 0000:04 [ 5.676000] pci_bus 0000:04: Unknown NUMA node; performance will be reduced [ 5.676000] pci_bus 0000:04: root bus resource [io 0x4000000-0x400ffff window] (bus address [0x0000-0xffff]) [ 5.676000] pci_bus 0000:04: root bus resource [mem 0xc0000000-0xdfffffff window] [ 5.676000] pci_bus 0000:04: root bus resource [mem 0x80404000000-0x804ffffffff window] [ 5.680000] pci_bus 0000:04: root bus resource [bus 04-05] [ 5.684000] pci 0000:04:00.0: [103c:403b] type 01 class 0x060400 [ 5.688000] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold [ 5.720000] pci 0000:05:00.0: [103c:3230] type 00 class 0x010400 [ 5.728000] pci 0000:05:00.0: reg 0x10: [mem 0xc0000000-0xc00fffff 64bit] [ 5.732000] pci 0000:05:00.0: reg 0x18: [io 0x4001000-0x40010ff] [ 5.740000] pci 0000:05:00.0: reg 0x1c: [mem 0xc0140000-0xc0140fff 64bit] [ 5.744000] pci 0000:05:00.0: reg 0x30: [mem 0xc0100000-0xc013ffff pref] [ 5.768000] pci 0000:05:00.0: supports D1 [ 5.792000] pci 0000:04:00.0: PCI bridge to [bus 05] [ 5.792000] pci 0000:04:00.0: bridge window [io 0x4001000-0x400ffff] [ 5.792000] pci 0000:04:00.0: bridge window [mem 0xc0000000-0xdfffffff] [ 5.792000] pci 0000:04:00.0: bridge window [mem 0x80480000000-0x804ffffffff 64bit pref] [ 5.836000] ACPI: PCI Root Bridge [L006] (domain 0000 [bus 06]) [ 5.836000] acpi HWP0002:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.880000] PCI host bridge to bus 0000:06 [ 5.880000] pci_bus 0000:06: Unknown NUMA node; performance will be reduced [ 5.880000] pci_bus 0000:06: root bus resource [io 0x5000000-0x500ffff window] (bus address [0x0000-0xffff]) [ 5.880000] pci_bus 0000:06: root bus resource [mem 0xe0000000-0xefffffff window] [ 5.880000] pci_bus 0000:06: root bus resource [mem 0x80604000000-0x806ffffffff window] [ 5.884000] pci_bus 0000:06: root bus resource [bus 06] [ 5.928000] ACPI: PCI Root Bridge [L007] (domain 0000 [bus 07-08]) [ 5.928000] acpi HPQ0002:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 5.936000] acpi HPQ0002:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR] [ 5.960000] PCI host bridge to bus 0000:07 [ 5.960000] pci_bus 0000:07: Unknown NUMA node; performance will be reduced [ 5.960000] pci_bus 0000:07: root bus resource [io 0x6000000-0x600ffff window] (bus address [0x0000-0xffff]) [ 5.960000] pci_bus 0000:07: root bus resource [mem 0xf0000000-0xfdffffff window] [ 5.964000] pci_bus 0000:07: root bus resource [mem 0x80704000000-0x807ffffffff window] [ 5.964000] pci_bus 0000:07: root bus resource [bus 07-08] [ 5.968000] pci 0000:07:00.0: [103c:403b] type 01 class 0x060400 [ 5.976000] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold [ 6.004000] pci 0000:07:00.0: PCI bridge to [bus 08] [ 6.004000] pci 0000:07:00.0: bridge window [io 0x6001000-0x600ffff] [ 6.004000] pci 0000:07:00.0: bridge window [mem 0xf0000000-0xfdffffff] [ 6.008000] pci 0000:07:00.0: bridge window [mem 0x80780000000-0x807ffffffff 64bit pref] [ 6.024000] pci 0000:00:03.0: vgaarb: setting as boot VGA device [ 6.024000] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none [ 6.024000] pci 0000:00:03.0: vgaarb: bridge control possible [ 6.024000] vgaarb: loaded [ 6.028000] pps_core: LinuxPPS API ver. 1 registered [ 6.028000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 6.028000] PTP clock support registered [ 6.028000] Registered efivars operations [ 6.044000] IOC: zx2 2.0 HPA 0xfed01000 IOVA space 1024Mb at 0x40000000 [ 6.124000] clocksource: Switched to clocksource itc [ 6.124000] VFS: Disk quotas dquot_6.6.0 [ 6.124000] VFS: Dquot-cache hash table entries: 2048 (order 0, 16384 bytes) [ 6.124000] AppArmor: AppArmor Filesystem Enabled [ 6.124000] pnp: PnP ACPI init [ 6.126997] GSI 24 (level, low) -> CPU 1 (0x0100) vector 50 [ 6.202997] pnp: PnP ACPI: found 1 devices [ 6.207936] NET: Registered protocol family 2 [ 6.207936] tcp_listen_portaddr_hash hash table entries: 1024 (order: 0, 16384 bytes, linear) [ 6.207936] TCP established hash table entries: 16384 (order: 3, 131072 bytes, linear) [ 6.207936] TCP bind hash table entries: 16384 (order: 4, 262144 bytes, linear) [ 6.213158] TCP: Hash tables configured (established 16384 bind 16384) [ 6.213158] UDP hash table entries: 1024 (order: 1, 32768 bytes, linear) [ 6.215024] UDP-Lite hash table entries: 1024 (order: 1, 32768 bytes, linear) [ 6.215024] NET: Registered protocol family 1 [ 6.217026] NET: Registered protocol family 44 [ 6.221158] GSI 17 (level, low) -> CPU 2 (0x0200) vector 51 [ 6.222998] GSI 17 (level, low) -> CPU 2 (0x0200) vector 51 unregistered [ 6.226997] GSI 18 (level, low) -> CPU 3 (0x0300) vector 51 [ 6.230998] GSI 18 (level, low) -> CPU 3 (0x0300) vector 51 unregistered [ 6.234997] GSI 19 (level, low) -> CPU 0 (0x0000) vector 51 [ 6.238998] GSI 19 (level, low) -> CPU 0 (0x0000) vector 51 unregistered [ 6.238998] pci 0000:00:03.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 6.238998] PCI: CLS 128 bytes, default 128 [ 6.241160] Initialise system trusted keyrings [ 6.241160] Key type blacklist registered [ 6.245029] workingset: timestamp_bits=38 max_order=17 bucket_order=0 [ 6.245029] zbud: loaded [ 6.246998] Key type asymmetric registered [ 6.246998] Asymmetric key parser 'x509' registered [ 6.246998] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 6.249158] io scheduler mq-deadline registered [ 6.250998] GSI 38 (level, low) -> CPU 1 (0x0100) vector 51 [ 6.255032] pcieport 0000:02:00.0: PME: Signaling with IRQ 55 [ 6.255936] pcieport 0000:02:00.0: AER: enabled with IRQ 55 [ 6.262998] GSI 45 (level, low) -> CPU 2 (0x0200) vector 53 [ 6.267081] pcieport 0000:04:00.0: PME: Signaling with IRQ 57 [ 6.267938] pcieport 0000:04:00.0: AER: enabled with IRQ 57 [ 6.274997] GSI 63 (level, low) -> CPU 3 (0x0300) vector 55 [ 6.279052] pcieport 0000:07:00.0: PME: Signaling with IRQ 59 [ 6.279938] pcieport 0000:07:00.0: AER: enabled with IRQ 59 [ 6.287010] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.287010] ACPI: button: Power Button [PWRF] [ 6.287010] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1 [ 6.287010] ACPI: button: Sleep Button [SLPF] [ 6.287939] thermal LNXTHERM:00: registered as thermal_zone0 [ 6.287939] ACPI: thermal: Thermal Zone [THM0] (27 C) [ 6.295936] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.295936] 00:00: ttyS0 at MMIO 0xff5e2000 (irq = 50, base_baud = 115200) is a 16550A [ 6.342998] GSI 16 (level, low) -> CPU 0 (0x0000) vector 57 6.343936] 0000:00:01.2: ttyS1 at MMIO 0x88033000 (irq = 60, base_baud = 115200) is a 16550A [ 6.343936] printk: console [ttyS1] enabled [ 6.343936] printk: console [ttyS1] enabled [ 6.343936] printk: bootconsole [uart8250] disabled [ 6.343936] printk: bootconsole [uart8250] disabled [ 6.351937] Linux agpgart interface v0.103 [ 6.403936] mousedev: PS/2 mouse device common for all mice [ 6.403936] rtc-efi rtc-efi.0: registered as rtc0 [ 6.403936] rtc-efi rtc-efi.0: setting system clock to 2021-02-23T09:11:10 UTC (1614071470) [ 6.405158] ledtrig-cpu: registered to indicate activity on CPUs [ 6.405158] EFI Variables Facility v0.08 2004-May-17 [ 6.408093] NET: Registered protocol family 10 [ 6.413162] Segment Routing with IPv6 [ 6.413162] mip6: Mobile IPv6 [ 6.413162] NET: Registered protocol family 17 [ 6.413162] mpls_gso: MPLS GSO support [ 6.413162] registered taskstats version 1 [ 6.413162] Loading compiled-in X.509 certificates [ 6.413162] zswap: loaded using pool lzo/zbud [ 6.413162] AppArmor: AppArmor sha1 policy hashing enabled [ 6.418998] List of all partitions: [ 6.418998] No filesystem could mount root, tried: [ 6.418998] [ 6.418998] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) [ 6.418998] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-02-23 9:13 ` John Paul Adrian Glaubitz @ 2021-02-23 12:36 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-02-23 12:36 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel On 2/23/21 10:13 AM, John Paul Adrian Glaubitz wrote: > On 2/23/21 9:35 AM, Sergei Trofimovich wrote:> On Tue, 23 Feb 2021 00:55:50 +0100 >> John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: >> >>> For me, Debian also stopped booting on ia64 with systemd and I bisected it to this >>> change. Glad to see I'm not the only one. >>> >>> The changes Jens suggested back then unfortunately didn't help :(. >> >> I wonder if systemd is less tolerant to kernel that throws signal 0 at it. >> >> Looks like https://marc.info/?l=linux-ia64&m=161404320617004&w=2 >> fixed all the issues for me. > > Just gave it a try and it still doesn't work. Maybe your other two patches to fix the strace issues are required as well? Or do you happen to have more patches in the Gentoo kernel? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-23 12:36 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-02-23 12:36 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel On 2/23/21 10:13 AM, John Paul Adrian Glaubitz wrote: > On 2/23/21 9:35 AM, Sergei Trofimovich wrote:> On Tue, 23 Feb 2021 00:55:50 +0100 >> John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: >> >>> For me, Debian also stopped booting on ia64 with systemd and I bisected it to this >>> change. Glad to see I'm not the only one. >>> >>> The changes Jens suggested back then unfortunately didn't help :(. >> >> I wonder if systemd is less tolerant to kernel that throws signal 0 at it. >> >> Looks like https://marc.info/?l=linux-ia64&m\x161404320617004&w=2 >> fixed all the issues for me. > > Just gave it a try and it still doesn't work. Maybe your other two patches to fix the strace issues are required as well? Or do you happen to have more patches in the Gentoo kernel? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
[parent not found: <20210223192743.0198d4a9@sf>]
[parent not found: <20210302222630.5056f243@sf>]
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210302222630.5056f243@sf> @ 2021-03-02 22:31 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-02 22:31 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi Sergei! On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > Gave v5.12-rc1 a try today and got a similar boot failure around > hpsa queue initialization, but my failure is later: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Maybe I get different error because I flipped on most debugging > kernel options :) > > Looks like 'ERROR: Invalid distance value range' while being > very scary are harmless. It's just a new spammy way for kernel > to report lack of NUMA config on the machine (no SRAT and SLIT > ACPI tables). > > At least I get hpsa detected on PCI bus. But I guess it's discovered > configuration is very wrong as I get unaligned accesses: > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > Bisecting now. Sounds good. I guess we should get Jens' fix for the signal regression merged as well as your two fixes for strace. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-02 22:31 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-02 22:31 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi Sergei! On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > Gave v5.12-rc1 a try today and got a similar boot failure around > hpsa queue initialization, but my failure is later: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Maybe I get different error because I flipped on most debugging > kernel options :) > > Looks like 'ERROR: Invalid distance value range' while being > very scary are harmless. It's just a new spammy way for kernel > to report lack of NUMA config on the machine (no SRAT and SLIT > ACPI tables). > > At least I get hpsa detected on PCI bus. But I guess it's discovered > configuration is very wrong as I get unaligned accesses: > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > Bisecting now. Sounds good. I guess we should get Jens' fix for the signal regression merged as well as your two fixes for strace. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-02 22:31 ` John Paul Adrian Glaubitz @ 2021-03-03 0:22 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-03 0:22 UTC (permalink / raw) To: John Paul Adrian Glaubitz, Don Brace Cc: linux-ia64, linux-kernel, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen On Tue, 2 Mar 2021 23:31:32 +0100 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > Hi Sergei! > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > Gave v5.12-rc1 a try today and got a similar boot failure around > > hpsa queue initialization, but my failure is later: > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > Maybe I get different error because I flipped on most debugging > > kernel options :) > > > > Looks like 'ERROR: Invalid distance value range' while being > > very scary are harmless. It's just a new spammy way for kernel > > to report lack of NUMA config on the machine (no SRAT and SLIT > > ACPI tables). > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > configuration is very wrong as I get unaligned accesses: > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > Bisecting now. > > Sounds good. I guess we should get Jens' fix for the signal regression > merged as well as your two fixes for strace. "bisected" (cheated halfway through) and verified that reverting f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. CCing authors who might be able to help us here. commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 Author: Don Brace <don.brace@microchip.com> Date: Mon Feb 15 16:26:57 2021 -0600 scsi: hpsa: Correct dev cmds outstanding for retried cmds Prevent incrementing device->commands_outstanding for ioaccel command retries that are driver initiated. If the command goes through the retry path, the device->commands_outstanding counter has already accounted for the number of commands outstanding to the device. Only commands going through function hpsa_cmd_resolve_events decrement this counter. - ioaccel commands go to either HBA disks or to logical volumes comprised of SSDs. The extra increment is causing device resets to hang. - Resets wait for all device outstanding commands to complete before returning. Replace unused field abort_pending with retry_pending. This is a maintenance driver so these changes have the least impact/risk. Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda Tested-by: Joe Szczypek <jszczype@redhat.com> Reviewed-by: Scott Benesh <scott.benesh@microchip.com> Reviewed-by: Scott Teel <scott.teel@microchip.com> Reviewed-by: Tomas Henzl <thenzl@redhat.com> Signed-off-by: Don Brace <don.brace@microchip.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> Don, do you happen to know why this patch caused some controller init failure for device 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 ? Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good The difference between the two boots is f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 in -good case. Looks like hpsa controller fails to initialize in bad case (could be a race?). -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" bre @ 2021-03-03 0:22 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-03 0:22 UTC (permalink / raw) To: John Paul Adrian Glaubitz, Don Brace Cc: linux-ia64, linux-kernel, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen On Tue, 2 Mar 2021 23:31:32 +0100 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > Hi Sergei! > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > Gave v5.12-rc1 a try today and got a similar boot failure around > > hpsa queue initialization, but my failure is later: > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > Maybe I get different error because I flipped on most debugging > > kernel options :) > > > > Looks like 'ERROR: Invalid distance value range' while being > > very scary are harmless. It's just a new spammy way for kernel > > to report lack of NUMA config on the machine (no SRAT and SLIT > > ACPI tables). > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > configuration is very wrong as I get unaligned accesses: > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > Bisecting now. > > Sounds good. I guess we should get Jens' fix for the signal regression > merged as well as your two fixes for strace. "bisected" (cheated halfway through) and verified that reverting f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. CCing authors who might be able to help us here. commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 Author: Don Brace <don.brace@microchip.com> Date: Mon Feb 15 16:26:57 2021 -0600 scsi: hpsa: Correct dev cmds outstanding for retried cmds Prevent incrementing device->commands_outstanding for ioaccel command retries that are driver initiated. If the command goes through the retry path, the device->commands_outstanding counter has already accounted for the number of commands outstanding to the device. Only commands going through function hpsa_cmd_resolve_events decrement this counter. - ioaccel commands go to either HBA disks or to logical volumes comprised of SSDs. The extra increment is causing device resets to hang. - Resets wait for all device outstanding commands to complete before returning. Replace unused field abort_pending with retry_pending. This is a maintenance driver so these changes have the least impact/risk. Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda Tested-by: Joe Szczypek <jszczype@redhat.com> Reviewed-by: Scott Benesh <scott.benesh@microchip.com> Reviewed-by: Scott Teel <scott.teel@microchip.com> Reviewed-by: Tomas Henzl <thenzl@redhat.com> Signed-off-by: Don Brace <don.brace@microchip.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> Don, do you happen to know why this patch caused some controller init failure for device 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 ? Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good The difference between the two boots is f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 in -good case. Looks like hpsa controller fails to initialize in bad case (could be a race?). -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-03 0:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" bre Sergei Trofimovich @ 2021-03-03 8:55 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-03 8:55 UTC (permalink / raw) To: John Paul Adrian Glaubitz, Don Brace, storagedev, linux-scsi Cc: linux-ia64, linux-kernel, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen On Wed, 3 Mar 2021 00:22:36 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Tue, 2 Mar 2021 23:31:32 +0100 > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > Hi Sergei! > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > hpsa queue initialization, but my failure is later: > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > Maybe I get different error because I flipped on most debugging > > > kernel options :) > > > > > > Looks like 'ERROR: Invalid distance value range' while being > > > very scary are harmless. It's just a new spammy way for kernel > > > to report lack of NUMA config on the machine (no SRAT and SLIT > > > ACPI tables). > > > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > > configuration is very wrong as I get unaligned accesses: > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > > > Bisecting now. > > > > Sounds good. I guess we should get Jens' fix for the signal regression > > merged as well as your two fixes for strace. > > "bisected" (cheated halfway through) and verified that reverting > f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. > > CCing authors who might be able to help us here. > > commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 > Author: Don Brace <don.brace@microchip.com> > Date: Mon Feb 15 16:26:57 2021 -0600 > > scsi: hpsa: Correct dev cmds outstanding for retried cmds > > Prevent incrementing device->commands_outstanding for ioaccel command > retries that are driver initiated. If the command goes through the retry > path, the device->commands_outstanding counter has already accounted for > the number of commands outstanding to the device. Only commands going > through function hpsa_cmd_resolve_events decrement this counter. > > - ioaccel commands go to either HBA disks or to logical volumes comprised > of SSDs. > > The extra increment is causing device resets to hang. > > - Resets wait for all device outstanding commands to complete before > returning. > > Replace unused field abort_pending with retry_pending. This is a > maintenance driver so these changes have the least impact/risk. > > Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda > Tested-by: Joe Szczypek <jszczype@redhat.com> > Reviewed-by: Scott Benesh <scott.benesh@microchip.com> > Reviewed-by: Scott Teel <scott.teel@microchip.com> > Reviewed-by: Tomas Henzl <thenzl@redhat.com> > Signed-off-by: Don Brace <don.brace@microchip.com> > Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> > > Don, do you happen to know why this patch caused some controller init failure > for device > 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 > ? > > Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good > > The difference between the two boots is > f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 > in -good case. > > Looks like hpsa controller fails to initialize in bad case (could be a race?). Also CCing hpsa maintainer mailing lists. Looking more into the suspect commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f749d8b7a9896bc6e5ffe104cc64345037e0b152 it roughly does the: @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ... @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); But I don't immediately see anything wrong with it. -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-03 8:55 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-03 8:55 UTC (permalink / raw) To: John Paul Adrian Glaubitz, Don Brace, storagedev, linux-scsi Cc: linux-ia64, linux-kernel, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen On Wed, 3 Mar 2021 00:22:36 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Tue, 2 Mar 2021 23:31:32 +0100 > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > Hi Sergei! > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > hpsa queue initialization, but my failure is later: > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > Maybe I get different error because I flipped on most debugging > > > kernel options :) > > > > > > Looks like 'ERROR: Invalid distance value range' while being > > > very scary are harmless. It's just a new spammy way for kernel > > > to report lack of NUMA config on the machine (no SRAT and SLIT > > > ACPI tables). > > > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > > configuration is very wrong as I get unaligned accesses: > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > > > Bisecting now. > > > > Sounds good. I guess we should get Jens' fix for the signal regression > > merged as well as your two fixes for strace. > > "bisected" (cheated halfway through) and verified that reverting > f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. > > CCing authors who might be able to help us here. > > commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 > Author: Don Brace <don.brace@microchip.com> > Date: Mon Feb 15 16:26:57 2021 -0600 > > scsi: hpsa: Correct dev cmds outstanding for retried cmds > > Prevent incrementing device->commands_outstanding for ioaccel command > retries that are driver initiated. If the command goes through the retry > path, the device->commands_outstanding counter has already accounted for > the number of commands outstanding to the device. Only commands going > through function hpsa_cmd_resolve_events decrement this counter. > > - ioaccel commands go to either HBA disks or to logical volumes comprised > of SSDs. > > The extra increment is causing device resets to hang. > > - Resets wait for all device outstanding commands to complete before > returning. > > Replace unused field abort_pending with retry_pending. This is a > maintenance driver so these changes have the least impact/risk. > > Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda > Tested-by: Joe Szczypek <jszczype@redhat.com> > Reviewed-by: Scott Benesh <scott.benesh@microchip.com> > Reviewed-by: Scott Teel <scott.teel@microchip.com> > Reviewed-by: Tomas Henzl <thenzl@redhat.com> > Signed-off-by: Don Brace <don.brace@microchip.com> > Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> > > Don, do you happen to know why this patch caused some controller init failure > for device > 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 > ? > > Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good > > The difference between the two boots is > f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 > in -good case. > > Looks like hpsa controller fails to initialize in bad case (could be a race?). Also CCing hpsa maintainer mailing lists. Looking more into the suspect commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id÷49d8b7a9896bc6e5ffe104cc64345037e0b152 it roughly does the: @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ... @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); But I don't immediately see anything wrong with it. -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-03 8:55 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Sergei Trofimovich @ 2021-03-03 17:33 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-03 17:33 UTC (permalink / raw) To: slyich, glaubitz, storagedev, linux-scsi Cc: linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen [-- Attachment #1: Type: text/plain, Size: 10316 bytes --] -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Wednesday, March 3, 2021 2:56 AM To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com>; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe On Wed, 3 Mar 2021 00:22:36 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Tue, 2 Mar 2021 23:31:32 +0100 > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > Hi Sergei! > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > hpsa queue initialization, but my failure is later: > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > Maybe I get different error because I flipped on most debugging > > > kernel options :) > > > > > > Looks like 'ERROR: Invalid distance value range' while being very > > > scary are harmless. It's just a new spammy way for kernel to > > > report lack of NUMA config on the machine (no SRAT and SLIT ACPI > > > tables). > > > > > > At least I get hpsa detected on PCI bus. But I guess it's > > > discovered configuration is very wrong as I get unaligned accesses: > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 Running pahole before the patch: struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ int abort_pending; /* 652 4 */ struct hpsa_scsi_dev_t * device; /* 656 8 */ atomic_t refcount; /* 664 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 100 */ } __attribute__((__aligned__(128))); Pahole after the patch: struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ struct hpsa_scsi_dev_t * device; /* 652 8 */ bool retry_pending; /* 660 1 */ atomic_t refcount; /* 661 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 103 */ } __attribute__((__aligned__(128))); So, I did replace abort_pending field (an int) with retry_pending (bool) Can I send you a patch to just rename abort_pending to retry_pending using the same type and position? It will mean some minor code changes in the driver... With the above changes, pahole is the same struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ int retry_pending; /* 652 4 */ struct hpsa_scsi_dev_t * device; /* 656 8 */ atomic_t refcount; /* 664 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 100 */ } __attribute__((__aligned__(128))); > > > > > > Bisecting now. > > > > Sounds good. I guess we should get Jens' fix for the signal > > regression merged as well as your two fixes for strace. > > "bisected" (cheated halfway through) and verified that reverting > f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. > > CCing authors who might be able to help us here. > > commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 > Author: Don Brace <don.brace@microchip.com> > Date: Mon Feb 15 16:26:57 2021 -0600 > > scsi: hpsa: Correct dev cmds outstanding for retried cmds > > Prevent incrementing device->commands_outstanding for ioaccel command > retries that are driver initiated. If the command goes through the retry > path, the device->commands_outstanding counter has already accounted for > the number of commands outstanding to the device. Only commands going > through function hpsa_cmd_resolve_events decrement this counter. > > - ioaccel commands go to either HBA disks or to logical volumes comprised > of SSDs. > > The extra increment is causing device resets to hang. > > - Resets wait for all device outstanding commands to complete before > returning. > > Replace unused field abort_pending with retry_pending. This is a > maintenance driver so these changes have the least impact/risk. > > Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda > Tested-by: Joe Szczypek <jszczype@redhat.com> > Reviewed-by: Scott Benesh <scott.benesh@microchip.com> > Reviewed-by: Scott Teel <scott.teel@microchip.com> > Reviewed-by: Tomas Henzl <thenzl@redhat.com> > Signed-off-by: Don Brace <don.brace@microchip.com> > Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> > > Don, do you happen to know why this patch caused some controller init > failure for device > 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array > P600 ? > > Boot failure: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Boot success: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good > > The difference between the two boots is > f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 > in -good case. > > Looks like hpsa controller fails to initialize in bad case (could be a race?). Also CCing hpsa maintainer mailing lists. Looking more into the suspect commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f749d8b7a9896bc6e5ffe104cc64345037e0b152 it roughly does the: @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ... @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); But I don't immediately see anything wrong with it. -- Sergei [-- Attachment #2: hpsa-correct-dev-cmd-outstanding-for-retried-cmds-alignment-update --] [-- Type: application/octet-stream, Size: 3733 bytes --] commit ea1723092d356dc4d1266942aa132340677e7460 Author: Don Brace <dbrace@redhat.com> Date: Wed Mar 3 11:21:59 2021 -0600 hpsa: correct dev cmd outstanding for retried cmds Prevent incrementing device->commands_outstanding for retried commands. diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c index f4d3747cfa0b..43c8f7e25eb5 100644 --- a/drivers/scsi/hpsa.c +++ b/drivers/scsi/hpsa.c @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); reply_queue = h->reply_map[raw_smp_processor_id()]; @@ -5623,6 +5626,16 @@ static void hpsa_command_resubmit_worker(struct work_struct *work) return hpsa_cmd_free_and_done(c->h, c, cmd); } + /* + * SML retries come in through queue_command and therefore + * go through cmd_tagged_alloc. So a new command. + * + * Set a retry_pending flag for a driver initiated retry attempt + * on an existing command to indicate to not increment the + * device->commands_outstanding when submitted. + */ + c->retry_pending = 1; + if (c->cmd_type == CMD_IOACCEL2) { struct ctlr_info *h = c->h; struct io_accel2_cmd *c2 = &h->ioaccel2_cmd_pool[c->cmdindex]; @@ -5646,6 +5659,7 @@ static void hpsa_command_resubmit_worker(struct work_struct *work) } } hpsa_cmd_partial_init(c->h, c->cmdindex, c); + if (hpsa_ciss_submit(c->h, c, cmd, dev)) { /* * If we get here, it means dma mapping failed. Try @@ -5708,6 +5722,10 @@ static int hpsa_scsi_queue_command(struct Scsi_Host *sh, struct scsi_cmnd *cmd) /* * Call alternate submit routine for I/O accelerated commands. * Retries always go down the normal I/O path. + * Note: If cmd->retries is non-zero, then this is a SML + * initiated retry and not a driver initiated retry. + * This command has been obtained from cmd_tagged_alloc + * and is therefore a brand-new command. */ if (likely(cmd->retries == 0 && !blk_rq_is_passthrough(cmd->request) && @@ -6107,6 +6125,7 @@ static int hpsa_eh_device_reset_handler(struct scsi_cmnd *scsicmd) * at init, and managed by cmd_tagged_alloc() and cmd_tagged_free() using the * block request tag as an index into a table of entries. cmd_tagged_free() is * the complement, although cmd_free() may be called instead. + * This function is only called for new requests from queue_command. */ static struct CommandList *cmd_tagged_alloc(struct ctlr_info *h, struct scsi_cmnd *scmd) @@ -6141,6 +6160,11 @@ static struct CommandList *cmd_tagged_alloc(struct ctlr_info *h, } atomic_inc(&c->refcount); + /* + * This is a new command obtained from queue_command so + * there have not been any driver initiated retry attempts. + */ + c->retry_pending = 0; hpsa_cmd_partial_init(h, idx, c); return c; @@ -6210,6 +6234,11 @@ static struct CommandList *cmd_alloc(struct ctlr_info *h) } hpsa_cmd_partial_init(h, i, c); c->device = NULL; + /* + * cmd_alloc is for "internal" commands and they are never + * retried. + */ + c->retry_pending = 0; return c; } diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 46df2e3ff89b..e86af4e9eef0 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ^ permalink raw reply related [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-03 17:33 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-03 17:33 UTC (permalink / raw) To: slyich, glaubitz, storagedev, linux-scsi Cc: linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen [-- Attachment #1: Type: text/plain, Size: 10316 bytes --] -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Wednesday, March 3, 2021 2:56 AM To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com>; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe On Wed, 3 Mar 2021 00:22:36 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Tue, 2 Mar 2021 23:31:32 +0100 > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > Hi Sergei! > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > hpsa queue initialization, but my failure is later: > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > Maybe I get different error because I flipped on most debugging > > > kernel options :) > > > > > > Looks like 'ERROR: Invalid distance value range' while being very > > > scary are harmless. It's just a new spammy way for kernel to > > > report lack of NUMA config on the machine (no SRAT and SLIT ACPI > > > tables). > > > > > > At least I get hpsa detected on PCI bus. But I guess it's > > > discovered configuration is very wrong as I get unaligned accesses: > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 Running pahole before the patch: struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ int abort_pending; /* 652 4 */ struct hpsa_scsi_dev_t * device; /* 656 8 */ atomic_t refcount; /* 664 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 100 */ } __attribute__((__aligned__(128))); Pahole after the patch: struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ struct hpsa_scsi_dev_t * device; /* 652 8 */ bool retry_pending; /* 660 1 */ atomic_t refcount; /* 661 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 103 */ } __attribute__((__aligned__(128))); So, I did replace abort_pending field (an int) with retry_pending (bool) Can I send you a patch to just rename abort_pending to retry_pending using the same type and position? It will mean some minor code changes in the driver... With the above changes, pahole is the same struct CommandList { struct CommandListHeader Header; /* 0 20 */ struct RequestBlock Request; /* 20 20 */ struct ErrDescriptor ErrDesc; /* 40 12 */ struct SGDescriptor SG[32]; /* 52 512 */ /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ u32 busaddr; /* 564 4 */ struct ErrorInfo * err_info; /* 568 8 */ /* --- cacheline 9 boundary (576 bytes) --- */ struct ctlr_info * h; /* 576 8 */ int cmd_type; /* 584 4 */ long int cmdindex; /* 588 8 */ struct completion * waiting; /* 596 8 */ struct scsi_cmnd * scsi_cmd; /* 604 8 */ struct work_struct work; /* 612 32 */ /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ int retry_pending; /* 652 4 */ struct hpsa_scsi_dev_t * device; /* 656 8 */ atomic_t refcount; /* 664 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 100 */ } __attribute__((__aligned__(128))); > > > > > > Bisecting now. > > > > Sounds good. I guess we should get Jens' fix for the signal > > regression merged as well as your two fixes for strace. > > "bisected" (cheated halfway through) and verified that reverting > f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. > > CCing authors who might be able to help us here. > > commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 > Author: Don Brace <don.brace@microchip.com> > Date: Mon Feb 15 16:26:57 2021 -0600 > > scsi: hpsa: Correct dev cmds outstanding for retried cmds > > Prevent incrementing device->commands_outstanding for ioaccel command > retries that are driver initiated. If the command goes through the retry > path, the device->commands_outstanding counter has already accounted for > the number of commands outstanding to the device. Only commands going > through function hpsa_cmd_resolve_events decrement this counter. > > - ioaccel commands go to either HBA disks or to logical volumes comprised > of SSDs. > > The extra increment is causing device resets to hang. > > - Resets wait for all device outstanding commands to complete before > returning. > > Replace unused field abort_pending with retry_pending. This is a > maintenance driver so these changes have the least impact/risk. > > Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda > Tested-by: Joe Szczypek <jszczype@redhat.com> > Reviewed-by: Scott Benesh <scott.benesh@microchip.com> > Reviewed-by: Scott Teel <scott.teel@microchip.com> > Reviewed-by: Tomas Henzl <thenzl@redhat.com> > Signed-off-by: Don Brace <don.brace@microchip.com> > Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> > > Don, do you happen to know why this patch caused some controller init > failure for device > 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array > P600 ? > > Boot failure: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > Boot success: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good > > The difference between the two boots is > f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 > in -good case. > > Looks like hpsa controller fails to initialize in bad case (could be a race?). Also CCing hpsa maintainer mailing lists. Looking more into the suspect commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f749d8b7a9896bc6e5ffe104cc64345037e0b152 it roughly does the: @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ... @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); But I don't immediately see anything wrong with it. -- Sergei [-- Attachment #2: hpsa-correct-dev-cmd-outstanding-for-retried-cmds-alignment-update --] [-- Type: application/octet-stream, Size: 3733 bytes --] commit ea1723092d356dc4d1266942aa132340677e7460 Author: Don Brace <dbrace@redhat.com> Date: Wed Mar 3 11:21:59 2021 -0600 hpsa: correct dev cmd outstanding for retried cmds Prevent incrementing device->commands_outstanding for retried commands. diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c index f4d3747cfa0b..43c8f7e25eb5 100644 --- a/drivers/scsi/hpsa.c +++ b/drivers/scsi/hpsa.c @@ -1151,7 +1151,10 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info *h, { dial_down_lockup_detection_during_fw_flash(h, c); atomic_inc(&h->commands_outstanding); - if (c->device) + /* + * Check to see if the command is being retried. + */ + if (c->device && !c->retry_pending) atomic_inc(&c->device->commands_outstanding); reply_queue = h->reply_map[raw_smp_processor_id()]; @@ -5623,6 +5626,16 @@ static void hpsa_command_resubmit_worker(struct work_struct *work) return hpsa_cmd_free_and_done(c->h, c, cmd); } + /* + * SML retries come in through queue_command and therefore + * go through cmd_tagged_alloc. So a new command. + * + * Set a retry_pending flag for a driver initiated retry attempt + * on an existing command to indicate to not increment the + * device->commands_outstanding when submitted. + */ + c->retry_pending = 1; + if (c->cmd_type == CMD_IOACCEL2) { struct ctlr_info *h = c->h; struct io_accel2_cmd *c2 = &h->ioaccel2_cmd_pool[c->cmdindex]; @@ -5646,6 +5659,7 @@ static void hpsa_command_resubmit_worker(struct work_struct *work) } } hpsa_cmd_partial_init(c->h, c->cmdindex, c); + if (hpsa_ciss_submit(c->h, c, cmd, dev)) { /* * If we get here, it means dma mapping failed. Try @@ -5708,6 +5722,10 @@ static int hpsa_scsi_queue_command(struct Scsi_Host *sh, struct scsi_cmnd *cmd) /* * Call alternate submit routine for I/O accelerated commands. * Retries always go down the normal I/O path. + * Note: If cmd->retries is non-zero, then this is a SML + * initiated retry and not a driver initiated retry. + * This command has been obtained from cmd_tagged_alloc + * and is therefore a brand-new command. */ if (likely(cmd->retries == 0 && !blk_rq_is_passthrough(cmd->request) && @@ -6107,6 +6125,7 @@ static int hpsa_eh_device_reset_handler(struct scsi_cmnd *scsicmd) * at init, and managed by cmd_tagged_alloc() and cmd_tagged_free() using the * block request tag as an index into a table of entries. cmd_tagged_free() is * the complement, although cmd_free() may be called instead. + * This function is only called for new requests from queue_command. */ static struct CommandList *cmd_tagged_alloc(struct ctlr_info *h, struct scsi_cmnd *scmd) @@ -6141,6 +6160,11 @@ static struct CommandList *cmd_tagged_alloc(struct ctlr_info *h, } atomic_inc(&c->refcount); + /* + * This is a new command obtained from queue_command so + * there have not been any driver initiated retry attempts. + */ + c->retry_pending = 0; hpsa_cmd_partial_init(h, idx, c); return c; @@ -6210,6 +6234,11 @@ static struct CommandList *cmd_alloc(struct ctlr_info *h) } hpsa_cmd_partial_init(h, i, c); c->device = NULL; + /* + * cmd_alloc is for "internal" commands and they are never + * retried. + */ + c->retry_pending = 0; return c; } diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 46df2e3ff89b..e86af4e9eef0 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -448,7 +448,7 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - int abort_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); ^ permalink raw reply related [flat|nested] 89+ messages in thread
[parent not found: <20210303220401.501449e5@sf>]
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 [not found] ` <20210303220401.501449e5@sf> @ 2021-03-04 17:00 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-04 17:00 UTC (permalink / raw) To: slyich Cc: glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Wednesday, March 3, 2021 4:04 PM To: Don Brace - C33706 <Don.Brace@microchip.com> Cc: glaubitz@physik.fu-berlin.de; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org; linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; jszczype@redhat.com; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; thenzl@redhat.com; martin.petersen@oracle.com Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Glad to hear the patch works. The P600 is an unsupported controller that was removed some time ago (by us) and re-added in this patch: commit 135ae6edeb51979d0998daf1357f149a7d6ebb08 scsi: hpsa: add support for legacy boards Author: Hannes Reinecke <hare@suse.de> Date: Tue Aug 15 08:58:04 2017 +0200 So, when testing the original patch, I no longer have a P600 to test with. It used to be supported by our obsoleted cciss driver. Since patch f749d8b7a9896bc6e5ffe104cc64345037e0b152 scsi: hpsa: Correct dev cmds outstanding for retried cmds has already been applied to Martin Petersons 5.13/scsi-queue, perhaps it would be better to send up another patch to correct your alignment issue on these legacy boards. Wondering what others think about this? Thanks, Don ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-04 17:00 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-04 17:00 UTC (permalink / raw) To: slyich Cc: glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Wednesday, March 3, 2021 4:04 PM To: Don Brace - C33706 <Don.Brace@microchip.com> Cc: glaubitz@physik.fu-berlin.de; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org; linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; jszczype@redhat.com; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; thenzl@redhat.com; martin.petersen@oracle.com Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Glad to hear the patch works. The P600 is an unsupported controller that was removed some time ago (by us) and re-added in this patch: commit 135ae6edeb51979d0998daf1357f149a7d6ebb08 scsi: hpsa: add support for legacy boards Author: Hannes Reinecke <hare@suse.de> Date: Tue Aug 15 08:58:04 2017 +0200 So, when testing the original patch, I no longer have a P600 to test with. It used to be supported by our obsoleted cciss driver. Since patch f749d8b7a9896bc6e5ffe104cc64345037e0b152 scsi: hpsa: Correct dev cmds outstanding for retried cmds has already been applied to Martin Petersons 5.13/scsi-queue, perhaps it would be better to send up another patch to correct your alignment issue on these legacy boards. Wondering what others think about this? Thanks, Don ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-04 17:00 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace @ 2021-03-05 13:26 ` Tomas Henzl -1 siblings, 0 replies; 89+ messages in thread From: Tomas Henzl @ 2021-03-05 13:26 UTC (permalink / raw) To: Don.Brace, slyich Cc: glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, martin.petersen On 3/4/21 6:00 PM, Don.Brace@microchip.com wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyich@gmail.com] > Sent: Wednesday, March 3, 2021 4:04 PM > To: Don Brace - C33706 <Don.Brace@microchip.com> > Cc: glaubitz@physik.fu-berlin.de; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org; linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; jszczype@redhat.com; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; thenzl@redhat.com; martin.petersen@oracle.com > Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 > > Glad to hear the patch works. > The P600 is an unsupported controller that was removed some time ago (by us) and re-added in this patch: > commit 135ae6edeb51979d0998daf1357f149a7d6ebb08 scsi: hpsa: add support for legacy boards > Author: Hannes Reinecke <hare@suse.de> > Date: Tue Aug 15 08:58:04 2017 +0200 > > So, when testing the original patch, I no longer have a P600 to test with. It used to be supported by our obsoleted cciss driver. Do you know why this is specific to P600, since the struct is used only internally in the driver and not used to communicate with the controller or am I wrong? > > Since patch f749d8b7a9896bc6e5ffe104cc64345037e0b152 scsi: hpsa: Correct dev cmds outstanding for retried cmds has > already been applied to Martin Petersons 5.13/scsi-queue, perhaps it would be better to send up another patch to correct your alignment issue on these legacy boards. > > Wondering what others think about this? I agree with you I'd prefer a new patch. Thanks, Tomas > > Thanks, > Don > ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-05 13:26 ` Tomas Henzl 0 siblings, 0 replies; 89+ messages in thread From: Tomas Henzl @ 2021-03-05 13:26 UTC (permalink / raw) To: Don.Brace, slyich Cc: glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, martin.petersen On 3/4/21 6:00 PM, Don.Brace@microchip.com wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyich@gmail.com] > Sent: Wednesday, March 3, 2021 4:04 PM > To: Don Brace - C33706 <Don.Brace@microchip.com> > Cc: glaubitz@physik.fu-berlin.de; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org; linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; jszczype@redhat.com; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; thenzl@redhat.com; martin.petersen@oracle.com > Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 > > Glad to hear the patch works. > The P600 is an unsupported controller that was removed some time ago (by us) and re-added in this patch: > commit 135ae6edeb51979d0998daf1357f149a7d6ebb08 scsi: hpsa: add support for legacy boards > Author: Hannes Reinecke <hare@suse.de> > Date: Tue Aug 15 08:58:04 2017 +0200 > > So, when testing the original patch, I no longer have a P600 to test with. It used to be supported by our obsoleted cciss driver. Do you know why this is specific to P600, since the struct is used only internally in the driver and not used to communicate with the controller or am I wrong? > > Since patch f749d8b7a9896bc6e5ffe104cc64345037e0b152 scsi: hpsa: Correct dev cmds outstanding for retried cmds has > already been applied to Martin Petersons 5.13/scsi-queue, perhaps it would be better to send up another patch to correct your alignment issue on these legacy boards. > > Wondering what others think about this? I agree with you I'd prefer a new patch. Thanks, Tomas > > Thanks, > Don > ^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-05 13:26 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Tomas Henzl @ 2021-03-12 22:27 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-12 22:27 UTC (permalink / raw) To: linux-kernel Cc: Sergei Trofimovich, linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace, John Paul Adrian Glaubitz The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ +#include <linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-12 22:27 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-12 22:27 UTC (permalink / raw) To: linux-kernel Cc: Sergei Trofimovich, linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace, John Paul Adrian Glaubitz The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ +#include <linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) = 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-12 22:27 ` Sergei Trofimovich @ 2021-03-16 16:30 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-16 16:30 UTC (permalink / raw) To: slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen, glaubitz -----Original Message----- From: Sergei Trofimovich [mailto:slyfox@gentoo.org] Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ #include +<linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we +break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual +structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % +__alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 Thank-you for your testing. I would rather you add the atomic_t alignment check only. The current patch under review has other changes... https://patchwork.kernel.org/project/linux-scsi/patch/161540317205.18786.5821926127237311408.stgit@brunhilda/ ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-16 16:30 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-16 16:30 UTC (permalink / raw) To: slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen, glaubitz -----Original Message----- From: Sergei Trofimovich [mailto:slyfox@gentoo.org] Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ #include +<linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we +break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual +structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % +__alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 Thank-you for your testing. I would rather you add the atomic_t alignment check only. The current patch under review has other changes... https://patchwork.kernel.org/project/linux-scsi/patch/161540317205.18786.5821926127237311408.stgit@brunhilda/ ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-16 16:30 ` Don.Brace @ 2021-03-16 18:28 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-16 18:28 UTC (permalink / raw) To: Don.Brace Cc: Sergei Trofimovich, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen, John Paul Adrian Glaubitz On Tue, Mar 16, 2021 at 6:12 PM <Don.Brace@microchip.com> wrote: > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ #include > +<linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we > +break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual > +structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % > +__alignof__(atomic_t) == 0); > + Actually that still feels wrong: the annotation of the struct is to pack every member, which causes the access to be done in byte units on architectures that do not have hardware unaligned load/store instructions, at least for things like atomic_read() that does not go through a cmpxchg() or ll/sc cycle. This change may fix itanium, but it's still not correct. Other architectures would have already been broken before the recent change, but that's not a reason against fixing them now. I'd recommend marking the entire structure as having default alignment, by adding the appropriate pragmas before and after it. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-16 18:28 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-16 18:28 UTC (permalink / raw) To: Don.Brace Cc: Sergei Trofimovich, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen, John Paul Adrian Glaubitz On Tue, Mar 16, 2021 at 6:12 PM <Don.Brace@microchip.com> wrote: > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ #include > +<linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we > +break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual > +structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % > +__alignof__(atomic_t) = 0); > + Actually that still feels wrong: the annotation of the struct is to pack every member, which causes the access to be done in byte units on architectures that do not have hardware unaligned load/store instructions, at least for things like atomic_read() that does not go through a cmpxchg() or ll/sc cycle. This change may fix itanium, but it's still not correct. Other architectures would have already been broken before the recent change, but that's not a reason against fixing them now. I'd recommend marking the entire structure as having default alignment, by adding the appropriate pragmas before and after it. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-16 18:28 ` Arnd Bergmann @ 2021-03-17 2:25 ` Martin K. Petersen -1 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-03-17 2:25 UTC (permalink / raw) To: Arnd Bergmann Cc: Don.Brace, Sergei Trofimovich, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen, John Paul Adrian Glaubitz Arnd, > Actually that still feels wrong: the annotation of the struct is to > pack every member, which causes the access to be done in byte units on > architectures that do not have hardware unaligned load/store > instructions, at least for things like atomic_read() that does not go > through a cmpxchg() or ll/sc cycle. > This change may fix itanium, but it's still not correct. Other > architectures would have already been broken before the recent change, > but that's not a reason against fixing them now. I agree. I understand why there are restrictions on fields consumed by the hardware. But for fields internal to the driver the packing doesn't make sense to me. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-17 2:25 ` Martin K. Petersen 0 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-03-17 2:25 UTC (permalink / raw) To: Arnd Bergmann Cc: Don.Brace, Sergei Trofimovich, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen, John Paul Adrian Glaubitz Arnd, > Actually that still feels wrong: the annotation of the struct is to > pack every member, which causes the access to be done in byte units on > architectures that do not have hardware unaligned load/store > instructions, at least for things like atomic_read() that does not go > through a cmpxchg() or ll/sc cycle. > This change may fix itanium, but it's still not correct. Other > architectures would have already been broken before the recent change, > but that's not a reason against fixing them now. I agree. I understand why there are restrictions on fields consumed by the hardware. But for fields internal to the driver the packing doesn't make sense to me. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-17 2:25 ` Martin K. Petersen (?) @ 2021-03-17 13:19 ` David Laight 2021-03-17 19:06 ` Don.Brace -1 siblings, 1 reply; 89+ messages in thread From: David Laight @ 2021-03-17 13:19 UTC (permalink / raw) To: 'Martin K. Petersen', Arnd Bergmann Cc: Don.Brace, Sergei Trofimovich, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, John Paul Adrian Glaubitz From: Martin K. Petersen > Sent: 17 March 2021 02:26 > > Arnd, > > > Actually that still feels wrong: the annotation of the struct is to > > pack every member, which causes the access to be done in byte units on > > architectures that do not have hardware unaligned load/store > > instructions, at least for things like atomic_read() that does not go > > through a cmpxchg() or ll/sc cycle. > > > This change may fix itanium, but it's still not correct. Other > > architectures would have already been broken before the recent change, > > but that's not a reason against fixing them now. > > I agree. I understand why there are restrictions on fields consumed by > the hardware. But for fields internal to the driver the packing doesn't > make sense to me. Jeepers -- that global #pragma pack(1) is bollocks. I think there are a couple of __u64 that are 32bit aligned. Just marking those field __packed __aligned(4) should have the desired effect. Or use a typedef for '__u64 with 32bit alignment'. (There probably ought to be one in types.h) Then add compile-time asserts that any non-trivial structures the hardware accesses are the right size. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-17 13:19 ` David Laight @ 2021-03-17 19:06 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-17 19:06 UTC (permalink / raw) To: David.Laight, martin.petersen, arnd Cc: slyfox, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, glaubitz -----Original Message----- From: David Laight [mailto:David.Laight@ACULAB.COM] Subject: RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) From: Martin K. Petersen > Sent: 17 March 2021 02:26 > > Arnd, > > > Actually that still feels wrong: the annotation of the struct is to > > pack every member, which causes the access to be done in byte units > > on architectures that do not have hardware unaligned load/store > > instructions, at least for things like atomic_read() that does not > > go through a cmpxchg() or ll/sc cycle. > > > This change may fix itanium, but it's still not correct. Other > > architectures would have already been broken before the recent > > change, but that's not a reason against fixing them now. > > I agree. I understand why there are restrictions on fields consumed by > the hardware. But for fields internal to the driver the packing > doesn't make sense to me. Jeepers -- that global #pragma pack(1) is bollocks. I think there are a couple of __u64 that are 32bit aligned. Just marking those field __packed __aligned(4) should have the desired effect. Or use a typedef for '__u64 with 32bit alignment'. (There probably ought to be one in types.h) Then add compile-time asserts that any non-trivial structures the hardware accesses are the right size. David Don: My dilemma is that hpsa is now a maintenance driver and making more packing/alignment changes would trigger a lot of regression testing. My last patch aligns the structure with what has been in place for a long time now. All I did was rename an unused variable. So making any more changes is not high on my todo list... ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-17 19:06 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-17 19:06 UTC (permalink / raw) To: David.Laight, martin.petersen, arnd Cc: slyfox, linux-kernel, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, glaubitz LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IERhdmlkIExhaWdodCBbbWFpbHRvOkRh dmlkLkxhaWdodEBBQ1VMQUIuQ09NXSANClN1YmplY3Q6IFJFOiBbUEFUQ0hdIGhwc2E6IGZpeCBi b290IG9uIGlhNjQgKGF0b21pY190IGFsaWdubWVudCkNCg0KRnJvbTogTWFydGluIEsuIFBldGVy c2VuDQo+IFNlbnQ6IDE3IE1hcmNoIDIwMjEgMDI6MjYNCj4NCj4gQXJuZCwNCj4NCj4gPiBBY3R1 YWxseSB0aGF0IHN0aWxsIGZlZWxzIHdyb25nOiB0aGUgYW5ub3RhdGlvbiBvZiB0aGUgc3RydWN0 IGlzIHRvIA0KPiA+IHBhY2sgZXZlcnkgbWVtYmVyLCB3aGljaCBjYXVzZXMgdGhlIGFjY2VzcyB0 byBiZSBkb25lIGluIGJ5dGUgdW5pdHMgDQo+ID4gb24gYXJjaGl0ZWN0dXJlcyB0aGF0IGRvIG5v dCBoYXZlIGhhcmR3YXJlIHVuYWxpZ25lZCBsb2FkL3N0b3JlIA0KPiA+IGluc3RydWN0aW9ucywg YXQgbGVhc3QgZm9yIHRoaW5ncyBsaWtlIGF0b21pY19yZWFkKCkgdGhhdCBkb2VzIG5vdCANCj4g PiBnbyB0aHJvdWdoIGEgY21weGNoZygpIG9yIGxsL3NjIGN5Y2xlLg0KPg0KPiA+IFRoaXMgY2hh bmdlIG1heSBmaXggaXRhbml1bSwgYnV0IGl0J3Mgc3RpbGwgbm90IGNvcnJlY3QuIE90aGVyIA0K PiA+IGFyY2hpdGVjdHVyZXMgd291bGQgaGF2ZSBhbHJlYWR5IGJlZW4gYnJva2VuIGJlZm9yZSB0 aGUgcmVjZW50IA0KPiA+IGNoYW5nZSwgYnV0IHRoYXQncyBub3QgYSByZWFzb24gYWdhaW5zdCBm aXhpbmcgdGhlbSBub3cuDQo+DQo+IEkgYWdyZWUuIEkgdW5kZXJzdGFuZCB3aHkgdGhlcmUgYXJl IHJlc3RyaWN0aW9ucyBvbiBmaWVsZHMgY29uc3VtZWQgYnkgDQo+IHRoZSBoYXJkd2FyZS4gQnV0 IGZvciBmaWVsZHMgaW50ZXJuYWwgdG8gdGhlIGRyaXZlciB0aGUgcGFja2luZyANCj4gZG9lc24n dCBtYWtlIHNlbnNlIHRvIG1lLg0KDQpKZWVwZXJzIC0tIHRoYXQgZ2xvYmFsICNwcmFnbWEgcGFj aygxKSBpcyBib2xsb2Nrcy4NCg0KSSB0aGluayB0aGVyZSBhcmUgYSBjb3VwbGUgb2YgX191NjQg dGhhdCBhcmUgMzJiaXQgYWxpZ25lZC4NCkp1c3QgbWFya2luZyB0aG9zZSBmaWVsZCBfX3BhY2tl ZCBfX2FsaWduZWQoNCkgc2hvdWxkIGhhdmUgdGhlIGRlc2lyZWQgZWZmZWN0Lg0KT3IgdXNlIGEg dHlwZWRlZiBmb3IgJ19fdTY0IHdpdGggMzJiaXQgYWxpZ25tZW50Jy4NCihUaGVyZSBwcm9iYWJs eSBvdWdodCB0byBiZSBvbmUgaW4gdHlwZXMuaCkNCg0KVGhlbiBhZGQgY29tcGlsZS10aW1lIGFz c2VydHMgdGhhdCBhbnkgbm9uLXRyaXZpYWwgc3RydWN0dXJlcyB0aGUgaGFyZHdhcmUgYWNjZXNz ZXMgYXJlIHRoZSByaWdodCBzaXplLg0KDQogICAgICAgIERhdmlkDQoNCkRvbjogTXkgZGlsZW1t YSBpcyB0aGF0IGhwc2EgaXMgbm93IGEgbWFpbnRlbmFuY2UgZHJpdmVyIGFuZCBtYWtpbmcgbW9y ZSBwYWNraW5nL2FsaWdubWVudCBjaGFuZ2VzIHdvdWxkIHRyaWdnZXIgYSBsb3Qgb2YgcmVncmVz c2lvbiB0ZXN0aW5nLiBNeSBsYXN0IHBhdGNoIGFsaWducyB0aGUgc3RydWN0dXJlIHdpdGggd2hh dCBoYXMgYmVlbiBpbiBwbGFjZSBmb3IgYSBsb25nIHRpbWUgbm93LiBBbGwgSSBkaWQgd2FzIHJl bmFtZSBhbiB1bnVzZWQgdmFyaWFibGUuIFNvIG1ha2luZyBhbnkgbW9yZSBjaGFuZ2VzIGlzIG5v dCBoaWdoIG9uIG15IHRvZG8gbGlzdC4uLg0KDQoNCg= ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-12 22:27 ` Sergei Trofimovich @ 2021-03-17 17:28 ` John Paul Adrian Glaubitz -1 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:28 UTC (permalink / raw) To: Sergei Trofimovich, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace Hi Sergei! On 3/12/21 11:27 PM, Sergei Trofimovich wrote: > The failure initially observed as boot failure on rx3600 ia64 machine > with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens > to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds > outstanding for retried cmds") introduced unexpected padding and > un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only > restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. I just gave it a try on my RX2660 and for me, the hpsa driver won't load even with your patch. Can you share your kernel configuration so I can give it a try? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-17 17:28 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:28 UTC (permalink / raw) To: Sergei Trofimovich, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace Hi Sergei! On 3/12/21 11:27 PM, Sergei Trofimovich wrote: > The failure initially observed as boot failure on rx3600 ia64 machine > with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens > to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds > outstanding for retried cmds") introduced unexpected padding and > un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only > restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. I just gave it a try on my RX2660 and for me, the hpsa driver won't load even with your patch. Can you share your kernel configuration so I can give it a try? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-17 17:28 ` John Paul Adrian Glaubitz (?) @ 2021-03-27 10:24 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-27 10:24 UTC (permalink / raw) To: John Paul Adrian Glaubitz Cc: linux-kernel, linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace On Wed, 17 Mar 2021 18:28:31 +0100 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > Hi Sergei! > > On 3/12/21 11:27 PM, Sergei Trofimovich wrote: > > The failure initially observed as boot failure on rx3600 ia64 machine > > with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > > swapper/0[1]: error during unaligned kernel access > > > > Here unaligned access comes from 'struct CommandList' that happens > > to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds > > outstanding for retried cmds") introduced unexpected padding and > > un-aligned atomic_t from natural alignment to something else. > > > > This change does not remove packing annotation from struct but only > > restores alignment of atomic variable. > > > > The change is tested on the same rx3600 machine. > > I just gave it a try on my RX2660 and for me, the hpsa driver won't load even > with your patch. > > Can you share your kernel configuration so I can give it a try? Sure! Here is a config from a few days ago: https://dev.gentoo.org/~slyfox/configs/guppy-config-5.12.0-rc4-00016-g427684abc9fd-dirty -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-12 22:27 ` Sergei Trofimovich @ 2021-03-24 7:08 ` John Paul Adrian Glaubitz -1 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-24 7:08 UTC (permalink / raw) To: Sergei Trofimovich, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace Hello! On 3/12/21 11:27 PM, Sergei Trofimovich wrote: > The failure initially observed as boot failure on rx3600 ia64 machine > with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens > to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds > outstanding for retried cmds") introduced unexpected padding and > un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only > restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. > > CC: linux-ia64@vger.kernel.org > CC: storagedev@microchip.com > CC: linux-scsi@vger.kernel.org > CC: Joe Szczypek <jszczype@redhat.com> > CC: Scott Benesh <scott.benesh@microchip.com> > CC: Scott Teel <scott.teel@microchip.com> > CC: Tomas Henzl <thenzl@redhat.com> > CC: "Martin K. Petersen" <martin.petersen@oracle.com> > CC: Don Brace <don.brace@microchip.com> > Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> > Suggested-by: Don Brace <don.brace@microchip.com> > Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" > Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> > --- > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h > index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ > +#include <linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) == 0); > + > /* Max S/G elements in I/O accelerator command */ > #define IOACCEL1_MAXSGENTRIES 24 > #define IOACCEL2_MAXSGENTRIES 28 I'm seeing this issue as well and without the patch, the kernel won't boot on multiple ia64 servers. Is there anything that speaks against fixing this? Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-24 7:08 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-24 7:08 UTC (permalink / raw) To: Sergei Trofimovich, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, Joe Szczypek, Scott Benesh, Scott Teel, Tomas Henzl, Martin K. Petersen, Don Brace Hello! On 3/12/21 11:27 PM, Sergei Trofimovich wrote: > The failure initially observed as boot failure on rx3600 ia64 machine > with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens > to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds > outstanding for retried cmds") introduced unexpected padding and > un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only > restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. > > CC: linux-ia64@vger.kernel.org > CC: storagedev@microchip.com > CC: linux-scsi@vger.kernel.org > CC: Joe Szczypek <jszczype@redhat.com> > CC: Scott Benesh <scott.benesh@microchip.com> > CC: Scott Teel <scott.teel@microchip.com> > CC: Tomas Henzl <thenzl@redhat.com> > CC: "Martin K. Petersen" <martin.petersen@oracle.com> > CC: Don Brace <don.brace@microchip.com> > Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> > Suggested-by: Don Brace <don.brace@microchip.com> > Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" > Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> > --- > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h > index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ > +#include <linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) = 0); > + > /* Max S/G elements in I/O accelerator command */ > #define IOACCEL1_MAXSGENTRIES 24 > #define IOACCEL2_MAXSGENTRIES 28 I'm seeing this issue as well and without the patch, the kernel won't boot on multiple ia64 servers. Is there anything that speaks against fixing this? Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-12 22:27 ` Sergei Trofimovich @ 2021-03-24 18:37 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-24 18:37 UTC (permalink / raw) To: slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen, glaubitz -----Original Message----- From: Sergei Trofimovich [mailto:slyfox@gentoo.org] Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ #include +<linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we +break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual +structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % +__alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 Acked-by: Don Brace <don.brace@microchip.com> Thanks for your patch and extra effort. ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-24 18:37 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-24 18:37 UTC (permalink / raw) To: slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen, glaubitz -----Original Message----- From: Sergei Trofimovich [mailto:slyfox@gentoo.org] Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change does not remove packing annotation from struct but only restores alignment of atomic variable. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,9 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/build_bug.h> /* static_assert */ #include +<linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -448,11 +451,20 @@ struct CommandList { */ struct hpsa_scsi_dev_t *phys_disk; - bool retry_pending; + int retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we +break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * Ideally this header should be cleaned up to only mark individual +structs as + * packed. + */ +static_assert(offsetof(struct CommandList, refcount) % +__alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.30.2 Acked-by: Don Brace <don.brace@microchip.com> Thanks for your patch and extra effort. ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-24 18:37 ` Don.Brace @ 2021-03-29 11:25 ` John Paul Adrian Glaubitz -1 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-29 11:25 UTC (permalink / raw) To: Don.Brace, slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen Hi! On 3/24/21 7:37 PM, Don.Brace@microchip.com wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyfox@gentoo.org] > Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) > > The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. > > CC: linux-ia64@vger.kernel.org > CC: storagedev@microchip.com > CC: linux-scsi@vger.kernel.org > CC: Joe Szczypek <jszczype@redhat.com> > CC: Scott Benesh <scott.benesh@microchip.com> > CC: Scott Teel <scott.teel@microchip.com> > CC: Tomas Henzl <thenzl@redhat.com> > CC: "Martin K. Petersen" <martin.petersen@oracle.com> > CC: Don Brace <don.brace@microchip.com> > Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> > Suggested-by: Don Brace <don.brace@microchip.com> > Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" > Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> > --- > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ #include > +<linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we > +break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual > +structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % > +__alignof__(atomic_t) == 0); > + > /* Max S/G elements in I/O accelerator command */ > #define IOACCEL1_MAXSGENTRIES 24 > #define IOACCEL2_MAXSGENTRIES 28 > -- > 2.30.2 > > > Acked-by: Don Brace <don.brace@microchip.com> > > Thanks for your patch and extra effort. Apologies for being so persistent, but has this patch been queued anywhere? This should be included for 5.12 if possible as it unbreaks the kernel on alot of Itanium servers (and potentially other machines with the HPSA controller). If no one wants to pick the patch up, it could go through Andrew Morton's tree (-mm). Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-29 11:25 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-29 11:25 UTC (permalink / raw) To: Don.Brace, slyfox, linux-kernel Cc: linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen Hi! On 3/24/21 7:37 PM, Don.Brace@microchip.com wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyfox@gentoo.org] > Subject: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) > > The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: > > kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 > kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 > hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. > swapper/0[1]: error during unaligned kernel access > > Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. > > This change does not remove packing annotation from struct but only restores alignment of atomic variable. > > The change is tested on the same rx3600 machine. > > CC: linux-ia64@vger.kernel.org > CC: storagedev@microchip.com > CC: linux-scsi@vger.kernel.org > CC: Joe Szczypek <jszczype@redhat.com> > CC: Scott Benesh <scott.benesh@microchip.com> > CC: Scott Teel <scott.teel@microchip.com> > CC: Tomas Henzl <thenzl@redhat.com> > CC: "Martin K. Petersen" <martin.petersen@oracle.com> > CC: Don Brace <don.brace@microchip.com> > Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> > Suggested-by: Don Brace <don.brace@microchip.com> > Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" > Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> > --- > drivers/scsi/hpsa_cmd.h | 14 +++++++++++++- > 1 file changed, 13 insertions(+), 1 deletion(-) > > diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..617bdae9a7de 100644 > --- a/drivers/scsi/hpsa_cmd.h > +++ b/drivers/scsi/hpsa_cmd.h > @@ -20,6 +20,9 @@ > #ifndef HPSA_CMD_H > #define HPSA_CMD_H > > +#include <linux/build_bug.h> /* static_assert */ #include > +<linux/stddef.h> /* offsetof */ > + > /* general boundary defintions */ > #define SENSEINFOBYTES 32 /* may vary between hbas */ > #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ > @@ -448,11 +451,20 @@ struct CommandList { > */ > struct hpsa_scsi_dev_t *phys_disk; > > - bool retry_pending; > + int retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); > > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we > +break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * Ideally this header should be cleaned up to only mark individual > +structs as > + * packed. > + */ > +static_assert(offsetof(struct CommandList, refcount) % > +__alignof__(atomic_t) = 0); > + > /* Max S/G elements in I/O accelerator command */ > #define IOACCEL1_MAXSGENTRIES 24 > #define IOACCEL2_MAXSGENTRIES 28 > -- > 2.30.2 > > > Acked-by: Don Brace <don.brace@microchip.com> > > Thanks for your patch and extra effort. Apologies for being so persistent, but has this patch been queued anywhere? This should be included for 5.12 if possible as it unbreaks the kernel on alot of Itanium servers (and potentially other machines with the HPSA controller). If no one wants to pick the patch up, it could go through Andrew Morton's tree (-mm). Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-29 11:25 ` John Paul Adrian Glaubitz @ 2021-03-29 14:22 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-29 14:22 UTC (permalink / raw) To: John Paul Adrian Glaubitz Cc: Don.Brace, Sergei Trofimovich, Linux Kernel Mailing List, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen On Mon, Mar 29, 2021 at 1:28 PM John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > On 3/24/21 7:37 PM, Don.Brace@microchip.com wrote: > > > > Acked-by: Don Brace <don.brace@microchip.com> > > > > Thanks for your patch and extra effort. > > Apologies for being so persistent, but has this patch been queued anywhere? > > This should be included for 5.12 if possible as it unbreaks the kernel on alot > of Itanium servers (and potentially other machines with the HPSA controller). > > If no one wants to pick the patch up, it could go through Andrew Morton's tree (-mm). > I think Martin is still waiting for a fixed version of the patch, as the proposed patch from March 12 only solves the immediate symptom, but not the underlying problem of the CommandList structure being marked as unaligned. If it gets fixed, the new version should work on all architectures. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-29 14:22 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-29 14:22 UTC (permalink / raw) To: John Paul Adrian Glaubitz Cc: Don.Brace, Sergei Trofimovich, Linux Kernel Mailing List, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen On Mon, Mar 29, 2021 at 1:28 PM John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > On 3/24/21 7:37 PM, Don.Brace@microchip.com wrote: > > > > Acked-by: Don Brace <don.brace@microchip.com> > > > > Thanks for your patch and extra effort. > > Apologies for being so persistent, but has this patch been queued anywhere? > > This should be included for 5.12 if possible as it unbreaks the kernel on alot > of Itanium servers (and potentially other machines with the HPSA controller). > > If no one wants to pick the patch up, it could go through Andrew Morton's tree (-mm). > I think Martin is still waiting for a fixed version of the patch, as the proposed patch from March 12 only solves the immediate symptom, but not the underlying problem of the CommandList structure being marked as unaligned. If it gets fixed, the new version should work on all architectures. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-29 14:22 ` Arnd Bergmann @ 2021-03-30 3:02 ` Martin K. Petersen -1 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-03-30 3:02 UTC (permalink / raw) To: Arnd Bergmann Cc: John Paul Adrian Glaubitz, Don.Brace, Sergei Trofimovich, Linux Kernel Mailing List, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen Arnd, > I think Martin is still waiting for a fixed version of the patch, as > the proposed patch from March 12 only solves the immediate symptom, > but not the underlying problem of the CommandList structure being > marked as unaligned. If it gets fixed, the new version should work on > all architectures. Yep. I unfortunately don't have any hpsa adapters to test with. Was hoping somebody with hardware would attempt to fix up the struct properly. Given -rc5 we're running out of time so for 5.12 it's probably best if we queue up the workaround. I would prefer an amalgamation of Don's and Sergei's patches, though. I do like the assert so we can catch problems early. But really, somebody should fix this. While hpsa may be out of commercial support, Linux will support the driver it until there are no more users. And the current structure packing is just wrong. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-30 3:02 ` Martin K. Petersen 0 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-03-30 3:02 UTC (permalink / raw) To: Arnd Bergmann Cc: John Paul Adrian Glaubitz, Don.Brace, Sergei Trofimovich, Linux Kernel Mailing List, linux-ia64, storagedev, linux-scsi, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen Arnd, > I think Martin is still waiting for a fixed version of the patch, as > the proposed patch from March 12 only solves the immediate symptom, > but not the underlying problem of the CommandList structure being > marked as unaligned. If it gets fixed, the new version should work on > all architectures. Yep. I unfortunately don't have any hpsa adapters to test with. Was hoping somebody with hardware would attempt to fix up the struct properly. Given -rc5 we're running out of time so for 5.12 it's probably best if we queue up the workaround. I would prefer an amalgamation of Don's and Sergei's patches, though. I do like the assert so we can catch problems early. But really, somebody should fix this. While hpsa may be out of commercial support, Linux will support the driver it until there are no more users. And the current structure packing is just wrong. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide 2021-03-30 3:02 ` Martin K. Petersen @ 2021-03-30 7:19 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich Some of the structs contain `atomic_t` values and are not intended to be sent to IO controller as is. The change adds __packed to every struct and union in the file. Follow-up commits will fix `atomic_t` problems. The commit is a no-op at least on ia64: $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 68 ++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..280e933d27e7 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,8 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/compiler.h> + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -200,12 +202,10 @@ union u64bit { MAX_EXT_TARGETS + 1) /* + 1 is for the controller itself */ /* SCSI-3 Commands */ -#pragma pack(1) - #define HPSA_INQUIRY 0x12 struct InquiryData { u8 data_byte[36]; -}; +} __packed; #define HPSA_REPORT_LOG 0xc2 /* Report Logical LUNs */ #define HPSA_REPORT_PHYS 0xc3 /* Report Physical LUNs */ @@ -221,7 +221,7 @@ struct raid_map_disk_data { u8 xor_mult[2]; /**< XOR multipliers for this position, * valid for data disks only */ u8 reserved[2]; -}; +} __packed; struct raid_map_data { __le32 structure_size; /* Size of entire structure in bytes */ @@ -247,14 +247,14 @@ struct raid_map_data { __le16 dekindex; /* Data encryption key index. */ u8 reserved[16]; struct raid_map_disk_data data[RAID_MAP_MAX_ENTRIES]; -}; +} __packed; struct ReportLUNdata { u8 LUNListLength[4]; u8 extended_response_flag; u8 reserved[3]; u8 LUN[HPSA_MAX_LUN][8]; -}; +} __packed; struct ext_report_lun_entry { u8 lunid[8]; @@ -269,20 +269,20 @@ struct ext_report_lun_entry { u8 lun_count; /* multi-lun device, how many luns */ u8 redundant_paths; u32 ioaccel_handle; /* ioaccel1 only uses lower 16 bits */ -}; +} __packed; struct ReportExtendedLUNdata { u8 LUNListLength[4]; u8 extended_response_flag; u8 reserved[3]; struct ext_report_lun_entry LUN[HPSA_MAX_PHYS_LUN]; -}; +} __packed; struct SenseSubsystem_info { u8 reserved[36]; u8 portname[8]; u8 reserved1[1108]; -}; +} __packed; /* BMIC commands */ #define BMIC_READ 0x26 @@ -317,7 +317,7 @@ union SCSI3Addr { u8 Targ:6; u8 Mode:2; /* b10 */ } LogUnit; -}; +} __packed; struct PhysDevAddr { u32 TargetId:24; @@ -325,20 +325,20 @@ struct PhysDevAddr { u32 Mode:2; /* 2 level target device addr */ union SCSI3Addr Target[2]; -}; +} __packed; struct LogDevAddr { u32 VolId:30; u32 Mode:2; u8 reserved[4]; -}; +} __packed; union LUNAddr { u8 LunAddrBytes[8]; union SCSI3Addr SCSI3Lun[4]; struct PhysDevAddr PhysDev; struct LogDevAddr LogDev; -}; +} __packed; struct CommandListHeader { u8 ReplyQueue; @@ -346,7 +346,7 @@ struct CommandListHeader { __le16 SGTotal; __le64 tag; union LUNAddr LUN; -}; +} __packed; struct RequestBlock { u8 CDBLen; @@ -365,18 +365,18 @@ struct RequestBlock { #define GET_DIR(tad) (((tad) >> 6) & 0x03) u16 Timeout; u8 CDB[16]; -}; +} __packed; struct ErrDescriptor { __le64 Addr; __le32 Len; -}; +} __packed; struct SGDescriptor { __le64 Addr; __le32 Len; __le32 Ext; -}; +} __packed; union MoreErrInfo { struct { @@ -390,7 +390,8 @@ union MoreErrInfo { u8 offense_num; /* byte # of offense 0-base */ u32 offense_value; } Invalid_Cmd; -}; +} __packed; + struct ErrorInfo { u8 ScsiStatus; u8 SenseLen; @@ -398,7 +399,7 @@ struct ErrorInfo { u32 ResidualCnt; union MoreErrInfo MoreErrInfo; u8 SenseInfo[SENSEINFOBYTES]; -}; +} __packed; /* Command types */ #define CMD_IOCTL_PEND 0x01 #define CMD_SCSI 0x03 @@ -451,7 +452,7 @@ struct CommandList { bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ -} __aligned(COMMANDLIST_ALIGNMENT); +} __packed __aligned(COMMANDLIST_ALIGNMENT); /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 @@ -489,7 +490,7 @@ struct io_accel1_cmd { __le64 host_addr; /* 0x70 - 0x77 */ u8 CISS_LUN[8]; /* 0x78 - 0x7F */ struct SGDescriptor SG[IOACCEL1_MAXSGENTRIES]; -} __aligned(IOACCEL1_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL1_COMMANDLIST_ALIGNMENT); #define IOACCEL1_FUNCTION_SCSIIO 0x00 #define IOACCEL1_SGLOFFSET 32 @@ -519,7 +520,7 @@ struct ioaccel2_sg_element { u8 chain_indicator; #define IOACCEL2_CHAIN 0x80 #define IOACCEL2_LAST_SG 0x40 -}; +} __packed; /* * SCSI Response Format structure for IO Accelerator Mode 2 @@ -559,7 +560,7 @@ struct io_accel2_scsi_response { u8 sense_data_len; /* sense/response data length */ u8 resid_cnt[4]; /* residual count */ u8 sense_data_buff[32]; /* sense/response data buffer */ -}; +} __packed; /* * Structure for I/O accelerator (mode 2 or m2) commands. @@ -592,7 +593,7 @@ struct io_accel2_cmd { __le32 tweak_upper; /* Encryption tweak, upper 4 bytes */ struct ioaccel2_sg_element sg[IOACCEL2_MAXSGENTRIES]; struct io_accel2_scsi_response error_data; -} __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); /* * defines for Mode 2 command struct @@ -618,7 +619,7 @@ struct hpsa_tmf_struct { __le64 abort_tag; /* cciss tag of SCSI cmd or TMF to abort */ __le64 error_ptr; /* Error Pointer */ __le32 error_len; /* Error Length */ -} __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); /* Configuration Table Structure */ struct HostWrite { @@ -626,7 +627,7 @@ struct HostWrite { __le32 command_pool_addr_hi; __le32 CoalIntDelay; __le32 CoalIntCount; -}; +} __packed; #define SIMPLE_MODE 0x02 #define PERFORMANT_MODE 0x04 @@ -675,7 +676,7 @@ struct CfgTable { #define HPSA_EVENT_NOTIFY_ACCEL_IO_PATH_STATE_CHANGE (1 << 30) #define HPSA_EVENT_NOTIFY_ACCEL_IO_PATH_CONFIG_CHANGE (1 << 31) __le32 clear_event_notify; -}; +} __packed; #define NUM_BLOCKFETCH_ENTRIES 8 struct TransTable_struct { @@ -686,14 +687,14 @@ struct TransTable_struct { __le32 RepQCtrAddrHigh32; #define MAX_REPLY_QUEUES 64 struct vals32 RepQAddr[MAX_REPLY_QUEUES]; -}; +} __packed; struct hpsa_pci_info { unsigned char bus; unsigned char dev_fn; unsigned short domain; u32 board_id; -}; +} __packed; struct bmic_identify_controller { u8 configured_logical_drive_count; /* offset 0 */ @@ -702,7 +703,7 @@ struct bmic_identify_controller { u8 pad2[136]; u8 controller_mode; /* offset 292 */ u8 pad3[32]; -}; +} __packed; struct bmic_identify_physical_device { @@ -845,7 +846,7 @@ struct bmic_identify_physical_device { u8 max_link_rate[256]; u8 neg_phys_link_rate[256]; u8 box_conn_name[8]; -} __attribute((aligned(512))); +} __packed __attribute((aligned(512))); struct bmic_sense_subsystem_info { u8 primary_slot_number; @@ -858,7 +859,7 @@ struct bmic_sense_subsystem_info { u8 secondary_array_serial_number[32]; u8 secondary_cache_serial_number[32]; u8 pad[332]; -}; +} __packed; struct bmic_sense_storage_box_params { u8 reserved[36]; @@ -870,7 +871,6 @@ struct bmic_sense_storage_box_params { u8 reserver_3[84]; u8 phys_connector[2]; u8 reserved_4[296]; -}; +} __packed; -#pragma pack() #endif /* HPSA_CMD_H */ -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide @ 2021-03-30 7:19 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich Some of the structs contain `atomic_t` values and are not intended to be sent to IO controller as is. The change adds __packed to every struct and union in the file. Follow-up commits will fix `atomic_t` problems. The commit is a no-op at least on ia64: $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 68 ++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index d126bb877250..280e933d27e7 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -20,6 +20,8 @@ #ifndef HPSA_CMD_H #define HPSA_CMD_H +#include <linux/compiler.h> + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -200,12 +202,10 @@ union u64bit { MAX_EXT_TARGETS + 1) /* + 1 is for the controller itself */ /* SCSI-3 Commands */ -#pragma pack(1) - #define HPSA_INQUIRY 0x12 struct InquiryData { u8 data_byte[36]; -}; +} __packed; #define HPSA_REPORT_LOG 0xc2 /* Report Logical LUNs */ #define HPSA_REPORT_PHYS 0xc3 /* Report Physical LUNs */ @@ -221,7 +221,7 @@ struct raid_map_disk_data { u8 xor_mult[2]; /**< XOR multipliers for this position, * valid for data disks only */ u8 reserved[2]; -}; +} __packed; struct raid_map_data { __le32 structure_size; /* Size of entire structure in bytes */ @@ -247,14 +247,14 @@ struct raid_map_data { __le16 dekindex; /* Data encryption key index. */ u8 reserved[16]; struct raid_map_disk_data data[RAID_MAP_MAX_ENTRIES]; -}; +} __packed; struct ReportLUNdata { u8 LUNListLength[4]; u8 extended_response_flag; u8 reserved[3]; u8 LUN[HPSA_MAX_LUN][8]; -}; +} __packed; struct ext_report_lun_entry { u8 lunid[8]; @@ -269,20 +269,20 @@ struct ext_report_lun_entry { u8 lun_count; /* multi-lun device, how many luns */ u8 redundant_paths; u32 ioaccel_handle; /* ioaccel1 only uses lower 16 bits */ -}; +} __packed; struct ReportExtendedLUNdata { u8 LUNListLength[4]; u8 extended_response_flag; u8 reserved[3]; struct ext_report_lun_entry LUN[HPSA_MAX_PHYS_LUN]; -}; +} __packed; struct SenseSubsystem_info { u8 reserved[36]; u8 portname[8]; u8 reserved1[1108]; -}; +} __packed; /* BMIC commands */ #define BMIC_READ 0x26 @@ -317,7 +317,7 @@ union SCSI3Addr { u8 Targ:6; u8 Mode:2; /* b10 */ } LogUnit; -}; +} __packed; struct PhysDevAddr { u32 TargetId:24; @@ -325,20 +325,20 @@ struct PhysDevAddr { u32 Mode:2; /* 2 level target device addr */ union SCSI3Addr Target[2]; -}; +} __packed; struct LogDevAddr { u32 VolId:30; u32 Mode:2; u8 reserved[4]; -}; +} __packed; union LUNAddr { u8 LunAddrBytes[8]; union SCSI3Addr SCSI3Lun[4]; struct PhysDevAddr PhysDev; struct LogDevAddr LogDev; -}; +} __packed; struct CommandListHeader { u8 ReplyQueue; @@ -346,7 +346,7 @@ struct CommandListHeader { __le16 SGTotal; __le64 tag; union LUNAddr LUN; -}; +} __packed; struct RequestBlock { u8 CDBLen; @@ -365,18 +365,18 @@ struct RequestBlock { #define GET_DIR(tad) (((tad) >> 6) & 0x03) u16 Timeout; u8 CDB[16]; -}; +} __packed; struct ErrDescriptor { __le64 Addr; __le32 Len; -}; +} __packed; struct SGDescriptor { __le64 Addr; __le32 Len; __le32 Ext; -}; +} __packed; union MoreErrInfo { struct { @@ -390,7 +390,8 @@ union MoreErrInfo { u8 offense_num; /* byte # of offense 0-base */ u32 offense_value; } Invalid_Cmd; -}; +} __packed; + struct ErrorInfo { u8 ScsiStatus; u8 SenseLen; @@ -398,7 +399,7 @@ struct ErrorInfo { u32 ResidualCnt; union MoreErrInfo MoreErrInfo; u8 SenseInfo[SENSEINFOBYTES]; -}; +} __packed; /* Command types */ #define CMD_IOCTL_PEND 0x01 #define CMD_SCSI 0x03 @@ -451,7 +452,7 @@ struct CommandList { bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ -} __aligned(COMMANDLIST_ALIGNMENT); +} __packed __aligned(COMMANDLIST_ALIGNMENT); /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 @@ -489,7 +490,7 @@ struct io_accel1_cmd { __le64 host_addr; /* 0x70 - 0x77 */ u8 CISS_LUN[8]; /* 0x78 - 0x7F */ struct SGDescriptor SG[IOACCEL1_MAXSGENTRIES]; -} __aligned(IOACCEL1_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL1_COMMANDLIST_ALIGNMENT); #define IOACCEL1_FUNCTION_SCSIIO 0x00 #define IOACCEL1_SGLOFFSET 32 @@ -519,7 +520,7 @@ struct ioaccel2_sg_element { u8 chain_indicator; #define IOACCEL2_CHAIN 0x80 #define IOACCEL2_LAST_SG 0x40 -}; +} __packed; /* * SCSI Response Format structure for IO Accelerator Mode 2 @@ -559,7 +560,7 @@ struct io_accel2_scsi_response { u8 sense_data_len; /* sense/response data length */ u8 resid_cnt[4]; /* residual count */ u8 sense_data_buff[32]; /* sense/response data buffer */ -}; +} __packed; /* * Structure for I/O accelerator (mode 2 or m2) commands. @@ -592,7 +593,7 @@ struct io_accel2_cmd { __le32 tweak_upper; /* Encryption tweak, upper 4 bytes */ struct ioaccel2_sg_element sg[IOACCEL2_MAXSGENTRIES]; struct io_accel2_scsi_response error_data; -} __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); /* * defines for Mode 2 command struct @@ -618,7 +619,7 @@ struct hpsa_tmf_struct { __le64 abort_tag; /* cciss tag of SCSI cmd or TMF to abort */ __le64 error_ptr; /* Error Pointer */ __le32 error_len; /* Error Length */ -} __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); +} __packed __aligned(IOACCEL2_COMMANDLIST_ALIGNMENT); /* Configuration Table Structure */ struct HostWrite { @@ -626,7 +627,7 @@ struct HostWrite { __le32 command_pool_addr_hi; __le32 CoalIntDelay; __le32 CoalIntCount; -}; +} __packed; #define SIMPLE_MODE 0x02 #define PERFORMANT_MODE 0x04 @@ -675,7 +676,7 @@ struct CfgTable { #define HPSA_EVENT_NOTIFY_ACCEL_IO_PATH_STATE_CHANGE (1 << 30) #define HPSA_EVENT_NOTIFY_ACCEL_IO_PATH_CONFIG_CHANGE (1 << 31) __le32 clear_event_notify; -}; +} __packed; #define NUM_BLOCKFETCH_ENTRIES 8 struct TransTable_struct { @@ -686,14 +687,14 @@ struct TransTable_struct { __le32 RepQCtrAddrHigh32; #define MAX_REPLY_QUEUES 64 struct vals32 RepQAddr[MAX_REPLY_QUEUES]; -}; +} __packed; struct hpsa_pci_info { unsigned char bus; unsigned char dev_fn; unsigned short domain; u32 board_id; -}; +} __packed; struct bmic_identify_controller { u8 configured_logical_drive_count; /* offset 0 */ @@ -702,7 +703,7 @@ struct bmic_identify_controller { u8 pad2[136]; u8 controller_mode; /* offset 292 */ u8 pad3[32]; -}; +} __packed; struct bmic_identify_physical_device { @@ -845,7 +846,7 @@ struct bmic_identify_physical_device { u8 max_link_rate[256]; u8 neg_phys_link_rate[256]; u8 box_conn_name[8]; -} __attribute((aligned(512))); +} __packed __attribute((aligned(512))); struct bmic_sense_subsystem_info { u8 primary_slot_number; @@ -858,7 +859,7 @@ struct bmic_sense_subsystem_info { u8 secondary_array_serial_number[32]; u8 secondary_cache_serial_number[32]; u8 pad[332]; -}; +} __packed; struct bmic_sense_storage_box_params { u8 reserved[36]; @@ -870,7 +871,6 @@ struct bmic_sense_storage_box_params { u8 reserver_3[84]; u8 phys_connector[2]; u8 reserved_4[296]; -}; +} __packed; -#pragma pack() #endif /* HPSA_CMD_H */ -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH v2 2/3] hpsa: fix boot on ia64 (atomic_t alignment) 2021-03-30 7:19 ` Sergei Trofimovich @ 2021-03-30 7:19 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change removes packing annotation from struct not intended to be sent to controller as is. This restores natural `atomic_t` alignment. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: linux-kernel@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 280e933d27e7..885b1f1fb20a 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -452,7 +452,7 @@ struct CommandList { bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ -} __packed __aligned(COMMANDLIST_ALIGNMENT); +} __aligned(COMMANDLIST_ALIGNMENT); /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH v2 2/3] hpsa: fix boot on ia64 (atomic_t alignment) @ 2021-03-30 7:19 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich The failure initially observed as boot failure on rx3600 ia64 machine with RAID bus controller: Hewlett-Packard Company Smart Array P600: kernel unaligned access to 0xe000000105dd8b95, ip=0xa000000100b87551 kernel unaligned access to 0xe000000105dd8e95, ip=0xa000000100b87551 hpsa 0000:14:01.0: Controller reports max supported commands of 0 Using 16 instead. Ensure that firmware is up to date. swapper/0[1]: error during unaligned kernel access Here unaligned access comes from 'struct CommandList' that happens to be packed. The change f749d8b7a ("scsi: hpsa: Correct dev cmds outstanding for retried cmds") introduced unexpected padding and un-aligned atomic_t from natural alignment to something else. This change removes packing annotation from struct not intended to be sent to controller as is. This restores natural `atomic_t` alignment. The change is tested on the same rx3600 machine. CC: linux-ia64@vger.kernel.org CC: linux-kernel@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 280e933d27e7..885b1f1fb20a 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -452,7 +452,7 @@ struct CommandList { bool retry_pending; struct hpsa_scsi_dev_t *device; atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ -} __packed __aligned(COMMANDLIST_ALIGNMENT); +} __aligned(COMMANDLIST_ALIGNMENT); /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction 2021-03-30 7:19 ` Sergei Trofimovich @ 2021-03-30 7:19 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 885b1f1fb20a..ba6a3aa8d954 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -22,6 +22,9 @@ #include <linux/compiler.h> +#include <linux/build_bug.h> /* static_assert */ +#include <linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -454,6 +457,15 @@ struct CommandList { atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * The assert guards against reintroductin against unwanted __packed to + * the struct CommandList. + */ +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) == 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction @ 2021-03-30 7:19 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-30 7:19 UTC (permalink / raw) To: Martin K. Petersen, Arnd Bergmann, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl Cc: linux-kernel, Sergei Trofimovich CC: linux-ia64@vger.kernel.org CC: storagedev@microchip.com CC: linux-scsi@vger.kernel.org CC: Joe Szczypek <jszczype@redhat.com> CC: Scott Benesh <scott.benesh@microchip.com> CC: Scott Teel <scott.teel@microchip.com> CC: Tomas Henzl <thenzl@redhat.com> CC: "Martin K. Petersen" <martin.petersen@oracle.com> CC: Don Brace <don.brace@microchip.com> Reported-by: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Suggested-by: Don Brace <don.brace@microchip.com> Fixes: f749d8b7a "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org> --- drivers/scsi/hpsa_cmd.h | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/scsi/hpsa_cmd.h b/drivers/scsi/hpsa_cmd.h index 885b1f1fb20a..ba6a3aa8d954 100644 --- a/drivers/scsi/hpsa_cmd.h +++ b/drivers/scsi/hpsa_cmd.h @@ -22,6 +22,9 @@ #include <linux/compiler.h> +#include <linux/build_bug.h> /* static_assert */ +#include <linux/stddef.h> /* offsetof */ + /* general boundary defintions */ #define SENSEINFOBYTES 32 /* may vary between hbas */ #define SG_ENTRIES_IN_CMD 32 /* Max SG entries excluding chain blocks */ @@ -454,6 +457,15 @@ struct CommandList { atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ } __aligned(COMMANDLIST_ALIGNMENT); +/* + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic + * operations on architectures that don't support unaligned atomics like IA64. + * + * The assert guards against reintroductin against unwanted __packed to + * the struct CommandList. + */ +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) = 0); + /* Max S/G elements in I/O accelerator command */ #define IOACCEL1_MAXSGENTRIES 24 #define IOACCEL2_MAXSGENTRIES 28 -- 2.31.1 ^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction 2021-03-30 7:19 ` Sergei Trofimovich @ 2021-03-30 7:34 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:34 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:23 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * The assert guards against reintroductin against unwanted __packed to > + * the struct CommandList. > + */ > +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) == 0); > + There are a few other members that need to be aligned: the work_struct has another atomic_t inside it, and there are a few pointers that might rely on being written to atomically. While you could add a static_assert for each member, the easier solution is to just not ask for the members to be misaligned in the first place. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction @ 2021-03-30 7:34 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:34 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:23 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > +/* > + * Make sure our embedded atomic variable is aligned. Otherwise we break atomic > + * operations on architectures that don't support unaligned atomics like IA64. > + * > + * The assert guards against reintroductin against unwanted __packed to > + * the struct CommandList. > + */ > +static_assert(offsetof(struct CommandList, refcount) % __alignof__(atomic_t) = 0); > + There are a few other members that need to be aligned: the work_struct has another atomic_t inside it, and there are a few pointers that might rely on being written to atomically. While you could add a static_assert for each member, the easier solution is to just not ask for the members to be misaligned in the first place. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction 2021-03-30 7:34 ` Arnd Bergmann @ 2021-04-02 14:40 ` Elliott, Robert (Servers) -1 siblings, 0 replies; 89+ messages in thread From: Elliott, Robert (Servers) @ 2021-04-02 14:40 UTC (permalink / raw) To: Arnd Bergmann, Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace - C33706, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List It looks like ia64 implements atomic_t as a 64-bit value and expects atomic_t to be 64-bit aligned, but does nothing to ensure that. For x86, atomic_t is a 32-bit value and atomic64_t is a 64-bit value, and the definition of atomic64_t is overridden in a way that ensures 64-bit (8 byte) alignment: Generic definitions are in include/linux/types.h: typedef struct { int counter; } atomic_t; #define ATOMIC_INIT(i) { (i) } #ifdef CONFIG_64BIT typedef struct { s64 counter; } atomic64_t; #endif Override in arch/x86/include/asm/atomic64_32.h: typedef struct { s64 __aligned(8) counter; } atomic64_t; Perhaps ia64 needs to take over the definition of both atomic_t and atomic64_t and do the same? ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction @ 2021-04-02 14:40 ` Elliott, Robert (Servers) 0 siblings, 0 replies; 89+ messages in thread From: Elliott, Robert (Servers) @ 2021-04-02 14:40 UTC (permalink / raw) To: Arnd Bergmann, Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace - C33706, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List SXQgbG9va3MgbGlrZSBpYTY0IGltcGxlbWVudHMgYXRvbWljX3QgYXMgYSA2NC1iaXQgdmFsdWUg YW5kIGV4cGVjdHMgYXRvbWljX3QNCnRvIGJlIDY0LWJpdCBhbGlnbmVkLCBidXQgZG9lcyBub3Ro aW5nIHRvIGVuc3VyZSB0aGF0Lg0KDQpGb3IgeDg2LCBhdG9taWNfdCBpcyBhIDMyLWJpdCB2YWx1 ZSBhbmQgYXRvbWljNjRfdCBpcyBhIDY0LWJpdCB2YWx1ZSwgYW5kDQp0aGUgZGVmaW5pdGlvbiBv ZiBhdG9taWM2NF90IGlzIG92ZXJyaWRkZW4gaW4gYSB3YXkgdGhhdCBlbnN1cmVzDQo2NC1iaXQg KDggYnl0ZSkgYWxpZ25tZW50Og0KDQpHZW5lcmljIGRlZmluaXRpb25zIGFyZSBpbiBpbmNsdWRl L2xpbnV4L3R5cGVzLmg6DQogICAgdHlwZWRlZiBzdHJ1Y3Qgew0KICAgICAgICAgICAgaW50IGNv dW50ZXI7DQogICAgfSBhdG9taWNfdDsNCg0KICAgICNkZWZpbmUgQVRPTUlDX0lOSVQoaSkgeyAo aSkgfQ0KDQogICAgI2lmZGVmIENPTkZJR182NEJJVA0KICAgIHR5cGVkZWYgc3RydWN0IHsNCiAg ICAgICAgICAgIHM2NCBjb3VudGVyOw0KICAgIH0gYXRvbWljNjRfdDsNCiAgICAjZW5kaWYNCg0K T3ZlcnJpZGUgaW4gYXJjaC94ODYvaW5jbHVkZS9hc20vYXRvbWljNjRfMzIuaDoNCiAgICB0eXBl ZGVmIHN0cnVjdCB7DQogICAgICAgICAgICBzNjQgX19hbGlnbmVkKDgpIGNvdW50ZXI7DQogICAg fSBhdG9taWM2NF90Ow0KDQpQZXJoYXBzIGlhNjQgbmVlZHMgdG8gdGFrZSBvdmVyIHRoZSBkZWZp bml0aW9uIG9mIGJvdGggYXRvbWljX3QgYW5kIGF0b21pYzY0X3QNCmFuZCBkbyB0aGUgc2FtZT8N Cg0K ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction 2021-04-02 14:40 ` Elliott, Robert (Servers) @ 2021-04-03 14:51 ` Sergei Trofimovich -1 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-04-03 14:51 UTC (permalink / raw) To: Elliott, Robert (Servers) Cc: Arnd Bergmann, Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace - C33706, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Fri, 2 Apr 2021 14:40:39 +0000 "Elliott, Robert (Servers)" <elliott@hpe.com> wrote: > It looks like ia64 implements atomic_t as a 64-bit value and expects atomic_t > to be 64-bit aligned, but does nothing to ensure that. > > For x86, atomic_t is a 32-bit value and atomic64_t is a 64-bit value, and > the definition of atomic64_t is overridden in a way that ensures > 64-bit (8 byte) alignment: > > Generic definitions are in include/linux/types.h: > typedef struct { > int counter; > } atomic_t; > > #define ATOMIC_INIT(i) { (i) } > > #ifdef CONFIG_64BIT > typedef struct { > s64 counter; > } atomic64_t; > #endif > > Override in arch/x86/include/asm/atomic64_32.h: > typedef struct { > s64 __aligned(8) counter; > } atomic64_t; > > Perhaps ia64 needs to take over the definition of both atomic_t and atomic64_t > and do the same? I don't think it's needed. ia64 is a 64-bit arch with expected natural alignment for s64: alignof(s64)=8. Also if my understanding is correct adding __aligned(8) would not fix use case of embedding locks into packed structs even on x86_64 (or i386): $ cat a.c #include <stdio.h> #include <stddef.h> typedef struct { unsigned long long __attribute__((aligned(8))) l; } lock_t; struct s { char c; lock_t lock; } __attribute__((packed)); int main() { printf ("offsetof(struct s, lock) = %lu\nsizeof(struct s) = %lu\n", offsetof(struct s, lock), sizeof(struct s)); } $ x86_64-pc-linux-gnu-gcc a.c -o a && ./a offsetof(struct s, lock) = 1 sizeof(struct s) = 9 $ x86_64-pc-linux-gnu-gcc a.c -o a -m32 && ./a offsetof(struct s, lock) = 1 sizeof(struct s) = 9 Note how alignment of 'lock' stays 1 byte in both cases. 8-byte alignment added for i386 in https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bbf2a330d92c5afccfd17592ba9ccd50f41cf748 is only as a performance optimization (not a correctness fix). Larger alignment on i386 is preferred because alignof(s64)=4 on that target which might make atomic op span cache lines that leads to performance degradation. -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction @ 2021-04-03 14:51 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-04-03 14:51 UTC (permalink / raw) To: Elliott, Robert (Servers) Cc: Arnd Bergmann, Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace - C33706, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Fri, 2 Apr 2021 14:40:39 +0000 "Elliott, Robert (Servers)" <elliott@hpe.com> wrote: > It looks like ia64 implements atomic_t as a 64-bit value and expects atomic_t > to be 64-bit aligned, but does nothing to ensure that. > > For x86, atomic_t is a 32-bit value and atomic64_t is a 64-bit value, and > the definition of atomic64_t is overridden in a way that ensures > 64-bit (8 byte) alignment: > > Generic definitions are in include/linux/types.h: > typedef struct { > int counter; > } atomic_t; > > #define ATOMIC_INIT(i) { (i) } > > #ifdef CONFIG_64BIT > typedef struct { > s64 counter; > } atomic64_t; > #endif > > Override in arch/x86/include/asm/atomic64_32.h: > typedef struct { > s64 __aligned(8) counter; > } atomic64_t; > > Perhaps ia64 needs to take over the definition of both atomic_t and atomic64_t > and do the same? I don't think it's needed. ia64 is a 64-bit arch with expected natural alignment for s64: alignof(s64)=8. Also if my understanding is correct adding __aligned(8) would not fix use case of embedding locks into packed structs even on x86_64 (or i386): $ cat a.c #include <stdio.h> #include <stddef.h> typedef struct { unsigned long long __attribute__((aligned(8))) l; } lock_t; struct s { char c; lock_t lock; } __attribute__((packed)); int main() { printf ("offsetof(struct s, lock) = %lu\nsizeof(struct s) = %lu\n", offsetof(struct s, lock), sizeof(struct s)); } $ x86_64-pc-linux-gnu-gcc a.c -o a && ./a offsetof(struct s, lock) = 1 sizeof(struct s) = 9 $ x86_64-pc-linux-gnu-gcc a.c -o a -m32 && ./a offsetof(struct s, lock) = 1 sizeof(struct s) = 9 Note how alignment of 'lock' stays 1 byte in both cases. 8-byte alignment added for i386 in https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id»f2a330d92c5afccfd17592ba9ccd50f41cf748 is only as a performance optimization (not a correctness fix). Larger alignment on i386 is preferred because alignof(s64)=4 on that target which might make atomic op span cache lines that leads to performance degradation. -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide 2021-03-30 7:19 ` Sergei Trofimovich @ 2021-03-30 7:30 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:30 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:22 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > > Some of the structs contain `atomic_t` values and are not intended to be > sent to IO controller as is. > > The change adds __packed to every struct and union in the file. > Follow-up commits will fix `atomic_t` problems. > > The commit is a no-op at least on ia64: > $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) This looks better to me, but I think it still has the same potential bug in the CommandList definition. Moving from #pragma to annotating the misaligned structures as __packed is more of a cleanup that could be done separately from the bugfix, but it does make it a little more robust. > #define HPSA_INQUIRY 0x12 > struct InquiryData { > u8 data_byte[36]; > -}; > +} __packed; Marking this one and a few others as __packed is a bit silly, but also obviously harmless, and closer to the original version, so that's ok. > @@ -451,7 +452,7 @@ struct CommandList { > bool retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > -} __aligned(COMMANDLIST_ALIGNMENT); > +} __packed __aligned(COMMANDLIST_ALIGNMENT); You are still marking CommandList as __packed here, which is what caused the original problem. Please don't mark this one as __packed at all. If there are individual members that you want to be misaligned inside of the structure, you could mark those explicitly. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide @ 2021-03-30 7:30 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:30 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:22 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > > Some of the structs contain `atomic_t` values and are not intended to be > sent to IO controller as is. > > The change adds __packed to every struct and union in the file. > Follow-up commits will fix `atomic_t` problems. > > The commit is a no-op at least on ia64: > $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) This looks better to me, but I think it still has the same potential bug in the CommandList definition. Moving from #pragma to annotating the misaligned structures as __packed is more of a cleanup that could be done separately from the bugfix, but it does make it a little more robust. > #define HPSA_INQUIRY 0x12 > struct InquiryData { > u8 data_byte[36]; > -}; > +} __packed; Marking this one and a few others as __packed is a bit silly, but also obviously harmless, and closer to the original version, so that's ok. > @@ -451,7 +452,7 @@ struct CommandList { > bool retry_pending; > struct hpsa_scsi_dev_t *device; > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > -} __aligned(COMMANDLIST_ALIGNMENT); > +} __packed __aligned(COMMANDLIST_ALIGNMENT); You are still marking CommandList as __packed here, which is what caused the original problem. Please don't mark this one as __packed at all. If there are individual members that you want to be misaligned inside of the structure, you could mark those explicitly. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide 2021-03-30 7:30 ` Arnd Bergmann @ 2021-03-30 7:43 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:43 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:30 AM Arnd Bergmann <arnd@kernel.org> wrote: > On Tue, Mar 30, 2021 at 9:22 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > > > @@ -451,7 +452,7 @@ struct CommandList { > > bool retry_pending; > > struct hpsa_scsi_dev_t *device; > > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > > -} __aligned(COMMANDLIST_ALIGNMENT); > > +} __packed __aligned(COMMANDLIST_ALIGNMENT); > > You are still marking CommandList as __packed here, which is > what caused the original problem. Please don't mark this one > as __packed at all. If there are individual members that you want > to be misaligned inside of the structure, you could mark those > explicitly. Nevermind, I just got patch 2/3, splitting up the patches like this seems fine to me. Whole series Reviewed-by: Arnd Bergmann <arnd@arndb.de> ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide @ 2021-03-30 7:43 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-30 7:43 UTC (permalink / raw) To: Sergei Trofimovich Cc: Martin K. Petersen, John Paul Adrian Glaubitz, Don Brace, linux-ia64, storagedev, linux-scsi, jszczype, Scott Benesh, Scott Teel, thenzl, Linux Kernel Mailing List On Tue, Mar 30, 2021 at 9:30 AM Arnd Bergmann <arnd@kernel.org> wrote: > On Tue, Mar 30, 2021 at 9:22 AM Sergei Trofimovich <slyfox@gentoo.org> wrote: > > > @@ -451,7 +452,7 @@ struct CommandList { > > bool retry_pending; > > struct hpsa_scsi_dev_t *device; > > atomic_t refcount; /* Must be last to avoid memset in hpsa_cmd_init() */ > > -} __aligned(COMMANDLIST_ALIGNMENT); > > +} __packed __aligned(COMMANDLIST_ALIGNMENT); > > You are still marking CommandList as __packed here, which is > what caused the original problem. Please don't mark this one > as __packed at all. If there are individual members that you want > to be misaligned inside of the structure, you could mark those > explicitly. Nevermind, I just got patch 2/3, splitting up the patches like this seems fine to me. Whole series Reviewed-by: Arnd Bergmann <arnd@arndb.de> ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide 2021-03-30 7:19 ` Sergei Trofimovich @ 2021-04-02 3:54 ` Martin K. Petersen -1 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-04-02 3:54 UTC (permalink / raw) To: Scott Teel, jszczype, storagedev, Don Brace, Sergei Trofimovich, John Paul Adrian Glaubitz, Arnd Bergmann, Scott Benesh, thenzl, linux-ia64, linux-scsi Cc: Martin K . Petersen, linux-kernel On Tue, 30 Mar 2021 08:19:56 +0100, Sergei Trofimovich wrote: > Some of the structs contain `atomic_t` values and are not intended to be > sent to IO controller as is. > > The change adds __packed to every struct and union in the file. > Follow-up commits will fix `atomic_t` problems. > > The commit is a no-op at least on ia64: > $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) Applied to 5.12/scsi-fixes, thanks! [1/3] hpsa: use __packed on individual structs, not header-wide https://git.kernel.org/mkp/scsi/c/5482a9a1a8fd [2/3] hpsa: fix boot on ia64 (atomic_t alignment) https://git.kernel.org/mkp/scsi/c/02ec144292bc [3/3] hpsa: add an assert to prevent from __packed reintroduction https://git.kernel.org/mkp/scsi/c/e01a00ff62ad -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide @ 2021-04-02 3:54 ` Martin K. Petersen 0 siblings, 0 replies; 89+ messages in thread From: Martin K. Petersen @ 2021-04-02 3:54 UTC (permalink / raw) To: Scott Teel, jszczype, storagedev, Don Brace, Sergei Trofimovich, John Paul Adrian Glaubitz, Arnd Bergmann, Scott Benesh, thenzl, linux-ia64, linux-scsi Cc: Martin K . Petersen, linux-kernel On Tue, 30 Mar 2021 08:19:56 +0100, Sergei Trofimovich wrote: > Some of the structs contain `atomic_t` values and are not intended to be > sent to IO controller as is. > > The change adds __packed to every struct and union in the file. > Follow-up commits will fix `atomic_t` problems. > > The commit is a no-op at least on ia64: > $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) Applied to 5.12/scsi-fixes, thanks! [1/3] hpsa: use __packed on individual structs, not header-wide https://git.kernel.org/mkp/scsi/c/5482a9a1a8fd [2/3] hpsa: fix boot on ia64 (atomic_t alignment) https://git.kernel.org/mkp/scsi/c/02ec144292bc [3/3] hpsa: add an assert to prevent from __packed reintroduction https://git.kernel.org/mkp/scsi/c/e01a00ff62ad -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide 2021-04-02 3:54 ` Martin K. Petersen @ 2021-04-15 18:41 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-04-15 18:41 UTC (permalink / raw) To: martin.petersen, Scott.Teel, jszczype, storagedev, slyfox, glaubitz, arnd, Scott.Benesh, thenzl, linux-ia64, linux-scsi Cc: linux-kernel -----Original Message----- From: Martin K. Petersen Subject: Re: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide > Some of the structs contain `atomic_t` values and are not intended to > be sent to IO controller as is. > > The change adds __packed to every struct and union in the file. > Follow-up commits will fix `atomic_t` problems. > > The commit is a no-op at least on ia64: > $ diff -u <(objdump -d -r old.o) <(objdump -d -r new.o) Applied to 5.12/scsi-fixes, thanks! Don: Thank-you for all of your efforts. I appreciate to work that you have done on these patches. Thanks, Don Brace [1/3] hpsa: use __packed on individual structs, not header-wide https://git.kernel.org/mkp/scsi/c/5482a9a1a8fd [2/3] hpsa: fix boot on ia64 (atomic_t alignment) https://git.kernel.org/mkp/scsi/c/02ec144292bc [3/3] hpsa: add an assert to prevent from __packed reintroduction https://git.kernel.org/mkp/scsi/c/e01a00ff62ad -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide @ 2021-04-15 18:41 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-04-15 18:41 UTC (permalink / raw) To: martin.petersen, Scott.Teel, jszczype, storagedev, slyfox, glaubitz, arnd, Scott.Benesh, thenzl, linux-ia64, linux-scsi Cc: linux-kernel LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE1hcnRpbiBLLiBQZXRlcnNlbiBTdWJq ZWN0OiBSZTogW1BBVENIIHYyIDEvM10gaHBzYTogdXNlIF9fcGFja2VkIG9uIGluZGl2aWR1YWwg c3RydWN0cywgbm90IGhlYWRlci13aWRlDQoNCj4gU29tZSBvZiB0aGUgc3RydWN0cyBjb250YWlu IGBhdG9taWNfdGAgdmFsdWVzIGFuZCBhcmUgbm90IGludGVuZGVkIHRvIA0KPiBiZSBzZW50IHRv IElPIGNvbnRyb2xsZXIgYXMgaXMuDQo+DQo+IFRoZSBjaGFuZ2UgYWRkcyBfX3BhY2tlZCB0byBl dmVyeSBzdHJ1Y3QgYW5kIHVuaW9uIGluIHRoZSBmaWxlLg0KPiBGb2xsb3ctdXAgY29tbWl0cyB3 aWxsIGZpeCBgYXRvbWljX3RgIHByb2JsZW1zLg0KPg0KPiBUaGUgY29tbWl0IGlzIGEgbm8tb3Ag YXQgbGVhc3Qgb24gaWE2NDoNCj4gICAgICQgZGlmZiAtdSA8KG9iamR1bXAgLWQgLXIgb2xkLm8p IDwob2JqZHVtcCAtZCAtciBuZXcubykNCg0KQXBwbGllZCB0byA1LjEyL3Njc2ktZml4ZXMsIHRo YW5rcyENCg0KRG9uOiBUaGFuay15b3UgZm9yIGFsbCBvZiB5b3VyIGVmZm9ydHMuIEkgYXBwcmVj aWF0ZSB0byB3b3JrIHRoYXQgeW91IGhhdmUgZG9uZSBvbiB0aGVzZSBwYXRjaGVzLg0KDQpUaGFu a3MsDQpEb24gQnJhY2UNCg0KWzEvM10gaHBzYTogdXNlIF9fcGFja2VkIG9uIGluZGl2aWR1YWwg c3RydWN0cywgbm90IGhlYWRlci13aWRlDQogICAgICBodHRwczovL2dpdC5rZXJuZWwub3JnL21r cC9zY3NpL2MvNTQ4MmE5YTFhOGZkDQpbMi8zXSBocHNhOiBmaXggYm9vdCBvbiBpYTY0IChhdG9t aWNfdCBhbGlnbm1lbnQpDQogICAgICBodHRwczovL2dpdC5rZXJuZWwub3JnL21rcC9zY3NpL2Mv MDJlYzE0NDI5MmJjDQpbMy8zXSBocHNhOiBhZGQgYW4gYXNzZXJ0IHRvIHByZXZlbnQgZnJvbSBf X3BhY2tlZCByZWludHJvZHVjdGlvbg0KICAgICAgaHR0cHM6Ly9naXQua2VybmVsLm9yZy9ta3Av c2NzaS9jL2UwMWEwMGZmNjJhZA0KDQotLQ0KTWFydGluIEsuIFBldGVyc2VuICAgICAgT3JhY2xl IExpbnV4IEVuZ2luZWVyaW5nDQo ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-03 17:33 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace @ 2021-03-05 9:22 ` Geert Uytterhoeven -1 siblings, 0 replies; 89+ messages in thread From: Geert Uytterhoeven @ 2021-03-05 9:22 UTC (permalink / raw) To: Don.Brace Cc: slyich, John Paul Adrian Glaubitz, storagedev, scsi, linux-ia64, Linux Kernel Mailing List, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen Hi Don, On Fri, Mar 5, 2021 at 12:26 AM <Don.Brace@microchip.com> wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyich@gmail.com] > Sent: Wednesday, March 3, 2021 2:56 AM > To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com>; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org > Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> > Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 > > EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe > > On Wed, 3 Mar 2021 00:22:36 +0000 > Sergei Trofimovich <slyich@gmail.com> wrote: > > > On Tue, 2 Mar 2021 23:31:32 +0100 > > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > > > Hi Sergei! > > > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > > hpsa queue initialization, but my failure is later: > > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > > Maybe I get different error because I flipped on most debugging > > > > kernel options :) > > > > > > > > Looks like 'ERROR: Invalid distance value range' while being very > > > > scary are harmless. It's just a new spammy way for kernel to > > > > report lack of NUMA config on the machine (no SRAT and SLIT ACPI > > > > tables). > > > > > > > > At least I get hpsa detected on PCI bus. But I guess it's > > > > discovered configuration is very wrong as I get unaligned accesses: > > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > Running pahole before the patch: > > struct CommandList { > struct CommandListHeader Header; /* 0 20 */ > struct RequestBlock Request; /* 20 20 */ > struct ErrDescriptor ErrDesc; /* 40 12 */ > struct SGDescriptor SG[32]; /* 52 512 */ > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > u32 busaddr; /* 564 4 */ > struct ErrorInfo * err_info; /* 568 8 */ > /* --- cacheline 9 boundary (576 bytes) --- */ > struct ctlr_info * h; /* 576 8 */ > int cmd_type; /* 584 4 */ > long int cmdindex; /* 588 8 */ > struct completion * waiting; /* 596 8 */ > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > struct work_struct work; /* 612 32 */ > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > int abort_pending; /* 652 4 */ > struct hpsa_scsi_dev_t * device; /* 656 8 */ > atomic_t refcount; /* 664 4 */ > > /* size: 768, cachelines: 12, members: 16 */ > /* padding: 100 */ > } __attribute__((__aligned__(128))); > > Pahole after the patch: > > struct CommandList { > struct CommandListHeader Header; /* 0 20 */ > struct RequestBlock Request; /* 20 20 */ > struct ErrDescriptor ErrDesc; /* 40 12 */ > struct SGDescriptor SG[32]; /* 52 512 */ > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > u32 busaddr; /* 564 4 */ > struct ErrorInfo * err_info; /* 568 8 */ > /* --- cacheline 9 boundary (576 bytes) --- */ > struct ctlr_info * h; /* 576 8 */ > int cmd_type; /* 584 4 */ > long int cmdindex; /* 588 8 */ > struct completion * waiting; /* 596 8 */ > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > struct work_struct work; /* 612 32 */ > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > struct hpsa_scsi_dev_t * device; /* 652 8 */ > bool retry_pending; /* 660 1 */ > atomic_t refcount; /* 661 4 */ How come this atomic_t is no longer aligned to its natural alignment? > > /* size: 768, cachelines: 12, members: 16 */ > /* padding: 103 */ > } __attribute__((__aligned__(128))); Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-05 9:22 ` Geert Uytterhoeven 0 siblings, 0 replies; 89+ messages in thread From: Geert Uytterhoeven @ 2021-03-05 9:22 UTC (permalink / raw) To: Don.Brace Cc: slyich, John Paul Adrian Glaubitz, storagedev, scsi, linux-ia64, Linux Kernel Mailing List, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen Hi Don, On Fri, Mar 5, 2021 at 12:26 AM <Don.Brace@microchip.com> wrote: > -----Original Message----- > From: Sergei Trofimovich [mailto:slyich@gmail.com] > Sent: Wednesday, March 3, 2021 2:56 AM > To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com>; storagedev <storagedev@microchip.com>; linux-scsi@vger.kernel.org > Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> > Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 > > EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe > > On Wed, 3 Mar 2021 00:22:36 +0000 > Sergei Trofimovich <slyich@gmail.com> wrote: > > > On Tue, 2 Mar 2021 23:31:32 +0100 > > John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > > > > > Hi Sergei! > > > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > > > Gave v5.12-rc1 a try today and got a similar boot failure around > > > > hpsa queue initialization, but my failure is later: > > > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > > > Maybe I get different error because I flipped on most debugging > > > > kernel options :) > > > > > > > > Looks like 'ERROR: Invalid distance value range' while being very > > > > scary are harmless. It's just a new spammy way for kernel to > > > > report lack of NUMA config on the machine (no SRAT and SLIT ACPI > > > > tables). > > > > > > > > At least I get hpsa detected on PCI bus. But I guess it's > > > > discovered configuration is very wrong as I get unaligned accesses: > > > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > Running pahole before the patch: > > struct CommandList { > struct CommandListHeader Header; /* 0 20 */ > struct RequestBlock Request; /* 20 20 */ > struct ErrDescriptor ErrDesc; /* 40 12 */ > struct SGDescriptor SG[32]; /* 52 512 */ > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > u32 busaddr; /* 564 4 */ > struct ErrorInfo * err_info; /* 568 8 */ > /* --- cacheline 9 boundary (576 bytes) --- */ > struct ctlr_info * h; /* 576 8 */ > int cmd_type; /* 584 4 */ > long int cmdindex; /* 588 8 */ > struct completion * waiting; /* 596 8 */ > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > struct work_struct work; /* 612 32 */ > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > int abort_pending; /* 652 4 */ > struct hpsa_scsi_dev_t * device; /* 656 8 */ > atomic_t refcount; /* 664 4 */ > > /* size: 768, cachelines: 12, members: 16 */ > /* padding: 100 */ > } __attribute__((__aligned__(128))); > > Pahole after the patch: > > struct CommandList { > struct CommandListHeader Header; /* 0 20 */ > struct RequestBlock Request; /* 20 20 */ > struct ErrDescriptor ErrDesc; /* 40 12 */ > struct SGDescriptor SG[32]; /* 52 512 */ > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > u32 busaddr; /* 564 4 */ > struct ErrorInfo * err_info; /* 568 8 */ > /* --- cacheline 9 boundary (576 bytes) --- */ > struct ctlr_info * h; /* 576 8 */ > int cmd_type; /* 584 4 */ > long int cmdindex; /* 588 8 */ > struct completion * waiting; /* 596 8 */ > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > struct work_struct work; /* 612 32 */ > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > struct hpsa_scsi_dev_t * device; /* 652 8 */ > bool retry_pending; /* 660 1 */ > atomic_t refcount; /* 661 4 */ How come this atomic_t is no longer aligned to its natural alignment? > > /* size: 768, cachelines: 12, members: 16 */ > /* padding: 103 */ > } __attribute__((__aligned__(128))); Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-05 9:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Geert Uytterhoeven @ 2021-03-05 13:31 ` Arnd Bergmann -1 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-05 13:31 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Don.Brace, slyich, John Paul Adrian Glaubitz, storagedev, scsi, linux-ia64, Linux Kernel Mailing List, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen On Fri, Mar 5, 2021 at 10:24 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > On Fri, Mar 5, 2021 at 12:26 AM <Don.Brace@microchip.com> wrote: > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > struct CommandList { > > struct CommandListHeader Header; /* 0 20 */ > > struct RequestBlock Request; /* 20 20 */ > > struct ErrDescriptor ErrDesc; /* 40 12 */ > > struct SGDescriptor SG[32]; /* 52 512 */ > > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > > u32 busaddr; /* 564 4 */ > > struct ErrorInfo * err_info; /* 568 8 */ > > /* --- cacheline 9 boundary (576 bytes) --- */ > > struct ctlr_info * h; /* 576 8 */ > > int cmd_type; /* 584 4 */ > > long int cmdindex; /* 588 8 */ > > struct completion * waiting; /* 596 8 */ > > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > > struct work_struct work; /* 612 32 */ > > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > > struct hpsa_scsi_dev_t * device; /* 652 8 */ > > bool retry_pending; /* 660 1 */ > > atomic_t refcount; /* 661 4 */ > > How come this atomic_t is no longer aligned to its natural alignment? There is a #pragma pack(1) in linux 203 of this file! It looks like some of the members in struct raid_map_data and struct CommandListHeader need to be annotated as packed, but the file accidentally packs everything until the '#pragma pack()' in line 875, including the kernel-side CommandList data structure that clearly must not be packed. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-05 13:31 ` Arnd Bergmann 0 siblings, 0 replies; 89+ messages in thread From: Arnd Bergmann @ 2021-03-05 13:31 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Don.Brace, slyich, John Paul Adrian Glaubitz, storagedev, scsi, linux-ia64, Linux Kernel Mailing List, jszczype, Scott.Benesh, Scott.Teel, thenzl, Martin K. Petersen On Fri, Mar 5, 2021 at 10:24 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > On Fri, Mar 5, 2021 at 12:26 AM <Don.Brace@microchip.com> wrote: > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > struct CommandList { > > struct CommandListHeader Header; /* 0 20 */ > > struct RequestBlock Request; /* 20 20 */ > > struct ErrDescriptor ErrDesc; /* 40 12 */ > > struct SGDescriptor SG[32]; /* 52 512 */ > > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > > u32 busaddr; /* 564 4 */ > > struct ErrorInfo * err_info; /* 568 8 */ > > /* --- cacheline 9 boundary (576 bytes) --- */ > > struct ctlr_info * h; /* 576 8 */ > > int cmd_type; /* 584 4 */ > > long int cmdindex; /* 588 8 */ > > struct completion * waiting; /* 596 8 */ > > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > > struct work_struct work; /* 612 32 */ > > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > > struct hpsa_scsi_dev_t * device; /* 652 8 */ > > bool retry_pending; /* 660 1 */ > > atomic_t refcount; /* 661 4 */ > > How come this atomic_t is no longer aligned to its natural alignment? There is a #pragma pack(1) in linux 203 of this file! It looks like some of the members in struct raid_map_data and struct CommandListHeader need to be annotated as packed, but the file accidentally packs everything until the '#pragma pack()' in line 875, including the kernel-side CommandList data structure that clearly must not be packed. Arnd ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-05 13:31 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Arnd Bergmann @ 2021-03-05 20:45 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-05 20:45 UTC (permalink / raw) To: arnd, geert Cc: slyich, glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen -----Original Message----- From: Arnd Bergmann [mailto:arnd@kernel.org] Sent: Friday, March 5, 2021 7:32 AM Subject: Re: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 On Fri, Mar 5, 2021 at 10:24 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > On Fri, Mar 5, 2021 at 12:26 AM <Don.Brace@microchip.com> wrote: > > > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > struct CommandList { > > struct CommandListHeader Header; /* 0 20 */ > > struct RequestBlock Request; /* 20 20 */ > > struct ErrDescriptor ErrDesc; /* 40 12 */ > > struct SGDescriptor SG[32]; /* 52 512 */ > > /* --- cacheline 8 boundary (512 bytes) was 52 bytes ago --- */ > > u32 busaddr; /* 564 4 */ > > struct ErrorInfo * err_info; /* 568 8 */ > > /* --- cacheline 9 boundary (576 bytes) --- */ > > struct ctlr_info * h; /* 576 8 */ > > int cmd_type; /* 584 4 */ > > long int cmdindex; /* 588 8 */ > > struct completion * waiting; /* 596 8 */ > > struct scsi_cmnd * scsi_cmd; /* 604 8 */ > > struct work_struct work; /* 612 32 */ > > /* --- cacheline 10 boundary (640 bytes) was 4 bytes ago --- */ > > struct hpsa_scsi_dev_t * phys_disk; /* 644 8 */ > > struct hpsa_scsi_dev_t * device; /* 652 8 */ > > bool retry_pending; /* 660 1 */ > > atomic_t refcount; /* 661 4 */ > > How come this atomic_t is no longer aligned to its natural alignment? There is a #pragma pack(1) in linux 203 of this file! It looks like some of the members in struct raid_map_data and struct CommandListHeader need to be annotated as packed, but the file accidentally packs everything until the '#pragma pack()' in line 875, including the kernel-side CommandList data structure that clearly must not be packed. Arnd --- Don: Thanks for your input. It helps a lot. The pragma setting predates my taking over the driver. It's true that there is a section of each command entry that is DMAed from the controller (from start of the CommandList up to busaddr) and the rest is driver housekeeping information. The unsupported controllers seem to be unable to handle the changed alignment. I have a patch I'll send up soon to change the alignment back... int retry_pending; /* 652 4 */ struct hpsa_scsi_dev_t * device; /* 656 8 */ atomic_t refcount; /* 664 4 */ /* size: 768, cachelines: 12, members: 16 */ /* padding: 100 */ } __attribute__((__aligned__(128))); Since this is a maintenance driver, I would rather not do too much surgery and invoke regression tests (and we no longer support these controllers). I'd rather just send up a patch to correct the issue on these legacy controllers. I have one ready to send up. Thanks for your observation and your attention. I'll send up the patch soon. Don ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-05 20:45 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-05 20:45 UTC (permalink / raw) To: arnd, geert Cc: slyich, glaubitz, storagedev, linux-scsi, linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEFybmQgQmVyZ21hbm4gW21haWx0bzph cm5kQGtlcm5lbC5vcmddIA0KU2VudDogRnJpZGF5LCBNYXJjaCA1LCAyMDIxIDc6MzIgQU0NClN1 YmplY3Q6IFJlOiBbYmlzZWN0ZWRdIDUuMTItcmMxIGhwc2EgcmVncmVzc2lvbjogInNjc2k6IGhw c2E6IENvcnJlY3QgZGV2IGNtZHMgb3V0c3RhbmRpbmcgZm9yIHJldHJpZWQgY21kcyIgYnJlYWtz IGhwc2EgUDYwMA0KDQpPbiBGcmksIE1hciA1LCAyMDIxIGF0IDEwOjI0IEFNIEdlZXJ0IFV5dHRl cmhvZXZlbiA8Z2VlcnRAbGludXgtbTY4ay5vcmc+IHdyb3RlOg0KPiBPbiBGcmksIE1hciA1LCAy MDIxIGF0IDEyOjI2IEFNIDxEb24uQnJhY2VAbWljcm9jaGlwLmNvbT4gd3JvdGU6DQo+ID4gPiA+ IE9uIDMvMi8yMSAxMToyNiBQTSwgU2VyZ2VpIFRyb2ZpbW92aWNoIHdyb3RlOg0KPiA+IHN0cnVj dCBDb21tYW5kTGlzdCB7DQo+ID4gICAgICAgICBzdHJ1Y3QgQ29tbWFuZExpc3RIZWFkZXIgSGVh ZGVyOyAgICAgICAgICAgICAgICAgLyogICAgIDAgICAgMjAgKi8NCj4gPiAgICAgICAgIHN0cnVj dCBSZXF1ZXN0QmxvY2sgUmVxdWVzdDsgICAgICAgICAgICAgICAgICAgICAvKiAgICAyMCAgICAy MCAqLw0KPiA+ICAgICAgICAgc3RydWN0IEVyckRlc2NyaXB0b3IgRXJyRGVzYzsgICAgICAgICAg ICAgICAgICAgIC8qICAgIDQwICAgIDEyICovDQo+ID4gICAgICAgICBzdHJ1Y3QgU0dEZXNjcmlw dG9yIFNHWzMyXTsgICAgICAgICAgICAgICAgICAgICAgLyogICAgNTIgICA1MTIgKi8NCj4gPiAg ICAgICAgIC8qIC0tLSBjYWNoZWxpbmUgOCBib3VuZGFyeSAoNTEyIGJ5dGVzKSB3YXMgNTIgYnl0 ZXMgYWdvIC0tLSAqLw0KPiA+ICAgICAgICAgdTMyICAgICAgICAgICAgICAgICAgICAgICAgYnVz YWRkcjsgICAgICAgICAgICAgIC8qICAgNTY0ICAgICA0ICovDQo+ID4gICAgICAgICBzdHJ1Y3Qg RXJyb3JJbmZvICogICAgICAgICBlcnJfaW5mbzsgICAgICAgICAgICAgLyogICA1NjggICAgIDgg Ki8NCj4gPiAgICAgICAgIC8qIC0tLSBjYWNoZWxpbmUgOSBib3VuZGFyeSAoNTc2IGJ5dGVzKSAt LS0gKi8NCj4gPiAgICAgICAgIHN0cnVjdCBjdGxyX2luZm8gKiAgICAgICAgIGg7ICAgICAgICAg ICAgICAgICAgICAvKiAgIDU3NiAgICAgOCAqLw0KPiA+ICAgICAgICAgaW50ICAgICAgICAgICAg ICAgICAgICAgICAgY21kX3R5cGU7ICAgICAgICAgICAgIC8qICAgNTg0ICAgICA0ICovDQo+ID4g ICAgICAgICBsb25nIGludCAgICAgICAgICAgICAgICAgICBjbWRpbmRleDsgICAgICAgICAgICAg LyogICA1ODggICAgIDggKi8NCj4gPiAgICAgICAgIHN0cnVjdCBjb21wbGV0aW9uICogICAgICAg IHdhaXRpbmc7ICAgICAgICAgICAgICAvKiAgIDU5NiAgICAgOCAqLw0KPiA+ICAgICAgICAgc3Ry dWN0IHNjc2lfY21uZCAqICAgICAgICAgc2NzaV9jbWQ7ICAgICAgICAgICAgIC8qICAgNjA0ICAg ICA4ICovDQo+ID4gICAgICAgICBzdHJ1Y3Qgd29ya19zdHJ1Y3Qgd29yazsgICAgICAgICAgICAg ICAgICAgICAgICAgLyogICA2MTIgICAgMzIgKi8NCj4gPiAgICAgICAgIC8qIC0tLSBjYWNoZWxp bmUgMTAgYm91bmRhcnkgKDY0MCBieXRlcykgd2FzIDQgYnl0ZXMgYWdvIC0tLSAqLw0KPiA+ICAg ICAgICAgc3RydWN0IGhwc2Ffc2NzaV9kZXZfdCAqICAgcGh5c19kaXNrOyAgICAgICAgICAgIC8q ICAgNjQ0ICAgICA4ICovDQo+ID4gICAgICAgICBzdHJ1Y3QgaHBzYV9zY3NpX2Rldl90ICogICBk ZXZpY2U7ICAgICAgICAgICAgICAgLyogICA2NTIgICAgIDggKi8NCj4gPiAgICAgICAgIGJvb2wg ICAgICAgICAgICAgICAgICAgICAgIHJldHJ5X3BlbmRpbmc7ICAgICAgICAvKiAgIDY2MCAgICAg MSAqLw0KPiA+ICAgICAgICAgYXRvbWljX3QgICAgICAgICAgICAgICAgICAgcmVmY291bnQ7ICAg ICAgICAgICAgIC8qICAgNjYxICAgICA0ICovDQo+DQo+IEhvdyBjb21lIHRoaXMgYXRvbWljX3Qg aXMgbm8gbG9uZ2VyIGFsaWduZWQgdG8gaXRzIG5hdHVyYWwgYWxpZ25tZW50Pw0KDQpUaGVyZSBp cyBhDQoNCiNwcmFnbWEgcGFjaygxKQ0KDQppbiBsaW51eCAyMDMgb2YgdGhpcyBmaWxlIQ0KDQpJ dCBsb29rcyBsaWtlIHNvbWUgb2YgdGhlIG1lbWJlcnMgaW4gc3RydWN0IHJhaWRfbWFwX2RhdGEg YW5kIHN0cnVjdCBDb21tYW5kTGlzdEhlYWRlciBuZWVkIHRvIGJlIGFubm90YXRlZCBhcyBwYWNr ZWQsIGJ1dCB0aGUgZmlsZSBhY2NpZGVudGFsbHkgcGFja3MgZXZlcnl0aGluZyB1bnRpbCB0aGUg JyNwcmFnbWEgcGFjaygpJw0KaW4gbGluZSA4NzUsIGluY2x1ZGluZyB0aGUga2VybmVsLXNpZGUg Q29tbWFuZExpc3QgZGF0YSBzdHJ1Y3R1cmUgdGhhdCBjbGVhcmx5IG11c3Qgbm90IGJlIHBhY2tl ZC4NCg0KICAgICAgICBBcm5kDQotLS0NCkRvbjoNClRoYW5rcyBmb3IgeW91ciBpbnB1dC4gSXQg aGVscHMgYSBsb3QuDQoNClRoZSBwcmFnbWEgc2V0dGluZyBwcmVkYXRlcyBteSB0YWtpbmcgb3Zl ciB0aGUgZHJpdmVyLg0KDQpJdCdzIHRydWUgdGhhdCB0aGVyZSBpcyBhIHNlY3Rpb24gb2YgZWFj aCBjb21tYW5kIGVudHJ5IHRoYXQgaXMgRE1BZWQgZnJvbSB0aGUgY29udHJvbGxlciAoZnJvbSBz dGFydCBvZiB0aGUgQ29tbWFuZExpc3QgdXAgdG8gYnVzYWRkcikgYW5kIHRoZSByZXN0IGlzIGRy aXZlciBob3VzZWtlZXBpbmcgaW5mb3JtYXRpb24uIFRoZSB1bnN1cHBvcnRlZCBjb250cm9sbGVy cyBzZWVtIHRvIGJlIHVuYWJsZSB0byBoYW5kbGUgdGhlIGNoYW5nZWQgYWxpZ25tZW50LiANCg0K SSBoYXZlIGEgcGF0Y2ggSSdsbCBzZW5kIHVwIHNvb24gdG8gY2hhbmdlIHRoZSBhbGlnbm1lbnQg YmFjay4uLg0KICAgICAgICBpbnQgICAgICAgICAgICAgICAgICAgICAgICByZXRyeV9wZW5kaW5n OyAgICAgICAgLyogICA2NTIgICAgIDQgKi8NCiAgICAgICAgc3RydWN0IGhwc2Ffc2NzaV9kZXZf dCAqICAgZGV2aWNlOyAgICAgICAgICAgICAgIC8qICAgNjU2ICAgICA4ICovDQogICAgICAgIGF0 b21pY190ICAgICAgICAgICAgICAgICAgIHJlZmNvdW50OyAgICAgICAgICAgICAvKiAgIDY2NCAg ICAgNCAqLw0KDQogICAgICAgIC8qIHNpemU6IDc2OCwgY2FjaGVsaW5lczogMTIsIG1lbWJlcnM6 IDE2ICovDQogICAgICAgIC8qIHBhZGRpbmc6IDEwMCAqLw0KfSBfX2F0dHJpYnV0ZV9fKChfX2Fs aWduZWRfXygxMjgpKSk7DQoNClNpbmNlIHRoaXMgaXMgYSBtYWludGVuYW5jZSBkcml2ZXIsIEkg d291bGQgcmF0aGVyIG5vdCBkbyB0b28gbXVjaCBzdXJnZXJ5IGFuZCBpbnZva2UgcmVncmVzc2lv biB0ZXN0cyAoYW5kIHdlIG5vIGxvbmdlciBzdXBwb3J0IHRoZXNlIGNvbnRyb2xsZXJzKS4gSSdk IHJhdGhlciBqdXN0IHNlbmQgdXAgYSBwYXRjaCB0byBjb3JyZWN0IHRoZSBpc3N1ZSBvbiB0aGVz ZSBsZWdhY3kgY29udHJvbGxlcnMuIEkgaGF2ZSBvbmUgcmVhZHkgdG8gc2VuZCB1cC4NCg0KVGhh bmtzIGZvciB5b3VyIG9ic2VydmF0aW9uIGFuZCB5b3VyIGF0dGVudGlvbi4NCkknbGwgc2VuZCB1 cCB0aGUgcGF0Y2ggc29vbi4NCg0KRG9uDQoNCg0KDQoNCg= ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 2021-03-03 0:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" bre Sergei Trofimovich @ 2021-03-03 15:42 ` Don.Brace -1 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-03 15:42 UTC (permalink / raw) To: slyich, glaubitz Cc: linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Tuesday, March 2, 2021 6:23 PM To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com> Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> Subject: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe On Tue, 2 Mar 2021 23:31:32 +0100 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > Hi Sergei! > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > Gave v5.12-rc1 a try today and got a similar boot failure around > > hpsa queue initialization, but my failure is later: > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > Maybe I get different error because I flipped on most debugging > > kernel options :) > > > > Looks like 'ERROR: Invalid distance value range' while being very > > scary are harmless. It's just a new spammy way for kernel to report > > lack of NUMA config on the machine (no SRAT and SLIT ACPI tables). > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > configuration is very wrong as I get unaligned accesses: > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > Bisecting now. > > Sounds good. I guess we should get Jens' fix for the signal regression > merged as well as your two fixes for strace. "bisected" (cheated halfway through) and verified that reverting f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. CCing authors who might be able to help us here. commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 Author: Don Brace <don.brace@microchip.com> Date: Mon Feb 15 16:26:57 2021 -0600 scsi: hpsa: Correct dev cmds outstanding for retried cmds Prevent incrementing device->commands_outstanding for ioaccel command retries that are driver initiated. If the command goes through the retry path, the device->commands_outstanding counter has already accounted for the number of commands outstanding to the device. Only commands going through function hpsa_cmd_resolve_events decrement this counter. - ioaccel commands go to either HBA disks or to logical volumes comprised of SSDs. The extra increment is causing device resets to hang. - Resets wait for all device outstanding commands to complete before returning. Replace unused field abort_pending with retry_pending. This is a maintenance driver so these changes have the least impact/risk. Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda Tested-by: Joe Szczypek <jszczype@redhat.com> Reviewed-by: Scott Benesh <scott.benesh@microchip.com> Reviewed-by: Scott Teel <scott.teel@microchip.com> Reviewed-by: Tomas Henzl <thenzl@redhat.com> Signed-off-by: Don Brace <don.brace@microchip.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> Don, do you happen to know why this patch caused some controller init failure for device 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 ? Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good The difference between the two boots is f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 in -good case. Looks like hpsa controller fails to initialize in bad case (could be a race?). -- Sergei Don: I see aligned access. Let me run pahole to see if anything jumps out. What controller are you using? ^ permalink raw reply [flat|nested] 89+ messages in thread
* RE: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" @ 2021-03-03 15:42 ` Don.Brace 0 siblings, 0 replies; 89+ messages in thread From: Don.Brace @ 2021-03-03 15:42 UTC (permalink / raw) To: slyich, glaubitz Cc: linux-ia64, linux-kernel, jszczype, Scott.Benesh, Scott.Teel, thenzl, martin.petersen -----Original Message----- From: Sergei Trofimovich [mailto:slyich@gmail.com] Sent: Tuesday, March 2, 2021 6:23 PM To: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>; Don Brace - C33706 <Don.Brace@microchip.com> Cc: linux-ia64@vger.kernel.org; linux-kernel@vger.kernel.org; Joe Szczypek <jszczype@redhat.com>; Scott Benesh - C33703 <Scott.Benesh@microchip.com>; Scott Teel - C33730 <Scott.Teel@microchip.com>; Tomas Henzl <thenzl@redhat.com>; Martin K. Petersen <martin.petersen@oracle.com> Subject: [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe On Tue, 2 Mar 2021 23:31:32 +0100 John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote: > Hi Sergei! > > On 3/2/21 11:26 PM, Sergei Trofimovich wrote: > > Gave v5.12-rc1 a try today and got a similar boot failure around > > hpsa queue initialization, but my failure is later: > > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 > > Maybe I get different error because I flipped on most debugging > > kernel options :) > > > > Looks like 'ERROR: Invalid distance value range' while being very > > scary are harmless. It's just a new spammy way for kernel to report > > lack of NUMA config on the machine (no SRAT and SLIT ACPI tables). > > > > At least I get hpsa detected on PCI bus. But I guess it's discovered > > configuration is very wrong as I get unaligned accesses: > > [ 19.811570] kernel unaligned access to 0xe000000105dd8295, ip=0xa000000100b874d1 > > > > Bisecting now. > > Sounds good. I guess we should get Jens' fix for the signal regression > merged as well as your two fixes for strace. "bisected" (cheated halfway through) and verified that reverting f749d8b7a9896bc6e5ffe104cc64345037e0b152 makes rx3600 boot again. CCing authors who might be able to help us here. commit f749d8b7a9896bc6e5ffe104cc64345037e0b152 Author: Don Brace <don.brace@microchip.com> Date: Mon Feb 15 16:26:57 2021 -0600 scsi: hpsa: Correct dev cmds outstanding for retried cmds Prevent incrementing device->commands_outstanding for ioaccel command retries that are driver initiated. If the command goes through the retry path, the device->commands_outstanding counter has already accounted for the number of commands outstanding to the device. Only commands going through function hpsa_cmd_resolve_events decrement this counter. - ioaccel commands go to either HBA disks or to logical volumes comprised of SSDs. The extra increment is causing device resets to hang. - Resets wait for all device outstanding commands to complete before returning. Replace unused field abort_pending with retry_pending. This is a maintenance driver so these changes have the least impact/risk. Link: https://lore.kernel.org/r/161342801747.29388.13045495968308188518.stgit@brunhilda Tested-by: Joe Szczypek <jszczype@redhat.com> Reviewed-by: Scott Benesh <scott.benesh@microchip.com> Reviewed-by: Scott Teel <scott.teel@microchip.com> Reviewed-by: Tomas Henzl <thenzl@redhat.com> Signed-off-by: Don Brace <don.brace@microchip.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> Don, do you happen to know why this patch caused some controller init failure for device 14:01.0 RAID bus controller: Hewlett-Packard Company Smart Array P600 ? Boot failure: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1 Boot success: https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.12-rc1-good The difference between the two boots is f749d8b7a9896bc6e5ffe104cc64345037e0b152 reverted on top of 5.12-rc1 in -good case. Looks like hpsa controller fails to initialize in bad case (could be a race?). -- Sergei Don: I see aligned access. Let me run pahole to see if anything jumps out. What controller are you using? ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210223192743.0198d4a9@sf> @ 2021-03-17 17:42 ` John Paul Adrian Glaubitz 2021-03-17 17:42 ` John Paul Adrian Glaubitz 1 sibling, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:42 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi! On 2/23/21 8:27 PM, Sergei Trofimovich wrote: >>> Just gave it a try and it still doesn't work. >> >> Maybe your other two patches to fix the strace issues are required as well? > > I'd say it's very unlikely they have any effect here. AFAIU they only amend > ptrace() behaviour called from userspace. Failure to boot so early is probably > way before any userspace. > >> Or do you happen to have more patches in the Gentoo kernel? > > Nope. It was a vanilla 5.11 release with 3 patches: 1 signal fix and > 2 ptrace() patches. > > Here are dmesg and config from my machine with successfull boot: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.11 > https://dev.gentoo.org/~slyfox/configs/guppy-config-5.11 Just gave it a try using this kernel configuration. It's never loading the hpsa module for me which I find really strange. The module isn't even showing up in the kernel message buffer. >> [ 0.036000] ERROR: Invalid distance value range >> [ 0.036000] >> [ 0.036000] 00 00 00 00 00 00 > > I don't see this string in the 5.11 kernel source. But > https://lore.kernel.org/lkml/161356785681.20312.13022545187499987936.tip-bot2@tip-bot2/T/ > hints it's might be something very new and you are in the > middle of 5.12-rc1? I'm seeing this using your exact kernel configuration. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-17 17:42 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:42 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi! On 2/23/21 8:27 PM, Sergei Trofimovich wrote: >>> Just gave it a try and it still doesn't work. >> >> Maybe your other two patches to fix the strace issues are required as well? > > I'd say it's very unlikely they have any effect here. AFAIU they only amend > ptrace() behaviour called from userspace. Failure to boot so early is probably > way before any userspace. > >> Or do you happen to have more patches in the Gentoo kernel? > > Nope. It was a vanilla 5.11 release with 3 patches: 1 signal fix and > 2 ptrace() patches. > > Here are dmesg and config from my machine with successfull boot: > https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.11 > https://dev.gentoo.org/~slyfox/configs/guppy-config-5.11 Just gave it a try using this kernel configuration. It's never loading the hpsa module for me which I find really strange. The module isn't even showing up in the kernel message buffer. >> [ 0.036000] ERROR: Invalid distance value range >> [ 0.036000] >> [ 0.036000] 00 00 00 00 00 00 > > I don't see this string in the 5.11 kernel source. But > https://lore.kernel.org/lkml/161356785681.20312.13022545187499987936.tip-bot2@tip-bot2/T/ > hints it's might be something very new and you are in the > middle of 5.12-rc1? I'm seeing this using your exact kernel configuration. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-03-17 17:42 ` John Paul Adrian Glaubitz @ 2021-03-17 17:53 ` John Paul Adrian Glaubitz -1 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:53 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi! On 3/17/21 6:42 PM, John Paul Adrian Glaubitz wrote: >> Here are dmesg and config from my machine with successfull boot: >> https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.11 >> https://dev.gentoo.org/~slyfox/configs/guppy-config-5.11 > > Just gave it a try using this kernel configuration. It's never loading the hpsa > module for me which I find really strange. The module isn't even showing up in > the kernel message buffer. Never mind. I just forgot to create an initrd this time, user error :-). Works again. >>> [ 0.036000] ERROR: Invalid distance value range >>> [ 0.036000] >>> [ 0.036000] 00 00 00 00 00 00 >> >> I don't see this string in the 5.11 kernel source. But >> https://lore.kernel.org/lkml/161356785681.20312.13022545187499987936.tip-bot2@tip-bot2/T/ >> hints it's might be something very new and you are in the >> middle of 5.12-rc1? > > I'm seeing this using your exact kernel configuration. This issue is still there, however. Do you see this as well? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-17 17:53 ` John Paul Adrian Glaubitz 0 siblings, 0 replies; 89+ messages in thread From: John Paul Adrian Glaubitz @ 2021-03-17 17:53 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: Jens Axboe, linux-ia64, linux-kernel Hi! On 3/17/21 6:42 PM, John Paul Adrian Glaubitz wrote: >> Here are dmesg and config from my machine with successfull boot: >> https://dev.gentoo.org/~slyfox/configs/guppy-dmesg-5.11 >> https://dev.gentoo.org/~slyfox/configs/guppy-config-5.11 > > Just gave it a try using this kernel configuration. It's never loading the hpsa > module for me which I find really strange. The module isn't even showing up in > the kernel message buffer. Never mind. I just forgot to create an initrd this time, user error :-). Works again. >>> [ 0.036000] ERROR: Invalid distance value range >>> [ 0.036000] >>> [ 0.036000] 00 00 00 00 00 00 >> >> I don't see this string in the 5.11 kernel source. But >> https://lore.kernel.org/lkml/161356785681.20312.13022545187499987936.tip-bot2@tip-bot2/T/ >> hints it's might be something very new and you are in the >> middle of 5.12-rc1? > > I'm seeing this using your exact kernel configuration. This issue is still there, however. Do you see this as well? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer - glaubitz@debian.org `. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de `- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913 ^ permalink raw reply [flat|nested] 89+ messages in thread
[parent not found: <20210222235359.75d1a912@sf>]
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210222235359.75d1a912@sf> @ 2021-02-23 0:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:34 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 4:53 PM, Sergei Trofimovich wrote: > On Mon, 22 Feb 2021 16:34:50 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > >> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>> Hia Jens! >>> >>> Tried 5.11 on rx3600 box and noticed it has >>> a problem handling init (5.10 booted fine): >>> >>> INIT: version 2.98 booting >>> >>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>> >>> mkdir `/run/openrc': Read-only file system >>> mkdir `/run/openrc/starting': No such file or directory >>> mkdir `/run/openrc/started': No such file or directory >>> mkdir `/run/openrc/stopping': No such file or directory >>> mkdir `/run/openrc/inactive': No such file or directory >>> mkdir `/run/openrc/wasinactive': No such file or directory >>> mkdir `/run/openrc/failed': No such file or directory >>> mkdir `/run/openrc/hotplugged': No such file or directory >>> mkdir `/run/openrc/daemons': No such file or directory >>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> >>> I suspect we build bad signal stack frame for userspace. >>> >>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>> >>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> >>> Bisect points at: >>> >>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>> Author: Jens Axboe <axboe@kernel.dk> >>> Date: Fri Oct 9 14:49:43 2020 -0600 >>> >>> ia64: add support for TIF_NOTIFY_SIGNAL >>> >>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>> >>> Cc: linux-ia64@vger.kernel.org >>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>> >>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>> index 64a1011f6812..51d20cb37706 100644 >>> --- a/arch/ia64/include/asm/thread_info.h >>> +++ b/arch/ia64/include/asm/thread_info.h >>> @@ -103,6 +103,7 @@ struct thread_info { >>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>> @@ -115,6 +116,7 @@ struct thread_info { >>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>> @@ -124,7 +126,7 @@ struct thread_info { >>> >>> /* "work to do on user-return" bits */ >>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>> >>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>> index 6b61a703bcf5..8d4e1cab9190 100644 >>> --- a/arch/ia64/kernel/process.c >>> +++ b/arch/ia64/kernel/process.c >>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>> } >>> >>> /* deal with pending signal delivery */ >>> - if (test_thread_flag(TIF_SIGPENDING)) { >>> + if (test_thread_flag(TIF_SIGPENDING) || >>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>> local_irq_enable(); /* force interrupt enable */ >>> ia64_do_signal(scr, in_syscall); >>> >>> which looks benign, but it enables a bit of conditional >>> TIF_NOTIFY_SIGNAL handling I don't understand. >>> >>> Can you help me get what is the interaction between >>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>> simple processes without io_uring use case? >>> >>> I wonder if it's ia64_do_signal()' generates a signal >>> delivery when it should not. >> >> Can you test: >> >> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >> >> with the addition mentioned here: > > Not enough: > > mkdir `/run/openrc': Read-only file system > mkdir `/run/openrc/starting': No such file or directory > mkdir `/run/openrc/started': No such file or directory > mkdir `/run/openrc/stopping': No such file or directory > mkdir `/run/openrc/inactive': No such file or directory > mkdir `/run/openrc/wasinactive': No such file or directory > mkdir `/run/openrc/failed': No such file or directory > mkdir `/run/openrc/hotplugged': No such file or directory > mkdir `/run/openrc/daemons': No such file or directory > [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > mkdir `/run/openrc/options': No such file or directory > mkdir `/run/openrc/exclusive': No such file or directory > mkdir `/run/openrc/scheduled': No such file or directory > mkdir `/run/openrc/tmp': No such file or directory > >> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >> >> if needed? > > Two patches above do fix the boot \o/ But have a lot of spam about > 'signal 0' delivery to a bunch of processes: > > * Mounting /proc ... > [ ok ] > * Mounting /run ... > * /run/openrc: creating directory > * /run/lock: creating directory > * /run/lock: correcting owner > * Caching service dependencies ... > [ ok ] > * Mounting /sys ... > [ ok ] > * Mounting debug filesystem ... > [ ok ] > * Mounting efivarfs filesystem ... > [ ok ] > * sysfs: caught unknown signal 0 > * openrc: caught unknown signal 0 > * Mounting cgroup filesystem ... That's an improvement! Let me take a look at this tonight and see if I can figure out what's going on. But yes, it's the ia64 signal delivery being just different enough from the norm that it apparently triggers some weirdness. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-23 0:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:34 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 4:53 PM, Sergei Trofimovich wrote: > On Mon, 22 Feb 2021 16:34:50 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > >> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>> Hia Jens! >>> >>> Tried 5.11 on rx3600 box and noticed it has >>> a problem handling init (5.10 booted fine): >>> >>> INIT: version 2.98 booting >>> >>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>> >>> mkdir `/run/openrc': Read-only file system >>> mkdir `/run/openrc/starting': No such file or directory >>> mkdir `/run/openrc/started': No such file or directory >>> mkdir `/run/openrc/stopping': No such file or directory >>> mkdir `/run/openrc/inactive': No such file or directory >>> mkdir `/run/openrc/wasinactive': No such file or directory >>> mkdir `/run/openrc/failed': No such file or directory >>> mkdir `/run/openrc/hotplugged': No such file or directory >>> mkdir `/run/openrc/daemons': No such file or directory >>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> >>> I suspect we build bad signal stack frame for userspace. >>> >>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>> >>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> >>> Bisect points at: >>> >>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>> Author: Jens Axboe <axboe@kernel.dk> >>> Date: Fri Oct 9 14:49:43 2020 -0600 >>> >>> ia64: add support for TIF_NOTIFY_SIGNAL >>> >>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>> >>> Cc: linux-ia64@vger.kernel.org >>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>> >>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>> index 64a1011f6812..51d20cb37706 100644 >>> --- a/arch/ia64/include/asm/thread_info.h >>> +++ b/arch/ia64/include/asm/thread_info.h >>> @@ -103,6 +103,7 @@ struct thread_info { >>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>> @@ -115,6 +116,7 @@ struct thread_info { >>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>> @@ -124,7 +126,7 @@ struct thread_info { >>> >>> /* "work to do on user-return" bits */ >>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>> >>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>> index 6b61a703bcf5..8d4e1cab9190 100644 >>> --- a/arch/ia64/kernel/process.c >>> +++ b/arch/ia64/kernel/process.c >>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>> } >>> >>> /* deal with pending signal delivery */ >>> - if (test_thread_flag(TIF_SIGPENDING)) { >>> + if (test_thread_flag(TIF_SIGPENDING) || >>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>> local_irq_enable(); /* force interrupt enable */ >>> ia64_do_signal(scr, in_syscall); >>> >>> which looks benign, but it enables a bit of conditional >>> TIF_NOTIFY_SIGNAL handling I don't understand. >>> >>> Can you help me get what is the interaction between >>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>> simple processes without io_uring use case? >>> >>> I wonder if it's ia64_do_signal()' generates a signal >>> delivery when it should not. >> >> Can you test: >> >> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >> >> with the addition mentioned here: > > Not enough: > > mkdir `/run/openrc': Read-only file system > mkdir `/run/openrc/starting': No such file or directory > mkdir `/run/openrc/started': No such file or directory > mkdir `/run/openrc/stopping': No such file or directory > mkdir `/run/openrc/inactive': No such file or directory > mkdir `/run/openrc/wasinactive': No such file or directory > mkdir `/run/openrc/failed': No such file or directory > mkdir `/run/openrc/hotplugged': No such file or directory > mkdir `/run/openrc/daemons': No such file or directory > [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > mkdir `/run/openrc/options': No such file or directory > mkdir `/run/openrc/exclusive': No such file or directory > mkdir `/run/openrc/scheduled': No such file or directory > mkdir `/run/openrc/tmp': No such file or directory > >> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >> >> if needed? > > Two patches above do fix the boot \o/ But have a lot of spam about > 'signal 0' delivery to a bunch of processes: > > * Mounting /proc ... > [ ok ] > * Mounting /run ... > * /run/openrc: creating directory > * /run/lock: creating directory > * /run/lock: correcting owner > * Caching service dependencies ... > [ ok ] > * Mounting /sys ... > [ ok ] > * Mounting debug filesystem ... > [ ok ] > * Mounting efivarfs filesystem ... > [ ok ] > * sysfs: caught unknown signal 0 > * openrc: caught unknown signal 0 > * Mounting cgroup filesystem ... That's an improvement! Let me take a look at this tonight and see if I can figure out what's going on. But yes, it's the ia64 signal delivery being just different enough from the norm that it apparently triggers some weirdness. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-02-23 0:34 ` Jens Axboe @ 2021-02-23 0:41 ` Jens Axboe -1 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:41 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 5:34 PM, Jens Axboe wrote: > On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >> On Mon, 22 Feb 2021 16:34:50 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>> Hia Jens! >>>> >>>> Tried 5.11 on rx3600 box and noticed it has >>>> a problem handling init (5.10 booted fine): >>>> >>>> INIT: version 2.98 booting >>>> >>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>> >>>> mkdir `/run/openrc': Read-only file system >>>> mkdir `/run/openrc/starting': No such file or directory >>>> mkdir `/run/openrc/started': No such file or directory >>>> mkdir `/run/openrc/stopping': No such file or directory >>>> mkdir `/run/openrc/inactive': No such file or directory >>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>> mkdir `/run/openrc/failed': No such file or directory >>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>> mkdir `/run/openrc/daemons': No such file or directory >>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>> >>>> I suspect we build bad signal stack frame for userspace. >>>> >>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>> >>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>> >>>> Bisect points at: >>>> >>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>> Author: Jens Axboe <axboe@kernel.dk> >>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>> >>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>> >>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>> >>>> Cc: linux-ia64@vger.kernel.org >>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>> >>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>> index 64a1011f6812..51d20cb37706 100644 >>>> --- a/arch/ia64/include/asm/thread_info.h >>>> +++ b/arch/ia64/include/asm/thread_info.h >>>> @@ -103,6 +103,7 @@ struct thread_info { >>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>> @@ -115,6 +116,7 @@ struct thread_info { >>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>> @@ -124,7 +126,7 @@ struct thread_info { >>>> >>>> /* "work to do on user-return" bits */ >>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>> >>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>> --- a/arch/ia64/kernel/process.c >>>> +++ b/arch/ia64/kernel/process.c >>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>> } >>>> >>>> /* deal with pending signal delivery */ >>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>> local_irq_enable(); /* force interrupt enable */ >>>> ia64_do_signal(scr, in_syscall); >>>> >>>> which looks benign, but it enables a bit of conditional >>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>> >>>> Can you help me get what is the interaction between >>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>> simple processes without io_uring use case? >>>> >>>> I wonder if it's ia64_do_signal()' generates a signal >>>> delivery when it should not. >>> >>> Can you test: >>> >>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >>> >>> with the addition mentioned here: >> >> Not enough: >> >> mkdir `/run/openrc': Read-only file system >> mkdir `/run/openrc/starting': No such file or directory >> mkdir `/run/openrc/started': No such file or directory >> mkdir `/run/openrc/stopping': No such file or directory >> mkdir `/run/openrc/inactive': No such file or directory >> mkdir `/run/openrc/wasinactive': No such file or directory >> mkdir `/run/openrc/failed': No such file or directory >> mkdir `/run/openrc/hotplugged': No such file or directory >> mkdir `/run/openrc/daemons': No such file or directory >> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >> mkdir `/run/openrc/options': No such file or directory >> mkdir `/run/openrc/exclusive': No such file or directory >> mkdir `/run/openrc/scheduled': No such file or directory >> mkdir `/run/openrc/tmp': No such file or directory >> >>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >>> >>> if needed? >> >> Two patches above do fix the boot \o/ But have a lot of spam about >> 'signal 0' delivery to a bunch of processes: >> >> * Mounting /proc ... >> [ ok ] >> * Mounting /run ... >> * /run/openrc: creating directory >> * /run/lock: creating directory >> * /run/lock: correcting owner >> * Caching service dependencies ... >> [ ok ] >> * Mounting /sys ... >> [ ok ] >> * Mounting debug filesystem ... >> [ ok ] >> * Mounting efivarfs filesystem ... >> [ ok ] >> * sysfs: caught unknown signal 0 >> * openrc: caught unknown signal 0 >> * Mounting cgroup filesystem ... > > That's an improvement! Let me take a look at this tonight and see if I > can figure out what's going on. But yes, it's the ia64 signal delivery > being just different enough from the norm that it apparently triggers > some weirdness. Is this any better? diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c index e67b22fc3c60..11891240aa5c 100644 --- a/arch/ia64/kernel/signal.c +++ b/arch/ia64/kernel/signal.c @@ -340,8 +340,10 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) * This only loops in the rare cases of handle_signal() failing, in which case we * need to push through a forced SIGSEGV. */ - while (1) { - get_signal(&ksig); + do { + ksig.sig = 0; + if (!get_signal(&ksig)) + break; /* * get_signal() may have run a debugger (via notify_parent()) @@ -358,9 +360,6 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) */ restart = 0; - if (ksig.sig <= 0) - break; - if (unlikely(restart)) { switch (errno) { case ERESTART_RESTARTBLOCK: @@ -387,7 +386,7 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) */ if (handle_signal(&ksig, scr)) return; - } + } while (ksig.sig > 0); /* Did we come from a system call? */ if (restart) { -- Jens Axboe ^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-23 0:41 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:41 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 5:34 PM, Jens Axboe wrote: > On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >> On Mon, 22 Feb 2021 16:34:50 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>> Hia Jens! >>>> >>>> Tried 5.11 on rx3600 box and noticed it has >>>> a problem handling init (5.10 booted fine): >>>> >>>> INIT: version 2.98 booting >>>> >>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>> >>>> mkdir `/run/openrc': Read-only file system >>>> mkdir `/run/openrc/starting': No such file or directory >>>> mkdir `/run/openrc/started': No such file or directory >>>> mkdir `/run/openrc/stopping': No such file or directory >>>> mkdir `/run/openrc/inactive': No such file or directory >>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>> mkdir `/run/openrc/failed': No such file or directory >>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>> mkdir `/run/openrc/daemons': No such file or directory >>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>> >>>> I suspect we build bad signal stack frame for userspace. >>>> >>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>> >>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>> >>>> Bisect points at: >>>> >>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>> Author: Jens Axboe <axboe@kernel.dk> >>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>> >>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>> >>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>> >>>> Cc: linux-ia64@vger.kernel.org >>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>> >>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>> index 64a1011f6812..51d20cb37706 100644 >>>> --- a/arch/ia64/include/asm/thread_info.h >>>> +++ b/arch/ia64/include/asm/thread_info.h >>>> @@ -103,6 +103,7 @@ struct thread_info { >>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>> @@ -115,6 +116,7 @@ struct thread_info { >>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>> @@ -124,7 +126,7 @@ struct thread_info { >>>> >>>> /* "work to do on user-return" bits */ >>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>> >>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>> --- a/arch/ia64/kernel/process.c >>>> +++ b/arch/ia64/kernel/process.c >>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>> } >>>> >>>> /* deal with pending signal delivery */ >>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>> local_irq_enable(); /* force interrupt enable */ >>>> ia64_do_signal(scr, in_syscall); >>>> >>>> which looks benign, but it enables a bit of conditional >>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>> >>>> Can you help me get what is the interaction between >>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>> simple processes without io_uring use case? >>>> >>>> I wonder if it's ia64_do_signal()' generates a signal >>>> delivery when it should not. >>> >>> Can you test: >>> >>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >>> >>> with the addition mentioned here: >> >> Not enough: >> >> mkdir `/run/openrc': Read-only file system >> mkdir `/run/openrc/starting': No such file or directory >> mkdir `/run/openrc/started': No such file or directory >> mkdir `/run/openrc/stopping': No such file or directory >> mkdir `/run/openrc/inactive': No such file or directory >> mkdir `/run/openrc/wasinactive': No such file or directory >> mkdir `/run/openrc/failed': No such file or directory >> mkdir `/run/openrc/hotplugged': No such file or directory >> mkdir `/run/openrc/daemons': No such file or directory >> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >> mkdir `/run/openrc/options': No such file or directory >> mkdir `/run/openrc/exclusive': No such file or directory >> mkdir `/run/openrc/scheduled': No such file or directory >> mkdir `/run/openrc/tmp': No such file or directory >> >>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >>> >>> if needed? >> >> Two patches above do fix the boot \o/ But have a lot of spam about >> 'signal 0' delivery to a bunch of processes: >> >> * Mounting /proc ... >> [ ok ] >> * Mounting /run ... >> * /run/openrc: creating directory >> * /run/lock: creating directory >> * /run/lock: correcting owner >> * Caching service dependencies ... >> [ ok ] >> * Mounting /sys ... >> [ ok ] >> * Mounting debug filesystem ... >> [ ok ] >> * Mounting efivarfs filesystem ... >> [ ok ] >> * sysfs: caught unknown signal 0 >> * openrc: caught unknown signal 0 >> * Mounting cgroup filesystem ... > > That's an improvement! Let me take a look at this tonight and see if I > can figure out what's going on. But yes, it's the ia64 signal delivery > being just different enough from the norm that it apparently triggers > some weirdness. Is this any better? diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c index e67b22fc3c60..11891240aa5c 100644 --- a/arch/ia64/kernel/signal.c +++ b/arch/ia64/kernel/signal.c @@ -340,8 +340,10 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) * This only loops in the rare cases of handle_signal() failing, in which case we * need to push through a forced SIGSEGV. */ - while (1) { - get_signal(&ksig); + do { + ksig.sig = 0; + if (!get_signal(&ksig)) + break; /* * get_signal() may have run a debugger (via notify_parent()) @@ -358,9 +360,6 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) */ restart = 0; - if (ksig.sig <= 0) - break; - if (unlikely(restart)) { switch (errno) { case ERESTART_RESTARTBLOCK: @@ -387,7 +386,7 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) */ if (handle_signal(&ksig, scr)) return; - } + } while (ksig.sig > 0); /* Did we come from a system call? */ if (restart) { -- Jens Axboe ^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-02-23 0:41 ` Jens Axboe @ 2021-02-23 0:43 ` Jens Axboe -1 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:43 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 5:41 PM, Jens Axboe wrote: > On 2/22/21 5:34 PM, Jens Axboe wrote: >> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>> On Mon, 22 Feb 2021 16:34:50 -0700 >>> Jens Axboe <axboe@kernel.dk> wrote: >>> >>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>> Hia Jens! >>>>> >>>>> Tried 5.11 on rx3600 box and noticed it has >>>>> a problem handling init (5.10 booted fine): >>>>> >>>>> INIT: version 2.98 booting >>>>> >>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>> >>>>> mkdir `/run/openrc': Read-only file system >>>>> mkdir `/run/openrc/starting': No such file or directory >>>>> mkdir `/run/openrc/started': No such file or directory >>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>> mkdir `/run/openrc/failed': No such file or directory >>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>> >>>>> I suspect we build bad signal stack frame for userspace. >>>>> >>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>> >>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>> >>>>> Bisect points at: >>>>> >>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>> >>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>> >>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>> >>>>> Cc: linux-ia64@vger.kernel.org >>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>> >>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>> index 64a1011f6812..51d20cb37706 100644 >>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>> >>>>> /* "work to do on user-return" bits */ >>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>> >>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>> --- a/arch/ia64/kernel/process.c >>>>> +++ b/arch/ia64/kernel/process.c >>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>> } >>>>> >>>>> /* deal with pending signal delivery */ >>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>> local_irq_enable(); /* force interrupt enable */ >>>>> ia64_do_signal(scr, in_syscall); >>>>> >>>>> which looks benign, but it enables a bit of conditional >>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>> >>>>> Can you help me get what is the interaction between >>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>> simple processes without io_uring use case? >>>>> >>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>> delivery when it should not. >>>> >>>> Can you test: >>>> >>>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >>>> >>>> with the addition mentioned here: >>> >>> Not enough: >>> >>> mkdir `/run/openrc': Read-only file system >>> mkdir `/run/openrc/starting': No such file or directory >>> mkdir `/run/openrc/started': No such file or directory >>> mkdir `/run/openrc/stopping': No such file or directory >>> mkdir `/run/openrc/inactive': No such file or directory >>> mkdir `/run/openrc/wasinactive': No such file or directory >>> mkdir `/run/openrc/failed': No such file or directory >>> mkdir `/run/openrc/hotplugged': No such file or directory >>> mkdir `/run/openrc/daemons': No such file or directory >>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> mkdir `/run/openrc/options': No such file or directory >>> mkdir `/run/openrc/exclusive': No such file or directory >>> mkdir `/run/openrc/scheduled': No such file or directory >>> mkdir `/run/openrc/tmp': No such file or directory >>> >>>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >>>> >>>> if needed? >>> >>> Two patches above do fix the boot \o/ But have a lot of spam about >>> 'signal 0' delivery to a bunch of processes: >>> >>> * Mounting /proc ... >>> [ ok ] >>> * Mounting /run ... >>> * /run/openrc: creating directory >>> * /run/lock: creating directory >>> * /run/lock: correcting owner >>> * Caching service dependencies ... >>> [ ok ] >>> * Mounting /sys ... >>> [ ok ] >>> * Mounting debug filesystem ... >>> [ ok ] >>> * Mounting efivarfs filesystem ... >>> [ ok ] >>> * sysfs: caught unknown signal 0 >>> * openrc: caught unknown signal 0 >>> * Mounting cgroup filesystem ... >> >> That's an improvement! Let me take a look at this tonight and see if I >> can figure out what's going on. But yes, it's the ia64 signal delivery >> being just different enough from the norm that it apparently triggers >> some weirdness. > > Is this any better? And if that one works, can you try this basic variant? diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c index e67b22fc3c60..c1b299760bf7 100644 --- a/arch/ia64/kernel/signal.c +++ b/arch/ia64/kernel/signal.c @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) * need to push through a forced SIGSEGV. */ while (1) { - get_signal(&ksig); + if (!get_signal(&ksig)) + break; /* * get_signal() may have run a debugger (via notify_parent()) -- Jens Axboe ^ permalink raw reply related [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-02-23 0:43 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-02-23 0:43 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 2/22/21 5:41 PM, Jens Axboe wrote: > On 2/22/21 5:34 PM, Jens Axboe wrote: >> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>> On Mon, 22 Feb 2021 16:34:50 -0700 >>> Jens Axboe <axboe@kernel.dk> wrote: >>> >>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>> Hia Jens! >>>>> >>>>> Tried 5.11 on rx3600 box and noticed it has >>>>> a problem handling init (5.10 booted fine): >>>>> >>>>> INIT: version 2.98 booting >>>>> >>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>> >>>>> mkdir `/run/openrc': Read-only file system >>>>> mkdir `/run/openrc/starting': No such file or directory >>>>> mkdir `/run/openrc/started': No such file or directory >>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>> mkdir `/run/openrc/failed': No such file or directory >>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>> >>>>> I suspect we build bad signal stack frame for userspace. >>>>> >>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>> >>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>> >>>>> Bisect points at: >>>>> >>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>> >>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>> >>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>> >>>>> Cc: linux-ia64@vger.kernel.org >>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>> >>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>> index 64a1011f6812..51d20cb37706 100644 >>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>> >>>>> /* "work to do on user-return" bits */ >>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>> >>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>> --- a/arch/ia64/kernel/process.c >>>>> +++ b/arch/ia64/kernel/process.c >>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>> } >>>>> >>>>> /* deal with pending signal delivery */ >>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>> local_irq_enable(); /* force interrupt enable */ >>>>> ia64_do_signal(scr, in_syscall); >>>>> >>>>> which looks benign, but it enables a bit of conditional >>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>> >>>>> Can you help me get what is the interaction between >>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>> simple processes without io_uring use case? >>>>> >>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>> delivery when it should not. >>>> >>>> Can you test: >>>> >>>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >>>> >>>> with the addition mentioned here: >>> >>> Not enough: >>> >>> mkdir `/run/openrc': Read-only file system >>> mkdir `/run/openrc/starting': No such file or directory >>> mkdir `/run/openrc/started': No such file or directory >>> mkdir `/run/openrc/stopping': No such file or directory >>> mkdir `/run/openrc/inactive': No such file or directory >>> mkdir `/run/openrc/wasinactive': No such file or directory >>> mkdir `/run/openrc/failed': No such file or directory >>> mkdir `/run/openrc/hotplugged': No such file or directory >>> mkdir `/run/openrc/daemons': No such file or directory >>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>> mkdir `/run/openrc/options': No such file or directory >>> mkdir `/run/openrc/exclusive': No such file or directory >>> mkdir `/run/openrc/scheduled': No such file or directory >>> mkdir `/run/openrc/tmp': No such file or directory >>> >>>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >>>> >>>> if needed? >>> >>> Two patches above do fix the boot \o/ But have a lot of spam about >>> 'signal 0' delivery to a bunch of processes: >>> >>> * Mounting /proc ... >>> [ ok ] >>> * Mounting /run ... >>> * /run/openrc: creating directory >>> * /run/lock: creating directory >>> * /run/lock: correcting owner >>> * Caching service dependencies ... >>> [ ok ] >>> * Mounting /sys ... >>> [ ok ] >>> * Mounting debug filesystem ... >>> [ ok ] >>> * Mounting efivarfs filesystem ... >>> [ ok ] >>> * sysfs: caught unknown signal 0 >>> * openrc: caught unknown signal 0 >>> * Mounting cgroup filesystem ... >> >> That's an improvement! Let me take a look at this tonight and see if I >> can figure out what's going on. But yes, it's the ia64 signal delivery >> being just different enough from the norm that it apparently triggers >> some weirdness. > > Is this any better? And if that one works, can you try this basic variant? diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c index e67b22fc3c60..c1b299760bf7 100644 --- a/arch/ia64/kernel/signal.c +++ b/arch/ia64/kernel/signal.c @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) * need to push through a forced SIGSEGV. */ while (1) { - get_signal(&ksig); + if (!get_signal(&ksig)) + break; /* * get_signal() may have run a debugger (via notify_parent()) -- Jens Axboe ^ permalink raw reply related [flat|nested] 89+ messages in thread
[parent not found: <20210223080830.23bccdbf@sf>]
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210223080830.23bccdbf@sf> @ 2021-03-02 22:07 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-02 22:07 UTC (permalink / raw) To: Jens Axboe; +Cc: linux-ia64, linux-kernel On Tue, 23 Feb 2021 08:08:30 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Mon, 22 Feb 2021 17:43:58 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > > > On 2/22/21 5:41 PM, Jens Axboe wrote: > > > On 2/22/21 5:34 PM, Jens Axboe wrote: > > >> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: > > >>> On Mon, 22 Feb 2021 16:34:50 -0700 > > >>> Jens Axboe <axboe@kernel.dk> wrote: > > >>> > > >>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: > > >>>>> Hia Jens! > > >>>>> > > >>>>> Tried 5.11 on rx3600 box and noticed it has > > >>>>> a problem handling init (5.10 booted fine): > > >>>>> > > >>>>> INIT: version 2.98 booting > > >>>>> > > >>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) > > >>>>> > > >>>>> mkdir `/run/openrc': Read-only file system > > >>>>> mkdir `/run/openrc/starting': No such file or directory > > >>>>> mkdir `/run/openrc/started': No such file or directory > > >>>>> mkdir `/run/openrc/stopping': No such file or directory > > >>>>> mkdir `/run/openrc/inactive': No such file or directory > > >>>>> mkdir `/run/openrc/wasinactive': No such file or directory > > >>>>> mkdir `/run/openrc/failed': No such file or directory > > >>>>> mkdir `/run/openrc/hotplugged': No such file or directory > > >>>>> mkdir `/run/openrc/daemons': No such file or directory > > >>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>>>> > > >>>>> I suspect we build bad signal stack frame for userspace. > > >>>>> > > >>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: > > >>>>> > > >>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 > > >>>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 > > >>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 > > >>>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 > > >>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 > > >>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>>>> > > >>>>> Bisect points at: > > >>>>> > > >>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 > > >>>>> Author: Jens Axboe <axboe@kernel.dk> > > >>>>> Date: Fri Oct 9 14:49:43 2020 -0600 > > >>>>> > > >>>>> ia64: add support for TIF_NOTIFY_SIGNAL > > >>>>> > > >>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. > > >>>>> > > >>>>> Cc: linux-ia64@vger.kernel.org > > >>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] > > >>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> > > >>>>> > > >>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h > > >>>>> index 64a1011f6812..51d20cb37706 100644 > > >>>>> --- a/arch/ia64/include/asm/thread_info.h > > >>>>> +++ b/arch/ia64/include/asm/thread_info.h > > >>>>> @@ -103,6 +103,7 @@ struct thread_info { > > >>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ > > >>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ > > >>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ > > >>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ > > >>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ > > >>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ > > >>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ > > >>>>> @@ -115,6 +116,7 @@ struct thread_info { > > >>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) > > >>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) > > >>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) > > >>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) > > >>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > > >>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > > >>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) > > >>>>> @@ -124,7 +126,7 @@ struct thread_info { > > >>>>> > > >>>>> /* "work to do on user-return" bits */ > > >>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ > > >>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) > > >>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) > > >>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ > > >>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) > > >>>>> > > >>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c > > >>>>> index 6b61a703bcf5..8d4e1cab9190 100644 > > >>>>> --- a/arch/ia64/kernel/process.c > > >>>>> +++ b/arch/ia64/kernel/process.c > > >>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) > > >>>>> } > > >>>>> > > >>>>> /* deal with pending signal delivery */ > > >>>>> - if (test_thread_flag(TIF_SIGPENDING)) { > > >>>>> + if (test_thread_flag(TIF_SIGPENDING) || > > >>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { > > >>>>> local_irq_enable(); /* force interrupt enable */ > > >>>>> ia64_do_signal(scr, in_syscall); > > >>>>> > > >>>>> which looks benign, but it enables a bit of conditional > > >>>>> TIF_NOTIFY_SIGNAL handling I don't understand. > > >>>>> > > >>>>> Can you help me get what is the interaction between > > >>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for > > >>>>> simple processes without io_uring use case? > > >>>>> > > >>>>> I wonder if it's ia64_do_signal()' generates a signal > > >>>>> delivery when it should not. > > >>>> > > >>>> Can you test: > > >>>> > > >>>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 > > >>>> > > >>>> with the addition mentioned here: > > >>> > > >>> Not enough: > > >>> > > >>> mkdir `/run/openrc': Read-only file system > > >>> mkdir `/run/openrc/starting': No such file or directory > > >>> mkdir `/run/openrc/started': No such file or directory > > >>> mkdir `/run/openrc/stopping': No such file or directory > > >>> mkdir `/run/openrc/inactive': No such file or directory > > >>> mkdir `/run/openrc/wasinactive': No such file or directory > > >>> mkdir `/run/openrc/failed': No such file or directory > > >>> mkdir `/run/openrc/hotplugged': No such file or directory > > >>> mkdir `/run/openrc/daemons': No such file or directory > > >>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>> mkdir `/run/openrc/options': No such file or directory > > >>> mkdir `/run/openrc/exclusive': No such file or directory > > >>> mkdir `/run/openrc/scheduled': No such file or directory > > >>> mkdir `/run/openrc/tmp': No such file or directory > > >>> > > >>>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 > > >>>> > > >>>> if needed? > > >>> > > >>> Two patches above do fix the boot \o/ But have a lot of spam about > > >>> 'signal 0' delivery to a bunch of processes: > > >>> > > >>> * Mounting /proc ... > > >>> [ ok ] > > >>> * Mounting /run ... > > >>> * /run/openrc: creating directory > > >>> * /run/lock: creating directory > > >>> * /run/lock: correcting owner > > >>> * Caching service dependencies ... > > >>> [ ok ] > > >>> * Mounting /sys ... > > >>> [ ok ] > > >>> * Mounting debug filesystem ... > > >>> [ ok ] > > >>> * Mounting efivarfs filesystem ... > > >>> [ ok ] > > >>> * sysfs: caught unknown signal 0 > > >>> * openrc: caught unknown signal 0 > > >>> * Mounting cgroup filesystem ... > > >> > > >> That's an improvement! Let me take a look at this tonight and see if I > > >> can figure out what's going on. But yes, it's the ia64 signal delivery > > >> being just different enough from the norm that it apparently triggers > > >> some weirdness. > > > > > > Is this any better? > > > > And if that one works, can you try this basic variant? > > Both patches boot successfully without 'caught unknown signal 0' spam \o/ > > > diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c > > index e67b22fc3c60..c1b299760bf7 100644 > > --- a/arch/ia64/kernel/signal.c > > +++ b/arch/ia64/kernel/signal.c > > @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) > > * need to push through a forced SIGSEGV. > > */ > > while (1) { > > - get_signal(&ksig); > > + if (!get_signal(&ksig)) > > + break; > > > > /* > > * get_signal() may have run a debugger (via notify_parent()) > > Should I send the patch in `git am`-able form or the patch already queued up in some other form? -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-02 22:07 ` Sergei Trofimovich 0 siblings, 0 replies; 89+ messages in thread From: Sergei Trofimovich @ 2021-03-02 22:07 UTC (permalink / raw) To: Jens Axboe; +Cc: linux-ia64, linux-kernel On Tue, 23 Feb 2021 08:08:30 +0000 Sergei Trofimovich <slyich@gmail.com> wrote: > On Mon, 22 Feb 2021 17:43:58 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > > > On 2/22/21 5:41 PM, Jens Axboe wrote: > > > On 2/22/21 5:34 PM, Jens Axboe wrote: > > >> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: > > >>> On Mon, 22 Feb 2021 16:34:50 -0700 > > >>> Jens Axboe <axboe@kernel.dk> wrote: > > >>> > > >>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: > > >>>>> Hia Jens! > > >>>>> > > >>>>> Tried 5.11 on rx3600 box and noticed it has > > >>>>> a problem handling init (5.10 booted fine): > > >>>>> > > >>>>> INIT: version 2.98 booting > > >>>>> > > >>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) > > >>>>> > > >>>>> mkdir `/run/openrc': Read-only file system > > >>>>> mkdir `/run/openrc/starting': No such file or directory > > >>>>> mkdir `/run/openrc/started': No such file or directory > > >>>>> mkdir `/run/openrc/stopping': No such file or directory > > >>>>> mkdir `/run/openrc/inactive': No such file or directory > > >>>>> mkdir `/run/openrc/wasinactive': No such file or directory > > >>>>> mkdir `/run/openrc/failed': No such file or directory > > >>>>> mkdir `/run/openrc/hotplugged': No such file or directory > > >>>>> mkdir `/run/openrc/daemons': No such file or directory > > >>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>>>> > > >>>>> I suspect we build bad signal stack frame for userspace. > > >>>>> > > >>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: > > >>>>> > > >>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 > > >>>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 > > >>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 > > >>>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 > > >>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 > > >>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>>>> > > >>>>> Bisect points at: > > >>>>> > > >>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 > > >>>>> Author: Jens Axboe <axboe@kernel.dk> > > >>>>> Date: Fri Oct 9 14:49:43 2020 -0600 > > >>>>> > > >>>>> ia64: add support for TIF_NOTIFY_SIGNAL > > >>>>> > > >>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. > > >>>>> > > >>>>> Cc: linux-ia64@vger.kernel.org > > >>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] > > >>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> > > >>>>> > > >>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h > > >>>>> index 64a1011f6812..51d20cb37706 100644 > > >>>>> --- a/arch/ia64/include/asm/thread_info.h > > >>>>> +++ b/arch/ia64/include/asm/thread_info.h > > >>>>> @@ -103,6 +103,7 @@ struct thread_info { > > >>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ > > >>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ > > >>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ > > >>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ > > >>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ > > >>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ > > >>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ > > >>>>> @@ -115,6 +116,7 @@ struct thread_info { > > >>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) > > >>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) > > >>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) > > >>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) > > >>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) > > >>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) > > >>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) > > >>>>> @@ -124,7 +126,7 @@ struct thread_info { > > >>>>> > > >>>>> /* "work to do on user-return" bits */ > > >>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ > > >>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) > > >>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) > > >>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ > > >>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) > > >>>>> > > >>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c > > >>>>> index 6b61a703bcf5..8d4e1cab9190 100644 > > >>>>> --- a/arch/ia64/kernel/process.c > > >>>>> +++ b/arch/ia64/kernel/process.c > > >>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) > > >>>>> } > > >>>>> > > >>>>> /* deal with pending signal delivery */ > > >>>>> - if (test_thread_flag(TIF_SIGPENDING)) { > > >>>>> + if (test_thread_flag(TIF_SIGPENDING) || > > >>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { > > >>>>> local_irq_enable(); /* force interrupt enable */ > > >>>>> ia64_do_signal(scr, in_syscall); > > >>>>> > > >>>>> which looks benign, but it enables a bit of conditional > > >>>>> TIF_NOTIFY_SIGNAL handling I don't understand. > > >>>>> > > >>>>> Can you help me get what is the interaction between > > >>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for > > >>>>> simple processes without io_uring use case? > > >>>>> > > >>>>> I wonder if it's ia64_do_signal()' generates a signal > > >>>>> delivery when it should not. > > >>>> > > >>>> Can you test: > > >>>> > > >>>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 > > >>>> > > >>>> with the addition mentioned here: > > >>> > > >>> Not enough: > > >>> > > >>> mkdir `/run/openrc': Read-only file system > > >>> mkdir `/run/openrc/starting': No such file or directory > > >>> mkdir `/run/openrc/started': No such file or directory > > >>> mkdir `/run/openrc/stopping': No such file or directory > > >>> mkdir `/run/openrc/inactive': No such file or directory > > >>> mkdir `/run/openrc/wasinactive': No such file or directory > > >>> mkdir `/run/openrc/failed': No such file or directory > > >>> mkdir `/run/openrc/hotplugged': No such file or directory > > >>> mkdir `/run/openrc/daemons': No such file or directory > > >>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b > > >>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- > > >>> mkdir `/run/openrc/options': No such file or directory > > >>> mkdir `/run/openrc/exclusive': No such file or directory > > >>> mkdir `/run/openrc/scheduled': No such file or directory > > >>> mkdir `/run/openrc/tmp': No such file or directory > > >>> > > >>>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 > > >>>> > > >>>> if needed? > > >>> > > >>> Two patches above do fix the boot \o/ But have a lot of spam about > > >>> 'signal 0' delivery to a bunch of processes: > > >>> > > >>> * Mounting /proc ... > > >>> [ ok ] > > >>> * Mounting /run ... > > >>> * /run/openrc: creating directory > > >>> * /run/lock: creating directory > > >>> * /run/lock: correcting owner > > >>> * Caching service dependencies ... > > >>> [ ok ] > > >>> * Mounting /sys ... > > >>> [ ok ] > > >>> * Mounting debug filesystem ... > > >>> [ ok ] > > >>> * Mounting efivarfs filesystem ... > > >>> [ ok ] > > >>> * sysfs: caught unknown signal 0 > > >>> * openrc: caught unknown signal 0 > > >>> * Mounting cgroup filesystem ... > > >> > > >> That's an improvement! Let me take a look at this tonight and see if I > > >> can figure out what's going on. But yes, it's the ia64 signal delivery > > >> being just different enough from the norm that it apparently triggers > > >> some weirdness. > > > > > > Is this any better? > > > > And if that one works, can you try this basic variant? > > Both patches boot successfully without 'caught unknown signal 0' spam \o/ > > > diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c > > index e67b22fc3c60..c1b299760bf7 100644 > > --- a/arch/ia64/kernel/signal.c > > +++ b/arch/ia64/kernel/signal.c > > @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) > > * need to push through a forced SIGSEGV. > > */ > > while (1) { > > - get_signal(&ksig); > > + if (!get_signal(&ksig)) > > + break; > > > > /* > > * get_signal() may have run a debugger (via notify_parent()) > > Should I send the patch in `git am`-able form or the patch already queued up in some other form? -- Sergei ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-03-02 22:07 ` Sergei Trofimovich @ 2021-03-02 22:31 ` Jens Axboe -1 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-02 22:31 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 3:07 PM, Sergei Trofimovich wrote: > On Tue, 23 Feb 2021 08:08:30 +0000 > Sergei Trofimovich <slyich@gmail.com> wrote: > >> On Mon, 22 Feb 2021 17:43:58 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>> >>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>> Hia Jens! >>>>>>>> >>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>> >>>>>>>> INIT: version 2.98 booting >>>>>>>> >>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>> >>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> >>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>> >>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>> >>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> >>>>>>>> Bisect points at: >>>>>>>> >>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>> >>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>> >>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>> >>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>> >>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>> >>>>>>>> /* "work to do on user-return" bits */ >>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>> >>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>> } >>>>>>>> >>>>>>>> /* deal with pending signal delivery */ >>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>> >>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>> >>>>>>>> Can you help me get what is the interaction between >>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>> simple processes without io_uring use case? >>>>>>>> >>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>> delivery when it should not. >>>>>>> >>>>>>> Can you test: >>>>>>> >>>>>>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >>>>>>> >>>>>>> with the addition mentioned here: >>>>>> >>>>>> Not enough: >>>>>> >>>>>> mkdir `/run/openrc': Read-only file system >>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>> >>>>>>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >>>>>>> >>>>>>> if needed? >>>>>> >>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>> 'signal 0' delivery to a bunch of processes: >>>>>> >>>>>> * Mounting /proc ... >>>>>> [ ok ] >>>>>> * Mounting /run ... >>>>>> * /run/openrc: creating directory >>>>>> * /run/lock: creating directory >>>>>> * /run/lock: correcting owner >>>>>> * Caching service dependencies ... >>>>>> [ ok ] >>>>>> * Mounting /sys ... >>>>>> [ ok ] >>>>>> * Mounting debug filesystem ... >>>>>> [ ok ] >>>>>> * Mounting efivarfs filesystem ... >>>>>> [ ok ] >>>>>> * sysfs: caught unknown signal 0 >>>>>> * openrc: caught unknown signal 0 >>>>>> * Mounting cgroup filesystem ... >>>>> >>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>> being just different enough from the norm that it apparently triggers >>>>> some weirdness. >>>> >>>> Is this any better? >>> >>> And if that one works, can you try this basic variant? >> >> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >> >>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>> index e67b22fc3c60..c1b299760bf7 100644 >>> --- a/arch/ia64/kernel/signal.c >>> +++ b/arch/ia64/kernel/signal.c >>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>> * need to push through a forced SIGSEGV. >>> */ >>> while (1) { >>> - get_signal(&ksig); >>> + if (!get_signal(&ksig)) >>> + break; >>> >>> /* >>> * get_signal() may have run a debugger (via notify_parent()) >>> > > Should I send the patch in `git am`-able form or the patch > already queued up in some other form? I'll be happy to queue it up. I take it you tried the above one-liner and it works for you. Just trying to get some clarity on what was tried so I can queue it up appropriately. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-02 22:31 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-02 22:31 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 3:07 PM, Sergei Trofimovich wrote: > On Tue, 23 Feb 2021 08:08:30 +0000 > Sergei Trofimovich <slyich@gmail.com> wrote: > >> On Mon, 22 Feb 2021 17:43:58 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>> >>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>> Hia Jens! >>>>>>>> >>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>> >>>>>>>> INIT: version 2.98 booting >>>>>>>> >>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>> >>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> >>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>> >>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>> >>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> >>>>>>>> Bisect points at: >>>>>>>> >>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>> >>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>> >>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>> >>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>> >>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>> >>>>>>>> /* "work to do on user-return" bits */ >>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>> >>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>> } >>>>>>>> >>>>>>>> /* deal with pending signal delivery */ >>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>> >>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>> >>>>>>>> Can you help me get what is the interaction between >>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>> simple processes without io_uring use case? >>>>>>>> >>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>> delivery when it should not. >>>>>>> >>>>>>> Can you test: >>>>>>> >>>>>>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >>>>>>> >>>>>>> with the addition mentioned here: >>>>>> >>>>>> Not enough: >>>>>> >>>>>> mkdir `/run/openrc': Read-only file system >>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>> >>>>>>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >>>>>>> >>>>>>> if needed? >>>>>> >>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>> 'signal 0' delivery to a bunch of processes: >>>>>> >>>>>> * Mounting /proc ... >>>>>> [ ok ] >>>>>> * Mounting /run ... >>>>>> * /run/openrc: creating directory >>>>>> * /run/lock: creating directory >>>>>> * /run/lock: correcting owner >>>>>> * Caching service dependencies ... >>>>>> [ ok ] >>>>>> * Mounting /sys ... >>>>>> [ ok ] >>>>>> * Mounting debug filesystem ... >>>>>> [ ok ] >>>>>> * Mounting efivarfs filesystem ... >>>>>> [ ok ] >>>>>> * sysfs: caught unknown signal 0 >>>>>> * openrc: caught unknown signal 0 >>>>>> * Mounting cgroup filesystem ... >>>>> >>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>> being just different enough from the norm that it apparently triggers >>>>> some weirdness. >>>> >>>> Is this any better? >>> >>> And if that one works, can you try this basic variant? >> >> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >> >>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>> index e67b22fc3c60..c1b299760bf7 100644 >>> --- a/arch/ia64/kernel/signal.c >>> +++ b/arch/ia64/kernel/signal.c >>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>> * need to push through a forced SIGSEGV. >>> */ >>> while (1) { >>> - get_signal(&ksig); >>> + if (!get_signal(&ksig)) >>> + break; >>> >>> /* >>> * get_signal() may have run a debugger (via notify_parent()) >>> > > Should I send the patch in `git am`-able form or the patch > already queued up in some other form? I'll be happy to queue it up. I take it you tried the above one-liner and it works for you. Just trying to get some clarity on what was tried so I can queue it up appropriately. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
[parent not found: <20210302232716.353ed49b@sf>]
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot [not found] ` <20210302232716.353ed49b@sf> @ 2021-03-03 0:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-03 0:34 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 4:27 PM, Sergei Trofimovich wrote: > On Tue, 2 Mar 2021 15:31:13 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > >> On 3/2/21 3:07 PM, Sergei Trofimovich wrote: >>> On Tue, 23 Feb 2021 08:08:30 +0000 >>> Sergei Trofimovich <slyich@gmail.com> wrote: >>> >>>> On Mon, 22 Feb 2021 17:43:58 -0700 >>>> Jens Axboe <axboe@kernel.dk> wrote: >>>> >>>>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>>>> >>>>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>>>> Hia Jens! >>>>>>>>>> >>>>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>>>> >>>>>>>>>> INIT: version 2.98 booting >>>>>>>>>> >>>>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>>>> >>>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>> >>>>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>>>> >>>>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>>>> >>>>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>> >>>>>>>>>> Bisect points at: >>>>>>>>>> >>>>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>>>> >>>>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>>>> >>>>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>>>> >>>>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>>>> >>>>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>>>> >>>>>>>>>> /* "work to do on user-return" bits */ >>>>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>>>> >>>>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> /* deal with pending signal delivery */ >>>>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>>>> >>>>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>>>> >>>>>>>>>> Can you help me get what is the interaction between >>>>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>>>> simple processes without io_uring use case? >>>>>>>>>> >>>>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>>>> delivery when it should not. >>>>>>>>> >>>>>>>>> Can you test: >>>>>>>>> >>>>>>>>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >>>>>>>>> >>>>>>>>> with the addition mentioned here: >>>>>>>> >>>>>>>> Not enough: >>>>>>>> >>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>>>> >>>>>>>>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >>>>>>>>> >>>>>>>>> if needed? >>>>>>>> >>>>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>>>> 'signal 0' delivery to a bunch of processes: >>>>>>>> >>>>>>>> * Mounting /proc ... >>>>>>>> [ ok ] >>>>>>>> * Mounting /run ... >>>>>>>> * /run/openrc: creating directory >>>>>>>> * /run/lock: creating directory >>>>>>>> * /run/lock: correcting owner >>>>>>>> * Caching service dependencies ... >>>>>>>> [ ok ] >>>>>>>> * Mounting /sys ... >>>>>>>> [ ok ] >>>>>>>> * Mounting debug filesystem ... >>>>>>>> [ ok ] >>>>>>>> * Mounting efivarfs filesystem ... >>>>>>>> [ ok ] >>>>>>>> * sysfs: caught unknown signal 0 >>>>>>>> * openrc: caught unknown signal 0 >>>>>>>> * Mounting cgroup filesystem ... >>>>>>> >>>>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>>>> being just different enough from the norm that it apparently triggers >>>>>>> some weirdness. >>>>>> >>>>>> Is this any better? >>>>> >>>>> And if that one works, can you try this basic variant? >>>> >>>> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >>>> >>>>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>>>> index e67b22fc3c60..c1b299760bf7 100644 >>>>> --- a/arch/ia64/kernel/signal.c >>>>> +++ b/arch/ia64/kernel/signal.c >>>>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>>>> * need to push through a forced SIGSEGV. >>>>> */ >>>>> while (1) { >>>>> - get_signal(&ksig); >>>>> + if (!get_signal(&ksig)) >>>>> + break; >>>>> >>>>> /* >>>>> * get_signal() may have run a debugger (via notify_parent()) >>>>> >>> >>> Should I send the patch in `git am`-able form or the patch >>> already queued up in some other form? >> >> I'll be happy to queue it up. I take it you tried the above one-liner and it >> works for you. Just trying to get some clarity on what was tried so I can >> queue it up appropriately. > > Yes, 1.5 liner with > >>>>> - get_signal(&ksig); >>>>> + if (!get_signal(&ksig)) >>>>> + break; > > made 5.11 boot successfully. > > Tested-by: Sergei Trofimovich <slyich@gmail.com> Awesome! Queued it up: https://git.kernel.dk/cgit/linux-block/commit/?h=misc-5.12&id=f5f4fc4649ae542b1a25670b17aaf3cbb6187acc -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-03 0:34 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-03 0:34 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 4:27 PM, Sergei Trofimovich wrote: > On Tue, 2 Mar 2021 15:31:13 -0700 > Jens Axboe <axboe@kernel.dk> wrote: > >> On 3/2/21 3:07 PM, Sergei Trofimovich wrote: >>> On Tue, 23 Feb 2021 08:08:30 +0000 >>> Sergei Trofimovich <slyich@gmail.com> wrote: >>> >>>> On Mon, 22 Feb 2021 17:43:58 -0700 >>>> Jens Axboe <axboe@kernel.dk> wrote: >>>> >>>>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>>>> >>>>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>>>> Hia Jens! >>>>>>>>>> >>>>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>>>> >>>>>>>>>> INIT: version 2.98 booting >>>>>>>>>> >>>>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>>>> >>>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>> >>>>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>>>> >>>>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>>>> >>>>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>>>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>>>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>> >>>>>>>>>> Bisect points at: >>>>>>>>>> >>>>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>>>> >>>>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>>>> >>>>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>>>> >>>>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>>>> >>>>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>>>> >>>>>>>>>> /* "work to do on user-return" bits */ >>>>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>>>> >>>>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> /* deal with pending signal delivery */ >>>>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>>>> >>>>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>>>> >>>>>>>>>> Can you help me get what is the interaction between >>>>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>>>> simple processes without io_uring use case? >>>>>>>>>> >>>>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>>>> delivery when it should not. >>>>>>>>> >>>>>>>>> Can you test: >>>>>>>>> >>>>>>>>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >>>>>>>>> >>>>>>>>> with the addition mentioned here: >>>>>>>> >>>>>>>> Not enough: >>>>>>>> >>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>>>> >>>>>>>>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >>>>>>>>> >>>>>>>>> if needed? >>>>>>>> >>>>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>>>> 'signal 0' delivery to a bunch of processes: >>>>>>>> >>>>>>>> * Mounting /proc ... >>>>>>>> [ ok ] >>>>>>>> * Mounting /run ... >>>>>>>> * /run/openrc: creating directory >>>>>>>> * /run/lock: creating directory >>>>>>>> * /run/lock: correcting owner >>>>>>>> * Caching service dependencies ... >>>>>>>> [ ok ] >>>>>>>> * Mounting /sys ... >>>>>>>> [ ok ] >>>>>>>> * Mounting debug filesystem ... >>>>>>>> [ ok ] >>>>>>>> * Mounting efivarfs filesystem ... >>>>>>>> [ ok ] >>>>>>>> * sysfs: caught unknown signal 0 >>>>>>>> * openrc: caught unknown signal 0 >>>>>>>> * Mounting cgroup filesystem ... >>>>>>> >>>>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>>>> being just different enough from the norm that it apparently triggers >>>>>>> some weirdness. >>>>>> >>>>>> Is this any better? >>>>> >>>>> And if that one works, can you try this basic variant? >>>> >>>> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >>>> >>>>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>>>> index e67b22fc3c60..c1b299760bf7 100644 >>>>> --- a/arch/ia64/kernel/signal.c >>>>> +++ b/arch/ia64/kernel/signal.c >>>>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>>>> * need to push through a forced SIGSEGV. >>>>> */ >>>>> while (1) { >>>>> - get_signal(&ksig); >>>>> + if (!get_signal(&ksig)) >>>>> + break; >>>>> >>>>> /* >>>>> * get_signal() may have run a debugger (via notify_parent()) >>>>> >>> >>> Should I send the patch in `git am`-able form or the patch >>> already queued up in some other form? >> >> I'll be happy to queue it up. I take it you tried the above one-liner and it >> works for you. Just trying to get some clarity on what was tried so I can >> queue it up appropriately. > > Yes, 1.5 liner with > >>>>> - get_signal(&ksig); >>>>> + if (!get_signal(&ksig)) >>>>> + break; > > made 5.11 boot successfully. > > Tested-by: Sergei Trofimovich <slyich@gmail.com> Awesome! Queued it up: https://git.kernel.dk/cgit/linux-block/commit/?h=misc-5.12&idõf4fc4649ae542b1a25670b17aaf3cbb6187acc -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot 2021-03-03 0:34 ` Jens Axboe @ 2021-03-03 3:51 ` Jens Axboe -1 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-03 3:51 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 5:34 PM, Jens Axboe wrote: > On 3/2/21 4:27 PM, Sergei Trofimovich wrote: >> On Tue, 2 Mar 2021 15:31:13 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 3/2/21 3:07 PM, Sergei Trofimovich wrote: >>>> On Tue, 23 Feb 2021 08:08:30 +0000 >>>> Sergei Trofimovich <slyich@gmail.com> wrote: >>>> >>>>> On Mon, 22 Feb 2021 17:43:58 -0700 >>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>> >>>>>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>>>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>>>>> >>>>>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>>>>> Hia Jens! >>>>>>>>>>> >>>>>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>>>>> >>>>>>>>>>> INIT: version 2.98 booting >>>>>>>>>>> >>>>>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>>>>> >>>>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>>> >>>>>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>>>>> >>>>>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>>>>> >>>>>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6aeaa0 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>>> [ 34.969948] SIG deliver (init:1): sig=17 sp=60000fffff1ccc50 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6adf90 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>>> [ 34.973948] SIG deliver (init:1): sig=17 sp=60000fffff1cc140 ip=a000000000040740 handler=000000004638b9e5 >>>>>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig=17 sp=60000fffff6ad480 ip=a000000000040740 handler=000000004b4c59b6 >>>>>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>>> >>>>>>>>>>> Bisect points at: >>>>>>>>>>> >>>>>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>>>>> >>>>>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>>>>> >>>>>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>>>>> >>>>>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>>>>> >>>>>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>>>>> >>>>>>>>>>> /* "work to do on user-return" bits */ >>>>>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>>>>> >>>>>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> /* deal with pending signal delivery */ >>>>>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>>>>> >>>>>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>>>>> >>>>>>>>>>> Can you help me get what is the interaction between >>>>>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>>>>> simple processes without io_uring use case? >>>>>>>>>>> >>>>>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>>>>> delivery when it should not. >>>>>>>>>> >>>>>>>>>> Can you test: >>>>>>>>>> >>>>>>>>>> https://marc.info/?l=linux-ia64&m=161187407609443&w=1 >>>>>>>>>> >>>>>>>>>> with the addition mentioned here: >>>>>>>>> >>>>>>>>> Not enough: >>>>>>>>> >>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>>>>> >>>>>>>>>> https://marc.info/?l=linux-ia64&m=161187470709706&w=1 >>>>>>>>>> >>>>>>>>>> if needed? >>>>>>>>> >>>>>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>>>>> 'signal 0' delivery to a bunch of processes: >>>>>>>>> >>>>>>>>> * Mounting /proc ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting /run ... >>>>>>>>> * /run/openrc: creating directory >>>>>>>>> * /run/lock: creating directory >>>>>>>>> * /run/lock: correcting owner >>>>>>>>> * Caching service dependencies ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting /sys ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting debug filesystem ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting efivarfs filesystem ... >>>>>>>>> [ ok ] >>>>>>>>> * sysfs: caught unknown signal 0 >>>>>>>>> * openrc: caught unknown signal 0 >>>>>>>>> * Mounting cgroup filesystem ... >>>>>>>> >>>>>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>>>>> being just different enough from the norm that it apparently triggers >>>>>>>> some weirdness. >>>>>>> >>>>>>> Is this any better? >>>>>> >>>>>> And if that one works, can you try this basic variant? >>>>> >>>>> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >>>>> >>>>>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>>>>> index e67b22fc3c60..c1b299760bf7 100644 >>>>>> --- a/arch/ia64/kernel/signal.c >>>>>> +++ b/arch/ia64/kernel/signal.c >>>>>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>>>>> * need to push through a forced SIGSEGV. >>>>>> */ >>>>>> while (1) { >>>>>> - get_signal(&ksig); >>>>>> + if (!get_signal(&ksig)) >>>>>> + break; >>>>>> >>>>>> /* >>>>>> * get_signal() may have run a debugger (via notify_parent()) >>>>>> >>>> >>>> Should I send the patch in `git am`-able form or the patch >>>> already queued up in some other form? >>> >>> I'll be happy to queue it up. I take it you tried the above one-liner and it >>> works for you. Just trying to get some clarity on what was tried so I can >>> queue it up appropriately. >> >> Yes, 1.5 liner with >> >>>>>> - get_signal(&ksig); >>>>>> + if (!get_signal(&ksig)) >>>>>> + break; >> >> made 5.11 boot successfully. >> >> Tested-by: Sergei Trofimovich <slyich@gmail.com> > > Awesome! Queued it up: > > https://git.kernel.dk/cgit/linux-block/commit/?h=misc-5.12&id=f5f4fc4649ae542b1a25670b17aaf3cbb6187acc This is now in Linus's tree. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
* Re: 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot @ 2021-03-03 3:51 ` Jens Axboe 0 siblings, 0 replies; 89+ messages in thread From: Jens Axboe @ 2021-03-03 3:51 UTC (permalink / raw) To: Sergei Trofimovich; +Cc: linux-ia64, linux-kernel On 3/2/21 5:34 PM, Jens Axboe wrote: > On 3/2/21 4:27 PM, Sergei Trofimovich wrote: >> On Tue, 2 Mar 2021 15:31:13 -0700 >> Jens Axboe <axboe@kernel.dk> wrote: >> >>> On 3/2/21 3:07 PM, Sergei Trofimovich wrote: >>>> On Tue, 23 Feb 2021 08:08:30 +0000 >>>> Sergei Trofimovich <slyich@gmail.com> wrote: >>>> >>>>> On Mon, 22 Feb 2021 17:43:58 -0700 >>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>> >>>>>> On 2/22/21 5:41 PM, Jens Axboe wrote: >>>>>>> On 2/22/21 5:34 PM, Jens Axboe wrote: >>>>>>>> On 2/22/21 4:53 PM, Sergei Trofimovich wrote: >>>>>>>>> On Mon, 22 Feb 2021 16:34:50 -0700 >>>>>>>>> Jens Axboe <axboe@kernel.dk> wrote: >>>>>>>>> >>>>>>>>>> On 2/22/21 4:05 PM, Sergei Trofimovich wrote: >>>>>>>>>>> Hia Jens! >>>>>>>>>>> >>>>>>>>>>> Tried 5.11 on rx3600 box and noticed it has >>>>>>>>>>> a problem handling init (5.10 booted fine): >>>>>>>>>>> >>>>>>>>>>> INIT: version 2.98 booting >>>>>>>>>>> >>>>>>>>>>> OpenRC 0.42.1 is starting up Gentoo Linux (ia64) >>>>>>>>>>> >>>>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>>>> mkdir `/run[ 14.595059] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>>> [ 14.599059] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>>> >>>>>>>>>>> I suspect we build bad signal stack frame for userspace. >>>>>>>>>>> >>>>>>>>>>> With a bit of #define DEBUG_SIG 1 enabled the signals are SIGCHLD: >>>>>>>>>>> >>>>>>>>>>> [ 34.969771] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6aeaa0 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>>> [ 34.969948] SIG deliver (init:1): sig\x17 sp`000fffff1ccc50 ip 00000000040740 handler\00000004638b9e5 >>>>>>>>>>> [ 34.969948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6adf90 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>>> [ 34.973948] SIG deliver (init:1): sig\x17 sp`000fffff1cc140 ip 00000000040740 handler\00000004638b9e5 >>>>>>>>>>> [ 34.973948] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>>>> [ 34.973948] SIG deliver (gendepends.sh:69): sig\x17 sp`000fffff6ad480 ip 00000000040740 handler\00000004b4c59b6 >>>>>>>>>>> [ 34.973948] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>>>> >>>>>>>>>>> Bisect points at: >>>>>>>>>>> >>>>>>>>>>> commit b269c229b0e89aedb7943c06673b56b6052cf5e5 >>>>>>>>>>> Author: Jens Axboe <axboe@kernel.dk> >>>>>>>>>>> Date: Fri Oct 9 14:49:43 2020 -0600 >>>>>>>>>>> >>>>>>>>>>> ia64: add support for TIF_NOTIFY_SIGNAL >>>>>>>>>>> >>>>>>>>>>> Wire up TIF_NOTIFY_SIGNAL handling for ia64. >>>>>>>>>>> >>>>>>>>>>> Cc: linux-ia64@vger.kernel.org >>>>>>>>>>> [axboe: added fixes from Mike Rapoport <rppt@kernel.org>] >>>>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>>>>>>>>> >>>>>>>>>>> diff --git a/arch/ia64/include/asm/thread_info.h b/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> index 64a1011f6812..51d20cb37706 100644 >>>>>>>>>>> --- a/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> +++ b/arch/ia64/include/asm/thread_info.h >>>>>>>>>>> @@ -103,6 +103,7 @@ struct thread_info { >>>>>>>>>>> #define TIF_SYSCALL_TRACE 2 /* syscall trace active */ >>>>>>>>>>> #define TIF_SYSCALL_AUDIT 3 /* syscall auditing active */ >>>>>>>>>>> #define TIF_SINGLESTEP 4 /* restore singlestep on return to user mode */ >>>>>>>>>>> +#define TIF_NOTIFY_SIGNAL 5 /* signal notification exist */ >>>>>>>>>>> #define TIF_NOTIFY_RESUME 6 /* resumption notification requested */ >>>>>>>>>>> #define TIF_MEMDIE 17 /* is terminating due to OOM killer */ >>>>>>>>>>> #define TIF_MCA_INIT 18 /* this task is processing MCA or INIT */ >>>>>>>>>>> @@ -115,6 +116,7 @@ struct thread_info { >>>>>>>>>>> #define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP) >>>>>>>>>>> #define _TIF_SYSCALL_TRACEAUDIT (_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SINGLESTEP) >>>>>>>>>>> #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) >>>>>>>>>>> +#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) >>>>>>>>>>> #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) >>>>>>>>>>> #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) >>>>>>>>>>> #define _TIF_MCA_INIT (1 << TIF_MCA_INIT) >>>>>>>>>>> @@ -124,7 +126,7 @@ struct thread_info { >>>>>>>>>>> >>>>>>>>>>> /* "work to do on user-return" bits */ >>>>>>>>>>> #define TIF_ALLWORK_MASK (_TIF_SIGPENDING|_TIF_NOTIFY_RESUME|_TIF_SYSCALL_AUDIT|\ >>>>>>>>>>> - _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE) >>>>>>>>>>> + _TIF_NEED_RESCHED|_TIF_SYSCALL_TRACE|_TIF_NOTIFY_SIGNAL) >>>>>>>>>>> /* like TIF_ALLWORK_BITS but sans TIF_SYSCALL_TRACE or TIF_SYSCALL_AUDIT */ >>>>>>>>>>> #define TIF_WORK_MASK (TIF_ALLWORK_MASK&~(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT)) >>>>>>>>>>> >>>>>>>>>>> diff --git a/arch/ia64/kernel/process.c b/arch/ia64/kernel/process.c >>>>>>>>>>> index 6b61a703bcf5..8d4e1cab9190 100644 >>>>>>>>>>> --- a/arch/ia64/kernel/process.c >>>>>>>>>>> +++ b/arch/ia64/kernel/process.c >>>>>>>>>>> @@ -171,7 +171,8 @@ do_notify_resume_user(sigset_t *unused, struct sigscratch *scr, long in_syscall) >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> /* deal with pending signal delivery */ >>>>>>>>>>> - if (test_thread_flag(TIF_SIGPENDING)) { >>>>>>>>>>> + if (test_thread_flag(TIF_SIGPENDING) || >>>>>>>>>>> + test_thread_flag(TIF_NOTIFY_SIGNAL)) { >>>>>>>>>>> local_irq_enable(); /* force interrupt enable */ >>>>>>>>>>> ia64_do_signal(scr, in_syscall); >>>>>>>>>>> >>>>>>>>>>> which looks benign, but it enables a bit of conditional >>>>>>>>>>> TIF_NOTIFY_SIGNAL handling I don't understand. >>>>>>>>>>> >>>>>>>>>>> Can you help me get what is the interaction between >>>>>>>>>>> TIF_NOTIFY_SIGNAL and TIF_SIGPENDING for >>>>>>>>>>> simple processes without io_uring use case? >>>>>>>>>>> >>>>>>>>>>> I wonder if it's ia64_do_signal()' generates a signal >>>>>>>>>>> delivery when it should not. >>>>>>>>>> >>>>>>>>>> Can you test: >>>>>>>>>> >>>>>>>>>> https://marc.info/?l=linux-ia64&m\x161187407609443&w=1 >>>>>>>>>> >>>>>>>>>> with the addition mentioned here: >>>>>>>>> >>>>>>>>> Not enough: >>>>>>>>> >>>>>>>>> mkdir `/run/openrc': Read-only file system >>>>>>>>> mkdir `/run/openrc/starting': No such file or directory >>>>>>>>> mkdir `/run/openrc/started': No such file or directory >>>>>>>>> mkdir `/run/openrc/stopping': No such file or directory >>>>>>>>> mkdir `/run/openrc/inactive': No such file or directory >>>>>>>>> mkdir `/run/openrc/wasinactive': No such file or directory >>>>>>>>> mkdir `/run/openrc/failed': No such file or directory >>>>>>>>> mkdir `/run/openrc/hotplugged': No such file or directory >>>>>>>>> mkdir `/run/openrc/daemons': No such file or directory >>>>>>>>> [ 14.554357] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b >>>>>>>>> [ 14.554357] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]--- >>>>>>>>> mkdir `/run/openrc/options': No such file or directory >>>>>>>>> mkdir `/run/openrc/exclusive': No such file or directory >>>>>>>>> mkdir `/run/openrc/scheduled': No such file or directory >>>>>>>>> mkdir `/run/openrc/tmp': No such file or directory >>>>>>>>> >>>>>>>>>> https://marc.info/?l=linux-ia64&m\x161187470709706&w=1 >>>>>>>>>> >>>>>>>>>> if needed? >>>>>>>>> >>>>>>>>> Two patches above do fix the boot \o/ But have a lot of spam about >>>>>>>>> 'signal 0' delivery to a bunch of processes: >>>>>>>>> >>>>>>>>> * Mounting /proc ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting /run ... >>>>>>>>> * /run/openrc: creating directory >>>>>>>>> * /run/lock: creating directory >>>>>>>>> * /run/lock: correcting owner >>>>>>>>> * Caching service dependencies ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting /sys ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting debug filesystem ... >>>>>>>>> [ ok ] >>>>>>>>> * Mounting efivarfs filesystem ... >>>>>>>>> [ ok ] >>>>>>>>> * sysfs: caught unknown signal 0 >>>>>>>>> * openrc: caught unknown signal 0 >>>>>>>>> * Mounting cgroup filesystem ... >>>>>>>> >>>>>>>> That's an improvement! Let me take a look at this tonight and see if I >>>>>>>> can figure out what's going on. But yes, it's the ia64 signal delivery >>>>>>>> being just different enough from the norm that it apparently triggers >>>>>>>> some weirdness. >>>>>>> >>>>>>> Is this any better? >>>>>> >>>>>> And if that one works, can you try this basic variant? >>>>> >>>>> Both patches boot successfully without 'caught unknown signal 0' spam \o/ >>>>> >>>>>> diff --git a/arch/ia64/kernel/signal.c b/arch/ia64/kernel/signal.c >>>>>> index e67b22fc3c60..c1b299760bf7 100644 >>>>>> --- a/arch/ia64/kernel/signal.c >>>>>> +++ b/arch/ia64/kernel/signal.c >>>>>> @@ -341,7 +341,8 @@ ia64_do_signal (struct sigscratch *scr, long in_syscall) >>>>>> * need to push through a forced SIGSEGV. >>>>>> */ >>>>>> while (1) { >>>>>> - get_signal(&ksig); >>>>>> + if (!get_signal(&ksig)) >>>>>> + break; >>>>>> >>>>>> /* >>>>>> * get_signal() may have run a debugger (via notify_parent()) >>>>>> >>>> >>>> Should I send the patch in `git am`-able form or the patch >>>> already queued up in some other form? >>> >>> I'll be happy to queue it up. I take it you tried the above one-liner and it >>> works for you. Just trying to get some clarity on what was tried so I can >>> queue it up appropriately. >> >> Yes, 1.5 liner with >> >>>>>> - get_signal(&ksig); >>>>>> + if (!get_signal(&ksig)) >>>>>> + break; >> >> made 5.11 boot successfully. >> >> Tested-by: Sergei Trofimovich <slyich@gmail.com> > > Awesome! Queued it up: > > https://git.kernel.dk/cgit/linux-block/commit/?h=misc-5.12&idõf4fc4649ae542b1a25670b17aaf3cbb6187acc This is now in Linus's tree. -- Jens Axboe ^ permalink raw reply [flat|nested] 89+ messages in thread
end of thread, other threads:[~2021-04-15 18:41 UTC | newest]
Thread overview: 89+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20210222230519.73f3e239@sf>
2021-02-22 23:34 ` 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot Jens Axboe
2021-02-22 23:34 ` Jens Axboe
2021-02-22 23:55 ` John Paul Adrian Glaubitz
2021-02-22 23:55 ` John Paul Adrian Glaubitz
[not found] ` <20210223083507.43b5a6dd@sf>
2021-02-23 9:13 ` John Paul Adrian Glaubitz
2021-02-23 12:36 ` John Paul Adrian Glaubitz
2021-02-23 12:36 ` John Paul Adrian Glaubitz
[not found] ` <20210223192743.0198d4a9@sf>
[not found] ` <20210302222630.5056f243@sf>
2021-03-02 22:31 ` John Paul Adrian Glaubitz
2021-03-02 22:31 ` John Paul Adrian Glaubitz
2021-03-03 0:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Sergei Trofimovich
2021-03-03 0:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" bre Sergei Trofimovich
2021-03-03 8:55 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Sergei Trofimovich
2021-03-03 8:55 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Sergei Trofimovich
2021-03-03 17:33 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Don.Brace
2021-03-03 17:33 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace
[not found] ` <20210303220401.501449e5@sf>
2021-03-04 17:00 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Don.Brace
2021-03-04 17:00 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace
2021-03-05 13:26 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Tomas Henzl
2021-03-05 13:26 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Tomas Henzl
2021-03-12 22:27 ` [PATCH] hpsa: fix boot on ia64 (atomic_t alignment) Sergei Trofimovich
2021-03-12 22:27 ` Sergei Trofimovich
2021-03-16 16:30 ` Don.Brace
2021-03-16 16:30 ` Don.Brace
2021-03-16 18:28 ` Arnd Bergmann
2021-03-16 18:28 ` Arnd Bergmann
2021-03-17 2:25 ` Martin K. Petersen
2021-03-17 2:25 ` Martin K. Petersen
2021-03-17 13:19 ` David Laight
2021-03-17 19:06 ` Don.Brace
2021-03-17 19:06 ` Don.Brace
2021-03-17 17:28 ` John Paul Adrian Glaubitz
2021-03-17 17:28 ` John Paul Adrian Glaubitz
2021-03-27 10:24 ` Sergei Trofimovich
2021-03-24 7:08 ` John Paul Adrian Glaubitz
2021-03-24 7:08 ` John Paul Adrian Glaubitz
2021-03-24 18:37 ` Don.Brace
2021-03-24 18:37 ` Don.Brace
2021-03-29 11:25 ` John Paul Adrian Glaubitz
2021-03-29 11:25 ` John Paul Adrian Glaubitz
2021-03-29 14:22 ` Arnd Bergmann
2021-03-29 14:22 ` Arnd Bergmann
2021-03-30 3:02 ` Martin K. Petersen
2021-03-30 3:02 ` Martin K. Petersen
2021-03-30 7:19 ` [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide Sergei Trofimovich
2021-03-30 7:19 ` Sergei Trofimovich
2021-03-30 7:19 ` [PATCH v2 2/3] hpsa: fix boot on ia64 (atomic_t alignment) Sergei Trofimovich
2021-03-30 7:19 ` Sergei Trofimovich
2021-03-30 7:19 ` [PATCH v2 3/3] hpsa: add an assert to prevent from __packed reintroduction Sergei Trofimovich
2021-03-30 7:19 ` Sergei Trofimovich
2021-03-30 7:34 ` Arnd Bergmann
2021-03-30 7:34 ` Arnd Bergmann
2021-04-02 14:40 ` Elliott, Robert (Servers)
2021-04-02 14:40 ` Elliott, Robert (Servers)
2021-04-03 14:51 ` Sergei Trofimovich
2021-04-03 14:51 ` Sergei Trofimovich
2021-03-30 7:30 ` [PATCH v2 1/3] hpsa: use __packed on individual structs, not header-wide Arnd Bergmann
2021-03-30 7:30 ` Arnd Bergmann
2021-03-30 7:43 ` Arnd Bergmann
2021-03-30 7:43 ` Arnd Bergmann
2021-04-02 3:54 ` Martin K. Petersen
2021-04-02 3:54 ` Martin K. Petersen
2021-04-15 18:41 ` Don.Brace
2021-04-15 18:41 ` Don.Brace
2021-03-05 9:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Geert Uytterhoeven
2021-03-05 9:22 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Geert Uytterhoeven
2021-03-05 13:31 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Arnd Bergmann
2021-03-05 13:31 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Arnd Bergmann
2021-03-05 20:45 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Don.Brace
2021-03-05 20:45 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace
2021-03-03 15:42 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" breaks hpsa P600 Don.Brace
2021-03-03 15:42 ` [bisected] 5.12-rc1 hpsa regression: "scsi: hpsa: Correct dev cmds outstanding for retried cmds" Don.Brace
2021-03-17 17:42 ` 5.11 regression: "ia64: add support for TIF_NOTIFY_SIGNAL" breaks ia64 boot John Paul Adrian Glaubitz
2021-03-17 17:42 ` John Paul Adrian Glaubitz
2021-03-17 17:53 ` John Paul Adrian Glaubitz
2021-03-17 17:53 ` John Paul Adrian Glaubitz
[not found] ` <20210222235359.75d1a912@sf>
2021-02-23 0:34 ` Jens Axboe
2021-02-23 0:34 ` Jens Axboe
2021-02-23 0:41 ` Jens Axboe
2021-02-23 0:41 ` Jens Axboe
2021-02-23 0:43 ` Jens Axboe
2021-02-23 0:43 ` Jens Axboe
[not found] ` <20210223080830.23bccdbf@sf>
2021-03-02 22:07 ` Sergei Trofimovich
2021-03-02 22:07 ` Sergei Trofimovich
2021-03-02 22:31 ` Jens Axboe
2021-03-02 22:31 ` Jens Axboe
[not found] ` <20210302232716.353ed49b@sf>
2021-03-03 0:34 ` Jens Axboe
2021-03-03 0:34 ` Jens Axboe
2021-03-03 3:51 ` Jens Axboe
2021-03-03 3:51 ` Jens Axboe
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.