Re: SGI-IP30

[Joshua Kinard <kumba@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 4939 bytes --]

On 6/24/2019 13:55, Paul Burton wrote:
> Hi Carlo,
> 
> On Sat, Jun 22, 2019 at 02:42:05PM +0200, Carlo Pisani wrote:
>> hi guys
>> I am new on this mailing list, is there anyone on SGI-IP30?
>> I am with kernel 4.18, there are a few quirks ... with the SMP
>>
>> =)
> 
> Since we don't have IP30 support upstream I presume you must be getting
> your kernel source from somewhere else?
> 
> I'm afraid I don't have access to any of these old SGI systems myself,
> but your best chances are probably:
> 
> - Thomas Bodendoerfer who's recently done some work refactoring IP27
>   code in an effort to also support IP30 & other systems.
> 
> - Maciej W. Rozycki who seems to have an affinity for MIPS machines from
>   this era, or at least plenty of knowledge about them :)
> 
> - Joshua Kinard who had SMP issues with IP30 over here:
>   https://lore.kernel.org/linux-mips/5457187D.6030708@gentoo.org/T/#u
>   That was 5 years ago though, so may be a long shot!
> 
> Thanks,
>     Paul


SMP works fine, on my machine at least.  I got it working back in 4.4 a few
years ago, and it's largely just worked fine since then on the one machine I
have.

See attached dmesg.  Below is the output of cat /proc/cpuinfo and cat
/proc/interrupts

# cat /proc/cpuinfo
system type             : SGI Octane
machine                 : Unknown
processor               : 0
cpu model               : R14000 V2.4  FPU V0.0
BogoMIPS                : 898.66
byteorder               : big endian
wait instruction        : no
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : no
hardware watchpoint     : yes, count: 0, address/irw mask: []
isa                     : mips1 mips2 mips3 mips4
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
package                 : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : R14000 V2.4  FPU V0.0
BogoMIPS                : 898.66
byteorder               : big endian
wait instruction        : no
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : no
hardware watchpoint     : yes, count: 0, address/irw mask: []
isa                     : mips1 mips2 mips3 mips4
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
package                 : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

# cat /proc/interrupts
           CPU0       CPU1
  0:      21151          0     HEART      qla1280
  1:      16904          0     HEART      qla1280
  2:         30          0     HEART      ioc3-eth
  3:        244          0     HEART      ioc3-io
  5:        433          0     HEART      acenic
 42:       3066          0     HEART      cpu0-rsch
 43:          0       4194     HEART      cpu1-rsch
 46:       1328          0     HEART      cpu0-call
 47:          0       3059     HEART      cpu1-call
 50:      31187          0     HEART      heart_timer
 71:         12      25426  MIPS_R10K      timer
ERR:          2

I poured over IRIX headers and even found Stan a few years back on Freenode,
and his recollection was that the CPU interrupts for SMP were not hardwired
to any specific function.  IRIX just chose to use 46-49 for symmon
debugging, and 42-45 for CPU IPI.  So I went with 42-45 for SMP reschedule
and 46-49 for SMP call_function in Linux.  Been working fine for the last
few years.  I even implemented similar logic in IP27's SMP code, so that its
IPI's could be tracked by Linux's IRQ subsystem (current IP27 code does
IPI's in a way that Linux can't track in /proc/interrupts)

I've been running 4.18.20 for the last few months.  4.19 is mostly synced up
to the patches I maintain in Gentoo, but the DMA changes that dropped have
caused some headaches that I haven't worked out yet.  5.0/5.1 is still a
ways off, but I assume will be easier than 4.19 until Thomas' IP27 changes
drop in 5.2.  No timeframe when I'll be able to catch up due to being
swamped with work.

My Octane is probably the most stable non-x86 platform I've got.  Ran it for
almost ~2 months straight for compiling Gentoo's install stages before a
thunderstorm knocked the power out, ending that uptime run.  So I can vouch
that the code should work.  I can't test all combinations of Octane's
hardware, but the base hardware I have will boot Linux and is quite usable.
 There are many known issues w/ PCI, some of which I hope Thomas' patches
will address that I can build my patches off of again.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[-- Attachment #2: ip30-dmesg-20190626.txt --]
[-- Type: text/plain, Size: 23624 bytes --]

[    0.000000] Linux version 4.18.20-mipsgit-20181112 (root@helcaraxe) (gcc version 9.1.0 (Gentoo 9.1.0 p1.0)) #4 SMP Mon May 13 03:59:58 EDT 2019
[    0.000000] ARCH: SGI-IP30
[    0.000000] PROMLIB: ARC firmware Version 64 Revision 0
[    0.000000] bootconsole [early_impact0] enabled
[    0.000000] CPU0 revision is: 00000f24 (R14000)
[    0.000000] FPU revision is: 00000900
[    0.000000] Checking for the multiply/shift bug... no.
[    0.000000] Checking for the daddiu bug... no.
[    0.000000] Detected 2048MB of physical memory.
[    0.000000] IP30: CPU0: 600 MHz CPU detected.
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 0000000000004000 @ 0000000000000000 (reserved)
[    0.000000]  memory: 0000000000acd000 @ 0000000020004000 (reserved)
[    0.000000]  memory: 000000000042f000 @ 0000000020ad1000 (usable)
[    0.000000]  memory: 0000000000100000 @ 0000000020f00000 (ROM data)
[    0.000000]  memory: 000000007f000000 @ 0000000021000000 (usable)
[    0.000000]  memory: 0000000000990000 @ 0000000020000000 (usable)
[    0.000000] cma: Reserved 128 MiB at 0x0000000098000000
[    0.000000] IP30: Slot: 0, PrID: 00000f24, PhyID: 0, VirtID: 0
[    0.000000] IP30: Slot: 1, PrID: 00000f24, PhyID: 1, VirtID: 1
[    0.000000] IP30: Detected 2 CPU(s) present.
[    0.000000] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    0.000000] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    0.000000] Unified secondary cache 2048kB 2-way, linesize 128 bytes.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000020000000-0x00000000ffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000020000000-0x0000000020acffff]
[    0.000000]   node   0: [mem 0x0000000020ae0000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000020000000-0x000000009fffffff]
[    0.000000] On node 0 totalpages: 32767
[    0.000000]   DMA32 zone: 32 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 32767 pages, LIFO batch:1
[    0.000000] random: get_random_bytes called from start_kernel+0xdc/0x64c with crng_init=0
[    0.000000] percpu: Embedded 2 pages/cpu @(____ptrval____) s44832 r0 d86240 u131072
[    0.000000] pcpu-alloc: s44832 r0 d86240 u131072 alloc=2*65536
[    0.000000] pcpu-alloc: [0] 0 [0] 1
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32735
[    0.000000] Kernel command line: root=xio(0)pci(15)scsi(0)disk(1)rdisk(0)partition(0) console=tty0 root=/dev/md0 consoleblank=0
[    0.000000] Dentry cache hash table entries: 262144 (order: 5, 2097152 bytes)
[    0.000000] Inode-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[    0.000000] Memory: 1948352K/2097088K available (7338K kernel code, 735K rwdata, 1680K rodata, 384K init, 898K bss, 17664K reserved, 131072K cma-reserved)
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] NR_IRQS: 128
[    0.000000] IP30: HEART interrupt controller initialized.
[    0.000000]  Offload RCU callbacks from CPUs: (none).
[    0.000000] clocksource: HEART: mask: 0xfffffffffffff max_cycles: 0x2e2049cda, max_idle_ns: 440795202628 ns
[    0.000008] sched_clock: 52 bits at 12MHz, resolution 80ns, wraps every 4398046511080ns
[    0.000559] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6369616738 ns
[    0.001095] sched_clock: 32 bits at 300MHz, resolution 3ns, wraps every 7156872702ns
[    0.002426] Console: colour dummy device 80x25
[    0.002784] console [tty0] enabled
[    0.003096] bootconsole [early_impact0] disabled
[    0.003482] Calibrating delay loop... 898.66 BogoMIPS (lpj=4493312)
[    0.090707] pid_max: default: 32768 minimum: 301
[    0.091568] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.091758] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes)
[    0.093344] Checking for the daddi bug... no.
[    0.096295] Hierarchical SRCU implementation.
[    0.097808] smp: Bringing up secondary CPUs ...
[    5.879607] Primary instruction cache 32kB, VIPT, 2-way, linesize 64 bytes.
[    5.879623] Primary data cache 32kB, 2-way, VIPT, no aliases, linesize 32 bytes
[    5.879629] Unified secondary cache 2048kB 2-way, linesize 128 bytes.
[   14.264095] IP30: CPU1: 600 MHz CPU detected.
[   14.264146] CPU1 revision is: 00000f24 (R14000)
[   14.264150] FPU revision is: 00000900
[    0.230888] Synchronize counters for CPU 1: done.
[    0.231229] smp: Brought up 1 node, 2 CPUs
[    0.233037] devtmpfs: initialized
[    0.234195] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.234238] futex hash table entries: 512 (order: 0, 65536 bytes)
[    0.235269] xor: measuring software checksum speed
[    0.330923]    8regs     :   844.800 MB/sec
[    0.431032]    8regs_prefetch:   806.400 MB/sec
[    0.531121]    32regs    :   723.200 MB/sec
[    0.631219]    32regs_prefetch:   748.800 MB/sec
[    0.631253] xor: using function: 8regs (844.800 MB/sec)
[    0.632049] NET: Registered protocol family 16
[    0.641306] cpuidle: using governor menu
[    0.641486] xtalk:n0/f bridge widget (rev D) registered as a platform device.
[    0.641564] xtalk:n0/d bridge widget (rev C) registered as a platform device.
[    0.641642] xtalk:n0/c impact widget (rev B) registered as a platform device.
[    0.641721] xtalk:n0/8 heart widget (rev F) registered as a platform device.
[    0.703867] cryptd: max_cpu_qlen set to 1000
[    0.871683] raid6: int64x1  gen()   333 MB/s
[    1.041946] raid6: int64x1  xor()    82 MB/s
[    1.211958] raid6: int64x2  gen()   487 MB/s
[    1.371663] random: fast init done
[    1.382173] raid6: int64x2  xor()   164 MB/s
[    1.552455] raid6: int64x4  gen()   460 MB/s
[    1.722652] raid6: int64x4  xor()   171 MB/s
[    1.892775] raid6: int64x8  gen()   299 MB/s
[    2.063209] raid6: int64x8  xor()   129 MB/s
[    2.063241] raid6: using algorithm int64x2 gen() 487 MB/s
[    2.063269] raid6: .... xor() 164 MB/s, rmw enabled
[    2.063297] raid6: using intx1 recovery algorithm
[    2.063774] SCSI subsystem initialized
[    2.064751] Advanced Linux Sound Architecture Driver Initialized.
[    2.065551] clocksource: Switched to clocksource HEART
[    2.065850] FS-Cache: Loaded
[    2.066097] CacheFiles: Loaded
[    2.075037] NET: Registered protocol family 2
[    2.075720] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes)
[    2.076023] TCP established hash table entries: 16384 (order: 1, 131072 bytes)
[    2.076625] TCP bind hash table entries: 16384 (order: 2, 262144 bytes)
[    2.077605] TCP: Hash tables configured (established 16384 bind 16384)
[    2.077861] UDP hash table entries: 2048 (order: 0, 65536 bytes)
[    2.078147] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes)
[    2.078624] NET: Registered protocol family 1
[    2.079804] RPC: Registered named UNIX socket transport module.
[    2.079839] RPC: Registered udp transport module.
[    2.079866] RPC: Registered tcp transport module.
[    2.079894] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.079933] PCI: CLS 0 bytes, default 128
[    2.081099] workingset: timestamp_bits=46 max_order=15 bucket_order=0
[    2.081243] zbud: loaded
[    2.082560] DLM installed
[    2.082673] FS-Cache: Netfs 'nfs' registered for caching
[    2.083225] NFS: Registering the id_resolver key type
[    2.083295] Key type id_resolver registered
[    2.083324] Key type id_legacy registered
[    2.083430] SGI XFS with ACLs, security attributes, realtime, no debug enabled
[    2.106459] NET: Registered protocol family 38
[    2.106946] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    2.107003] io scheduler noop registered
[    2.107032] io scheduler deadline registered (default)
[    2.107061] io scheduler mq-deadline registered (default)
[    2.145475] Console: switching to colour frame buffer device 160x64
[    2.177051] fb0: ImpactSR 1RSS frame buffer device
[    2.218447] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.236248] loop: module loaded
[    2.237068] mousedev: PS/2 mouse device common for all mice
[    2.238623] PCI host bridge to bus 0000:00
[    2.238814] pci_bus 0000:00: root bus resource [mem 0x1f200000-0x1f9fffff]
[    2.239091] pci_bus 0000:00: root bus resource [io  0x1fa00000-0x1fbfffff]
[    2.239367] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.239631] pci 0000:00:00.0: [1077:1020] type 00 class 0x010000
[    2.239693] pci 0000:00:00.0: reg 0x10: [io  0x200000-0x2000ff]
[    2.239718] pci 0000:00:00.0: reg 0x14: [mem 0x00200000-0x00200fff]
[    2.239800] pci 0000:00:00.0: reg 0x30: [mem 0x00210000-0x0021ffff pref]
[    2.240004] pci 0000:00:01.0: [1077:1020] type 00 class 0x010000
[    2.240049] pci 0000:00:01.0: reg 0x10: [io  0x400000-0x4000ff]
[    2.240073] pci 0000:00:01.0: reg 0x14: [mem 0x00400000-0x00400fff]
[    2.240155] pci 0000:00:01.0: reg 0x30: [mem 0x00410000-0x0041ffff pref]
[    2.240319] pci 0000:00:02.0: [10a9:0003] type 00 class 0xff0000
[    2.240369] pci 0000:00:02.0: reg 0x10: [mem 0x00500000-0x005fffff]
[    2.240614] pci 0000:00:03.0: [10a9:0005] type 00 class 0x000000
[    2.240656] pci 0000:00:03.0: reg 0x10: [mem 0x00600000-0x00601fff]
[    2.240994] pci 0000:00:02.0: BAR 0: assigned [mem 0x1f200000-0x1f2fffff]
[    2.241280] pci 0000:00:00.0: BAR 6: assigned [mem 0x1f300000-0x1f30ffff pref]
[    2.241570] pci 0000:00:01.0: BAR 6: assigned [mem 0x1f310000-0x1f31ffff pref]
[    2.241857] pci 0000:00:00.0: BAR 1: assigned [mem 0x1f320000-0x1f320fff]
[    2.242133] pci 0000:00:01.0: BAR 1: assigned [mem 0x1f321000-0x1f321fff]
[    2.242408] pci 0000:00:00.0: BAR 0: assigned [io  0x1fa00000-0x1fa000ff]
[    2.242684] pci 0000:00:01.0: BAR 0: assigned [io  0x1fa00400-0x1fa004ff]
[    2.243018] qla1280: QLA1040 found on PCI bus 0, dev 0
[    2.243240] ip30-bridge: 0000:00:00.0 Bar 0 with size 0x00000100 at bus 0x00000000 vma 0x000000f100000000 is Direct I/O.
[    2.243662] ip30-bridge: 0000:00:00.0 Bar 1 with size 0x00001000 at bus 0x00000000 vma 0x000000f080000000 is Direct 64-bit.
[    2.244094] ip30-bridge: 0000:00:00.0 Bar 6 with size 0x00010000 at bus 0x00010000 vma 0x000000f080010000 is Direct 64-bit.
[    2.244527] qla1280 0000:00:00.0: enabling device (0006 -> 0007)
[    2.245490] scsi(0): Enabling vchannel on BRIDGE for SGI/MIPS
[    2.662300] random: crng init done
[    2.820039] scsi(0:0): Resetting SCSI BUS
[    5.855694] scsi host0: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[    5.862361] qla1280: QLA1040 found on PCI bus 0, dev 1
[    5.862628] ip30-bridge: 0000:00:01.0 Bar 0 with size 0x00000100 at bus 0x00000100 vma 0x000000f100000100 is Direct I/O.
[    5.863110] ip30-bridge: 0000:00:01.0 Bar 1 with size 0x00001000 at bus 0x00020000 vma 0x000000f080020000 is Direct 64-bit.
[    5.863622] ip30-bridge: 0000:00:01.0 Bar 6 with size 0x00010000 at bus 0x00030000 vma 0x000000f080030000 is Direct 64-bit.
[    5.864140] qla1280 0000:00:01.0: enabling device (0006 -> 0007)
[    6.047211] scsi 0:0:1:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[    6.048346] scsi(0:0:1:0):
[    6.048353]  Sync: period 10, offset 12
[    6.048479] , Wide
[    6.048642] , Tagged queuing: depth 31

[    6.096065] scsi 0:0:2:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[    6.106075] scsi(0:0:2:0):
[    6.106082]  Sync: period 10, offset 12
[    6.115061] , Wide
[    6.124143] , Tagged queuing: depth 31

[    6.169359] scsi 0:0:3:0: Direct-Access     IBM-ESXS ST973401LC    FN B41D PQ: 0 ANSI: 4
[    6.179384] scsi(0:0:3:0):
[    6.179390]  Sync: period 10, offset 12
[    6.188384] , Wide
[    6.197545] , Tagged queuing: depth 31

[    6.491039] scsi(1:0): Resetting SCSI BUS
[    8.101368] sd 0:0:1:0: Power-on or device reset occurred
[    8.109771] sd 0:0:2:0: Power-on or device reset occurred
[    8.117926] sd 0:0:3:0: Power-on or device reset occurred
[    8.128996] sd 0:0:1:0: [sda] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[    8.137191] sd 0:0:2:0: [sdb] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[    8.145257] sd 0:0:3:0: [sdc] 143374000 512-byte logical blocks: (73.4 GB/68.4 GiB)
[    8.153116] sd 0:0:1:0: [sda] Write Protect is off
[    8.160936] sd 0:0:1:0: [sda] Mode Sense: b3 00 10 08
[    8.160979] sd 0:0:3:0: [sdc] Write Protect is off
[    8.168704] sd 0:0:3:0: [sdc] Mode Sense: b3 00 10 08
[    8.168745] sd 0:0:2:0: [sdb] Write Protect is off
[    8.176316] sd 0:0:2:0: [sdb] Mode Sense: b3 00 10 08
[    8.177557] sd 0:0:1:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    8.185308] sd 0:0:3:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    8.192915] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    8.212790]  sda: sda1 sda2 sda3 sda4 sda5 sda6 sda9 sda11
[    8.220513]  sdc: sdc1 sdc2 sdc3 sdc4 sdc5 sdc6 sdc9 sdc11
[    8.227919]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb9 sdb11
[    8.237809] sd 0:0:1:0: [sda] Attached SCSI disk
[    8.245936] sd 0:0:3:0: [sdc] Attached SCSI disk
[    8.258431] sd 0:0:2:0: [sdb] Attached SCSI disk
[    9.535586] scsi host1: QLogic QLA1040 PCI to SCSI Host Adapter
                      Firmware version:  7.65.06, Driver version 3.27.1
[    9.556383] ip30-bridge: 0000:00:02.0 Bar 0 with size 0x00100000 at bus 0x00100000 vma 0x000000f080100000 is Direct 64-bit.
[   11.677929] ioc3: part: [030-0891-003], serial: [KBB731] => class IP30 System Board
[   11.688189] ioc3-eth: Ethernet address is 08:00:69:0e:11:0c.
[   11.697264] IOC3 0000:00:02.0 eth0: link up, 100Mbps, full-duplex, lpa 0x85E1
[   11.705591] eth0: Using PHY 1, vendor 0x15f42, model 2, rev 3.
[   11.713921] eth0: IOC3 SSRAM has 128 kbyte.
[   11.722850] rtc-ds1685 rtc-ds1685: registered as rtc0
[   11.756679] 0000:00:02.0: ttyS0 at IOC3 0xf080120178 (irq = 0, base_baud = 458333) is a 16550A
[   11.785950] 0000:00:02.0: ttyS1 at IOC3 0xf080120170 (irq = 0, base_baud = 458333) is a 16550A
[   11.794433] IOC3 Master Driver loaded for 0000:00:02.0
[   11.802753] ip30-bridge: 0000:00:03.0 Bar 0 with size 0x00002000 at bus 0x00200000 vma 0x000000f080200000 is Direct 64-bit.
[   11.829263] PCI host bridge to bus 0001:00
[   11.837875] pci_bus 0001:00: root bus resource [mem 0x1d200000-0x1d9fffff]
[   11.846593] pci_bus 0001:00: root bus resource [io  0x1da00000-0x1dbfffff]
[   11.855193] pci_bus 0001:00: root bus resource [bus 01-ff]
[   11.863744] pci 0001:00:01.0: [11fe:080e] type 00 class 0x078000
[   11.863804] pci 0001:00:01.0: reg 0x10: [mem 0x00200000-0x0020007f]
[   11.863829] pci 0001:00:01.0: reg 0x14: [io  0x200000-0x20007f]
[   11.863853] pci 0001:00:01.0: reg 0x18: [io  0x204000-0x2040ff]
[   11.864130] pci 0001:00:02.0: [10b9:5237] type 00 class 0x0c0310
[   11.864179] pci 0001:00:02.0: reg 0x10: [mem 0x00300000-0x00300fff]
[   11.864339] pci 0001:00:02.0: PME# supported from D0 D1 D3hot D3cold
[   11.864499] pci 0001:00:02.1: [10b9:5237] type 00 class 0x0c0310
[   11.864545] pci 0001:00:02.1: reg 0x10: [mem 0x00000000-0x00000fff]
[   11.864683] pci 0001:00:02.1: PME# supported from D0 D1 D3hot D3cold
[   11.864843] pci 0001:00:02.2: [10b9:5237] type 00 class 0x0c0310
[   11.864889] pci 0001:00:02.2: reg 0x10: [mem 0x00000000-0x00000fff]
[   11.865028] pci 0001:00:02.2: PME# supported from D0 D1 D3hot D3cold
[   11.865187] pci 0001:00:02.3: [10b9:5239] type 00 class 0x0c0320
[   11.865236] pci 0001:00:02.3: reg 0x10: [mem 0x00000000-0x000000ff]
[   11.865388] pci 0001:00:02.3: PME# supported from D0 D3hot D3cold
[   11.865588] pci 0001:00:03.0: [10a9:0009] type 00 class 0x020000
[   11.865635] pci 0001:00:03.0: reg 0x10: [mem 0x00400000-0x00403fff]
[   11.865987] pci 0001:00:03.0: BAR 0: assigned [mem 0x1d200000-0x1d203fff]
[   11.874286] pci 0001:00:02.0: BAR 0: assigned [mem 0x1d204000-0x1d204fff]
[   11.882663] pci 0001:00:02.1: BAR 0: assigned [mem 0x1d205000-0x1d205fff]
[   11.890998] pci 0001:00:02.2: BAR 0: assigned [mem 0x1d206000-0x1d206fff]
[   11.899080] pci 0001:00:01.0: BAR 2: assigned [io  0x1da00000-0x1da000ff]
[   11.906938] pci 0001:00:02.3: BAR 0: assigned [mem 0x1d207000-0x1d2070ff]
[   11.914661] pci 0001:00:01.0: BAR 0: assigned [mem 0x1d207100-0x1d20717f]
[   11.922395] pci 0001:00:01.0: BAR 1: assigned [io  0x1da00400-0x1da0047f]
[   11.930194] acenic.c: v0.92 08/05/2002  Jes Sorensen, linux-acenic@SunSITE.dk
                                           http://home.cern.ch/~jes/gige/acenic.html
[   11.945884] ip30-bridge: 0001:00:03.0 Bar 0 with size 0x00004000 at bus 0x00000000 vma 0x000000d080000000 is Direct 64-bit.
[   11.954084] 0001:00:03.0: SGI AceNIC
[   11.954090] Gigabit Ethernet at 0xd080000000,
[   11.962168] irq 5
[   11.978799]   Tigon II (Rev. 6), Firmware: 0.0.0,
[   11.984167] MAC: 08:00:69:14:69:9c
[   12.000501]   PCI cache line size set incorrectly (0 bytes) by BIOS/FW,
[   12.000508] correcting to 128
[   12.016988]   PCI bus width: 64 bits, speed: 33MHz, latency: 64 clks
[   12.074154] 0001:00:03.0: Firmware up and running
[   12.083733] NET: Registered protocol family 26
[   12.092552] NET: Registered protocol family 10
[   12.102794] Segment Routing with IPv6
[   12.111562] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   12.120564] NET: Registered protocol family 17
[   12.129128] 8021q: 802.1Q VLAN Support v1.8
[   12.137981] sctp: Hash tables configured (bind 4096/4096)
[   12.146796] Key type dns_resolver registered
[   12.156664] rtc-ds1685 rtc-ds1685: setting system clock to 2019-06-20 05:16:04 UTC (1561007764)
[   12.165737] ALSA device list:
[   12.174699]   #0: SGI RAD Audio at 0xf080200000
[   12.222749] input: AT Raw Set 2 keyboard as /devices/pci0000:00/0000:00:02.0/serio0/input/input0
[   12.473637] input: ImPS/2 Logitech Wheel Mouse as /devices/pci0000:00/0000:00:02.0/serio1/input/input2
[   14.293233] md: Waiting for all devices to be available before autodetect
[   14.302992] md: If you don't use raid, use raid=noautodetect
[   14.313625] md: Autodetecting RAID arrays.
[   14.456847] md: autorun ...
[   14.466680] md: sdb5 has different UUID to sdb6
[   14.476492] md: sdb4 has different UUID to sdb6
[   14.486233] md: sdb3 has different UUID to sdb6
[   14.495971] md: sdb1 has different UUID to sdb6
[   14.505729] md: sdc5 has different UUID to sdb6
[   14.515283] md: sdc4 has different UUID to sdb6
[   14.524746] md: sdc3 has different UUID to sdb6
[   14.533869] md: sdc1 has different UUID to sdb6
[   14.542676] md: sda5 has different UUID to sdb6
[   14.551268] md: sda4 has different UUID to sdb6
[   14.559680] md: sda3 has different UUID to sdb6
[   14.567897] md: sda1 has different UUID to sdb6
[   14.576720] md: running: <sdb6><sdc6><sda6>
[   14.585742] md/raid:md4: device sdb6 operational as raid disk 1
[   14.593657] md/raid:md4: device sdc6 operational as raid disk 2
[   14.601704] md/raid:md4: device sda6 operational as raid disk 0
[   14.612270] md/raid:md4: raid level 5 active with 3 out of 3 devices, algorithm 2
[   14.768222] md4: detected capacity change from 0 to 2263613440
[   14.775792] md: sdb4 has different UUID to sdb5
[   14.783133] md: sdb3 has different UUID to sdb5
[   14.790589] md: sdb1 has different UUID to sdb5
[   14.797953] md: sdc4 has different UUID to sdb5
[   14.805098] md: sdc3 has different UUID to sdb5
[   14.812309] md: sdc1 has different UUID to sdb5
[   14.819531] md: sda4 has different UUID to sdb5
[   14.826678] md: sda3 has different UUID to sdb5
[   14.833801] md: sda1 has different UUID to sdb5
[   14.841631] md: running: <sdb5><sdc5><sda5>
[   14.849690] md/raid:md3: device sdb5 operational as raid disk 1
[   14.856933] md/raid:md3: device sdc5 operational as raid disk 2
[   14.864023] md/raid:md3: device sda5 operational as raid disk 0
[   14.873665] md/raid:md3: raid level 5 active with 3 out of 3 devices, algorithm 2
[   14.887017] md3: detected capacity change from 0 to 4714135552
[   14.893694] md: sdb3 has different UUID to sdb4
[   14.900334] md: sdb1 has different UUID to sdb4
[   14.906932] md: sdc3 has different UUID to sdb4
[   14.913356] md: sdc1 has different UUID to sdb4
[   14.919793] md: sda3 has different UUID to sdb4
[   14.925963] md: sda1 has different UUID to sdb4
[   14.932690] md: running: <sdb4><sdc4><sda4>
[   14.939582] md/raid:md2: device sdb4 operational as raid disk 1
[   14.945481] md/raid:md2: device sdc4 operational as raid disk 2
[   14.951398] md/raid:md2: device sda4 operational as raid disk 0
[   14.959929] md/raid:md2: raid level 5 active with 3 out of 3 devices, algorithm 2
[   14.971449] md2: detected capacity change from 0 to 109521403904
[   14.977476] md: sdb1 has different UUID to sdb3
[   14.983353] md: sdc1 has different UUID to sdb3
[   14.989248] md: sda1 has different UUID to sdb3
[   14.995616] md: running: <sdb3><sdc3><sda3>
[   15.002158] md/raid:md1: device sdb3 operational as raid disk 1
[   15.007713] md/raid:md1: device sdc3 operational as raid disk 2
[   15.013066] md/raid:md1: device sda3 operational as raid disk 0
[   15.021113] md/raid:md1: raid level 5 active with 3 out of 3 devices, algorithm 2
[   15.039410] md1: detected capacity change from 0 to 23622057984
[   15.044772] md: running: <sdb1><sdc1><sda1>
[   15.050679] md/raid:md0: device sdb1 operational as raid disk 1
[   15.055629] md/raid:md0: device sdc1 operational as raid disk 2
[   15.060460] md/raid:md0: device sda1 operational as raid disk 0
[   15.067921] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
[   15.081453] md0: detected capacity change from 0 to 4294705152
[   15.086508] md: ... autorun DONE.
[   15.157204] XFS (md0): Mounting V5 Filesystem
[   15.797708] XFS (md0): Ending clean mount
[   15.823891] VFS: Mounted root (xfs filesystem) readonly on device 9:0.
[   15.833380] devtmpfs: mounted
[   15.840112] Freeing unused kernel memory: 384K
[   15.845766] This architecture does not have kernel memory protection.
[   18.145242] eth1: Optical link UP (Full Duplex, Flow Control: TX RX)
[   31.408622] Adding 1048512k swap on /dev/sda2.  Priority:1 extents:1 across:1048512k
[   31.424506] Adding 1048512k swap on /dev/sdb2.  Priority:1 extents:1 across:1048512k
[   31.440530] Adding 1048512k swap on /dev/sdc2.  Priority:1 extents:1 across:1048512k
[   31.916775] XFS (md1): Mounting V5 Filesystem
[   33.243814] XFS (md1): Ending clean mount
[   33.439977] XFS (md2): Mounting V5 Filesystem
[   34.094700] XFS (md2): Ending clean mount
[   34.516999] XFS (md3): Mounting V5 Filesystem
[   35.230208] XFS (md3): Ending clean mount
[   35.317999] XFS (md4): Mounting V5 Filesystem
[   35.963730] XFS (md4): Ending clean mount
[   49.174603] udevd[1701]: starting version 3.2.7
[   49.259590] udevd[1701]: specified group 'video' unknown
[   49.260533] udevd[1701]: specified group 'tape' unknown
[   49.433771] udevd[1701]: starting eudev-3.2.7
[   55.316732] eth1: Enabling Jumbo frame support