iwlwifi module crash

iwlwifi module crash

[Balakrishnan Balasubramanian]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

I am using iwd demon for wifi. Once a while I loose connectivity. Restarting 
the demon does not help. But once I restart the system, it starts working fine. 
Attaching stack trace from journal.

Regards,
Bala

[-- Attachment #2: Denis Kenzior <denkenz@gmail.com>: Re: iwd crashes randomly --]
[-- Type: message/rfc822, Size: 3271 bytes --]

From: Denis Kenzior <denkenz@gmail.com>
To: Balakrishnan Balasubramanian <iwd-lists@balki.me>, iwd@lists.01.org
Subject: Re: iwd crashes randomly
Date: Thu, 06 Jun 2019 18:07:40 -0500
Message-ID: <9a29fc57-0547-203a-9e9f-417b16f9617e@gmail.com>

Hi Bala,

On 06/06/2019 06:00 PM, Balakrishnan Balasubramanian wrote:
> Sometimes after a week and sometimes after two days. Once crashed, restarting
> the service does not help. Had to restart the computer. Attaching stack trace
> from journal.

That implies that your kernel is crashing, not iwd.  The attached log 
shows a kernel stack trace somewhere inside iwlwifi module.  I would 
post this trace to linux-wireless@vger.kernel.org.

If you have an associated iwd backtrace, then certainly post this here, 
but if the kernel module is crashing, there isn't much we can do.

Regards,
-Denis

[-- Attachment #3: iwd_crash.log --]
[-- Type: text/x-log, Size: 14052 bytes --]

Jun 03 19:53:47 zadesk dbus-daemon[475]: [system] Successfully activated service 'org.kde.powerdevil.backlighthelper'
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: fe80::481d:70ff:feaf:2a13 is unreachable, expiring it
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: carrier lost
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: deleting address 2607:f2c0:e568:c9:3c6f:aab:250c:45d4/128
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: deleting address 2607:f2c0:e568:c9:16d1:ac43:f2f7:9142/64
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: deleting route to 2607:f2c0:e568:c9::/64
Jun 03 21:32:49 zadesk dhcpcd[505]: wlan0: deleting default route via fe80::481d:70ff:feaf:2a13
Jun 03 21:32:52 zadesk iwd[474]: Received Deauthentication event, reason: 4, from_ap: false
Jun 03 21:32:52 zadesk dhcpcd[505]: wlan0: deleting route to 192.168.0.0/24
Jun 03 21:32:52 zadesk kernel: wlan0: Connection to AP 00:00:00:00:00:00 lost
Jun 03 21:32:52 zadesk dhcpcd[505]: wlan0: deleting default route via 192.168.0.1
Jun 03 21:33:14 zadesk kernel: wlan0: authenticate with d4:5d:df:25:ee:90
Jun 03 21:33:14 zadesk kernel: wlan0: send auth to d4:5d:df:25:ee:90 (try 1/3)
Jun 03 21:33:14 zadesk kernel: wlan0: authenticated
Jun 03 21:33:14 zadesk kernel: wlan0: associate with d4:5d:df:25:ee:90 (try 1/3)
Jun 03 21:33:14 zadesk kernel: wlan0: associate with d4:5d:df:25:ee:90 (try 2/3)
Jun 03 21:33:14 zadesk kernel: wlan0: RX AssocResp from d4:5d:df:25:ee:90 (capab=0x411 status=0 aid=1)
Jun 03 21:33:14 zadesk kernel: wlan0: associated
Jun 03 21:33:14 zadesk dhcpcd[505]: wlan0: carrier acquired
Jun 03 21:33:14 zadesk dhcpcd[505]: wlan0: IAID ce:1e:6b:76
Jun 03 21:33:14 zadesk kernel: wlan0: Limiting TX power to 14 (17 - 3) dBm as advertised by d4:5d:df:25:ee:90
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: soliciting an IPv6 router
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: Router Advertisement from fe80::481d:70ff:feaf:2a13
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: adding address 2607:f2c0:e568:c9:16d1:ac43:f2f7:9142/64
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: adding route to 2607:f2c0:e568:c9::/64
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: adding default route via fe80::481d:70ff:feaf:2a13
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: confirming prior DHCPv6 lease
Jun 03 21:33:15 zadesk dhcpcd[505]: wlan0: rebinding lease of 192.168.0.16
Jun 03 21:33:16 zadesk dhcpcd[505]: wlan0: probing address 192.168.0.16/24
Jun 03 21:33:16 zadesk dhcpcd[505]: wlan0: REPLY6 received from fe80::481d:70ff:feaf:2a13
Jun 03 21:33:16 zadesk dhcpcd[505]: wlan0: adding address 2607:f2c0:e568:c9:3c6f:aab:250c:45d4/128
Jun 03 21:33:16 zadesk dhcpcd[505]: wlan0: renew in 302400, rebind in 483840, expire in 604800 seconds
Jun 03 21:33:21 zadesk dhcpcd[505]: wlan0: leased 192.168.0.16 for 604800 seconds
Jun 03 21:33:21 zadesk dhcpcd[505]: wlan0: adding route to 192.168.0.0/24
Jun 03 21:33:21 zadesk dhcpcd[505]: wlan0: adding default route via 192.168.0.1
Jun 03 21:34:03 zadesk kernel: ------------[ cut here ]------------
Jun 03 21:34:03 zadesk kernel: Timeout waiting for hardware access (CSR_GP_CNTRL 0xffffffff)
Jun 03 21:34:03 zadesk kernel: WARNING: CPU: 2 PID: 0 at drivers/net/wireless/intel/iwlwifi/pcie/trans.c:1988 iwl_trans_pcie_grab_nic_access+0x1e6/0x220 [iwlwifi]
Jun 03 21:34:03 zadesk kernel: Modules linked in: fuse snd_hda_codec_hdmi ccm algif_aead des_generic algif_skcipher 8021q garp mrp stp cmac llc md4 algif_hash af_alg arc4 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915 kvm iwlmvm uvcvideo irqbypass i2c_algo_bit mac80211 videobuf2_vmalloc videobuf2_memops snd_hda_codec_realtek drm_kms_helper snd_hda_codec_generic videobuf2_v4l2 ledtrig_audio videobuf2_common snd_hda_intel crct10dif_pclmul videodev snd_hda_codec drm crc32_pclmul iwlwifi ghash_clmulni_intel btusb btrtl btbcm btintel bluetooth snd_hda_core ofpart cmdlinepart media intel_spi_platform snd_hwdep intel_spi spi_nor iTCO_wdt mei_wdt mei_hdcp mtd iTCO_vendor_support aesni_intel snd_pcm aes_x86_64 crypto_simd cryptd snd_timer intel_gtt glue_helper intel_cstate intel_uncore joydev mousedev agpgart input_leds syscopyarea r8169 intel_rapl_perf ecdh_generic cfg80211 snd sysfillrect mei_me pcspkr i2c_i801 realtek sysimgblt soundcore mei libphy fb_sys_fops rfkill lpc_ich fintek_cir
Jun 03 21:34:03 zadesk kernel:  rc_core tpm_tis tpm_tis_core tpm evdev rng_core mac_hid pcc_cpufreq pkcs8_key_parser crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom sd_mod ahci xhci_pci libahci xhci_hcd libata crc32c_intel scsi_mod ehci_pci ehci_hcd
Jun 03 21:34:03 zadesk kernel: CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.1.5-arch1-2-ARCH #1
Jun 03 21:34:03 zadesk kernel: Hardware name: AIO H87H3-TI/H87H3-TI, BIOS 4.6.5 07/01/2014
Jun 03 21:34:03 zadesk kernel: RIP: 0010:iwl_trans_pcie_grab_nic_access+0x1e6/0x220 [iwlwifi]
Jun 03 21:34:03 zadesk kernel: Code: b1 cf 49 8d 56 08 bf 40 01 00 00 e8 94 1b 88 ce e9 35 ff ff ff 89 c6 48 c7 c7 70 7b c4 c0 c6 05 b7 7d 03 00 01 e8 94 62 86 ce <0f> 0b e9 f0 fe ff ff 48 8b 7b 30 48 c7 c1 d8 7b c4 c0 31 d2 31 f6
Jun 03 21:34:03 zadesk kernel: RSP: 0018:ffff981696903de8 EFLAGS: 00010082
Jun 03 21:34:03 zadesk kernel: RAX: 0000000000000000 RBX: ffff981693d80018 RCX: 0000000000000000
Jun 03 21:34:03 zadesk kernel: RDX: 0000000000000007 RSI: 0000000000000092 RDI: 00000000ffffffff
Jun 03 21:34:03 zadesk kernel: RBP: 0000000000000000 R08: 00000000000003a1 R09: 0000000000000001
Jun 03 21:34:03 zadesk kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff981693d8a3a4
Jun 03 21:34:03 zadesk kernel: R13: ffff981696903e18 R14: 00000000ffffffff R15: 0000000000000004
Jun 03 21:34:03 zadesk kernel: FS:  0000000000000000(0000) GS:ffff981696900000(0000) knlGS:0000000000000000
Jun 03 21:34:03 zadesk kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 03 21:34:03 zadesk kernel: CR2: 00007ffa69d3c020 CR3: 00000001b820e002 CR4: 00000000001606e0
Jun 03 21:34:03 zadesk kernel: Call Trace:
Jun 03 21:34:03 zadesk kernel:  <IRQ>
Jun 03 21:34:03 zadesk kernel:  iwl_read_prph+0x32/0x80 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  iwl_trans_pcie_log_scd_error+0x139/0x210 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  iwl_pcie_txq_stuck_timer+0x44/0x60 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  ? iwl_pcie_txq_inc_wr_ptr+0x100/0x100 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  ? iwl_pcie_txq_inc_wr_ptr+0x100/0x100 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  call_timer_fn+0x2b/0x160
Jun 03 21:34:03 zadesk kernel:  ? iwl_pcie_txq_inc_wr_ptr+0x100/0x100 [iwlwifi]
Jun 03 21:34:03 zadesk kernel:  expire_timers+0x99/0x110
Jun 03 21:34:03 zadesk kernel:  run_timer_softirq+0x8a/0x160
Jun 03 21:34:03 zadesk kernel:  ? sched_clock+0x5/0x10
Jun 03 21:34:03 zadesk kernel:  ? sched_clock_cpu+0xe/0xd0
Jun 03 21:34:03 zadesk kernel:  ? irqtime_account_irq+0x3d/0xc0
Jun 03 21:34:03 zadesk kernel:  __do_softirq+0x112/0x356
Jun 03 21:34:03 zadesk kernel:  irq_exit+0xd9/0xf0
Jun 03 21:34:03 zadesk kernel:  smp_apic_timer_interrupt+0x87/0x180
Jun 03 21:34:03 zadesk kernel:  apic_timer_interrupt+0xf/0x20
Jun 03 21:34:03 zadesk kernel:  </IRQ>
Jun 03 21:34:03 zadesk kernel: RIP: 0010:cpuidle_enter_state+0xbc/0x480
Jun 03 21:34:03 zadesk kernel: Code: e8 a9 c0 9f ff 80 7c 24 13 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 99 03 00 00 31 ff e8 3b 03 a6 ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 88 c4 02 00 00 49 63 cc 4c 8b 3c 24 4c 2b 7c 24 08 48
Jun 03 21:34:03 zadesk kernel: RSP: 0018:ffff9fec80cf7e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Jun 03 21:34:03 zadesk kernel: RAX: ffff981696900000 RBX: ffffffff906bd5c0 RCX: 000000000000001f
Jun 03 21:34:03 zadesk kernel: RDX: 0000083837ad6417 RSI: 000000002c3d7be3 RDI: 0000000000000000
Jun 03 21:34:03 zadesk kernel: RBP: ffff98169692be00 R08: 0000000000000002 R09: 0000000000021740
Jun 03 21:34:03 zadesk kernel: R10: 000017ceeb3a3497 R11: ffff981696920d64 R12: 0000000000000005
Jun 03 21:34:03 zadesk kernel: R13: ffffffff906bd7b8 R14: 0000000000000005 R15: 0000000000000000
Jun 03 21:34:03 zadesk kernel:  ? cpuidle_enter_state+0x97/0x480
Jun 03 21:34:03 zadesk kernel:  do_idle+0x217/0x250
Jun 03 21:34:03 zadesk kernel:  cpu_startup_entry+0x19/0x20
Jun 03 21:34:03 zadesk kernel:  start_secondary+0x180/0x1d0
Jun 03 21:34:03 zadesk kernel:  secondary_startup_64+0xa4/0xb0
Jun 03 21:34:03 zadesk kernel: ---[ end trace d9201afd2c4b3e67 ]---
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: iwlwifi transaction failed, dumping registers
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: iwlwifi device config registers:
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000000: 08b18086 00100000 028000bb 00000000 00000004 00000000 00000000 00000000
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000020: 00000000 00000000 00000000 40708086 00000000 000000c8 00000000 00000100
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: iwlwifi device memory mapped registers:
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000000: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000020: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: iwlwifi device AER capability structure:
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000000: 14010001 00100000 00000000 00462031 000031c1 00002000 00000014 40000001
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: 00000020: 0000000f f7d00460 00000000
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: iwlwifi parent port (0000:00:1c.2) config registers:
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:00:1c.2: 00000000: 8c148086 00100407 060400d5 00810010 00000000 00000000 00020200 200000f0
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:00:1c.2: 00000020: f7d0f7d0 0001fff1 00000000 00000000 00000000 00000040 00000000 00120304
Jun 03 21:34:03 zadesk kernel: iwlwifi 0000:02:00.0: Queue 4 is active on fifo 2 and stuck for 10000 ms. SW [87, 103] HW [90, 90] FH TRB=0x05a5a5a5a
Jun 03 21:34:23 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:34:23 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:34:23 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:34:23 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:35:19 zadesk dhcpcd[505]: wlan0: fe80::481d:70ff:feaf:2a13 is unreachable, expiring it
Jun 03 21:35:23 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:35:23 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:35:23 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:35:23 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:36:23 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:36:23 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:36:23 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:36:23 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:37:23 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:37:23 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:37:23 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:37:23 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:38:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:38:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:38:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:38:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:39:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:39:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:39:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:39:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:40:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:40:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:40:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:40:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:41:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:41:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:41:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:41:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:42:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:42:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:42:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:42:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:43:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 03 21:43:24 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 03 21:43:24 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 03 21:43:24 zadesk iwd[474]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)
Jun 03 21:44:24 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd

Re: iwlwifi module crash

[Emmanuel Grumbach]

On Fri, Jun 7, 2019 at 5:22 AM Balakrishnan Balasubramanian
<linux-wireless-list@balki.me> wrote:
>
> I am using iwd demon for wifi. Once a while I loose connectivity. Restarting
> the demon does not help. But once I restart the system, it starts working fine.
> Attaching stack trace from journal.

This is because the device is removed from the PCI bus. Nothing from
iwlwifi side can be done.
If that happens upon suspend / resume, I know there are been fixes in
PCI bus driver. If not, check that the device sits correctly in its
socket.

>
> Regards,
> Bala
>
>
> ---------- Forwarded message ----------
> From: Denis Kenzior <denkenz@gmail.com>
> To: Balakrishnan Balasubramanian <iwd-lists@balki.me>, iwd@lists.01.org
> Cc:
> Bcc:
> Date: Thu, 06 Jun 2019 18:07:40 -0500
> Subject: Re: iwd crashes randomly
> Hi Bala,
>
> On 06/06/2019 06:00 PM, Balakrishnan Balasubramanian wrote:
> > Sometimes after a week and sometimes after two days. Once crashed, restarting
> > the service does not help. Had to restart the computer. Attaching stack trace
> > from journal.
>
> That implies that your kernel is crashing, not iwd.  The attached log
> shows a kernel stack trace somewhere inside iwlwifi module.  I would
> post this trace to linux-wireless@vger.kernel.org.
>
> If you have an associated iwd backtrace, then certainly post this here,
> but if the kernel module is crashing, there isn't much we can do.
>
> Regards,
> -Denis

Re: iwlwifi module crash

[Balakrishnan Balasubramanian]

> This is because the device is removed from the PCI bus. Nothing from
> iwlwifi side can be done.

I am sure the device is not physically disturbed. If that was the case, should it not stay down when restarting the system?

> If that happens upon suspend / resume, I know there are been fixes in
> PCI bus driver.

To my knowledge I have disabled all power/suspend features and I don't see releated logs in journal except the below. Not sure if relevant.

    Jun 03 21:33:14 zadesk kernel: wlan0: Limiting TX power to 14 (17 - 3) dBm as advertised by d4:5d:df:25:ee:90

Is there a way to restart the module safely without restarting the system?

Regards,
Bala


On Friday, June 7, 2019 5:25:41 AM EDT Emmanuel Grumbach wrote:
> On Fri, Jun 7, 2019 at 5:22 AM Balakrishnan Balasubramanian
> 
> <linux-wireless-list@balki.me> wrote:
> > I am using iwd demon for wifi. Once a while I loose connectivity.
> > Restarting the demon does not help. But once I restart the system, it
> > starts working fine. Attaching stack trace from journal.
> 
> This is because the device is removed from the PCI bus. Nothing from
> iwlwifi side can be done.
> If that happens upon suspend / resume, I know there are been fixes in
> PCI bus driver. If not, check that the device sits correctly in its
> socket.
> 
> > Regards,
> > Bala
> > 
> > 
> > ---------- Forwarded message ----------
> > From: Denis Kenzior <denkenz@gmail.com>
> > To: Balakrishnan Balasubramanian <iwd-lists@balki.me>, iwd@lists.01.org
> > Cc:
> > Bcc:
> > Date: Thu, 06 Jun 2019 18:07:40 -0500
> > Subject: Re: iwd crashes randomly
> > Hi Bala,
> > 
> > On 06/06/2019 06:00 PM, Balakrishnan Balasubramanian wrote:
> > > Sometimes after a week and sometimes after two days. Once crashed,
> > > restarting the service does not help. Had to restart the computer.
> > > Attaching stack trace from journal.
> > 
> > That implies that your kernel is crashing, not iwd.  The attached log
> > shows a kernel stack trace somewhere inside iwlwifi module.  I would
> > post this trace to linux-wireless@vger.kernel.org.
> > 
> > If you have an associated iwd backtrace, then certainly post this here,
> > but if the kernel module is crashing, there isn't much we can do.
> > 
> > Regards,
> > -Denis

Re: iwlwifi module crash

[Emmanuel Grumbach]

On Fri, Jun 7, 2019 at 2:41 PM Balakrishnan Balasubramanian
<linux-wireless-list@balki.me> wrote:
>
> > This is because the device is removed from the PCI bus. Nothing from
> > iwlwifi side can be done.
>
> I am sure the device is not physically disturbed. If that was the case, should it not stay down when restarting the system?

Not necessarily. The disturbance may impact ASPM or something alike.

>
> > If that happens upon suspend / resume, I know there are been fixes in
> > PCI bus driver.
>
> To my knowledge I have disabled all power/suspend features and I don't see releated logs in journal except the below. Not sure if relevant.
>
>     Jun 03 21:33:14 zadesk kernel: wlan0: Limiting TX power to 14 (17 - 3) dBm as advertised by d4:5d:df:25:ee:90
>
> Is there a way to restart the module safely without restarting the system?

echo 1 > /sys/module/iwlwifi/devices/0000\:02\:00.0/remove
echo 1 > /sys/bus/pci/rescan

>
> Regards,
> Bala
>
>
> On Friday, June 7, 2019 5:25:41 AM EDT Emmanuel Grumbach wrote:
> > On Fri, Jun 7, 2019 at 5:22 AM Balakrishnan Balasubramanian
> >
> > <linux-wireless-list@balki.me> wrote:
> > > I am using iwd demon for wifi. Once a while I loose connectivity.
> > > Restarting the demon does not help. But once I restart the system, it
> > > starts working fine. Attaching stack trace from journal.
> >
> > This is because the device is removed from the PCI bus. Nothing from
> > iwlwifi side can be done.
> > If that happens upon suspend / resume, I know there are been fixes in
> > PCI bus driver. If not, check that the device sits correctly in its
> > socket.
> >
> > > Regards,
> > > Bala
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: Denis Kenzior <denkenz@gmail.com>
> > > To: Balakrishnan Balasubramanian <iwd-lists@balki.me>, iwd@lists.01.org
> > > Cc:
> > > Bcc:
> > > Date: Thu, 06 Jun 2019 18:07:40 -0500
> > > Subject: Re: iwd crashes randomly
> > > Hi Bala,
> > >
> > > On 06/06/2019 06:00 PM, Balakrishnan Balasubramanian wrote:
> > > > Sometimes after a week and sometimes after two days. Once crashed,
> > > > restarting the service does not help. Had to restart the computer.
> > > > Attaching stack trace from journal.
> > >
> > > That implies that your kernel is crashing, not iwd.  The attached log
> > > shows a kernel stack trace somewhere inside iwlwifi module.  I would
> > > post this trace to linux-wireless@vger.kernel.org.
> > >
> > > If you have an associated iwd backtrace, then certainly post this here,
> > > but if the kernel module is crashing, there isn't much we can do.
> > >
> > > Regards,
> > > -Denis
>
>
>
>

Re: iwlwifi module crash

[Balakrishnan Balasubramanian]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

The issue occured again today. I tried to restart the module

> echo 1 > /sys/module/iwlwifi/devices/0000\:02\:00.0/remove

There is no folder 'devices'

zadesk% ls /sys/module/iwlwifi 
coresize  drivers  holders  initsize  initstate  notes  parameters  refcnt  
sections  srcversion  taint  uevent

> echo 1 > /sys/bus/pci/rescan

Attached the error when trying to rescan.

Thanks,
Bala





[-- Attachment #2: error --]
[-- Type: text/plain, Size: 367 bytes --]

Jun 13 21:41:56 zadesk kernel: iwlwifi 0000:02:00.0: Failed to wake NIC for hcmd
Jun 13 21:41:56 zadesk kernel: iwlwifi 0000:02:00.0: Error sending SCAN_OFFLOAD_REQUEST_CMD: enqueue_hcmd failed: -5
Jun 13 21:41:56 zadesk kernel: iwlwifi 0000:02:00.0: Scan failed! ret -5
Jun 13 21:41:56 zadesk iwd[483]: Received error during CMD_TRIGGER_SCAN: Input/output error (5)

Re: iwlwifi module crash

[b.K.il.h.u+tigbuh]

devices/ is probably just a symlink. Try to find it manually:
find /sys -iname remove
lspci

The interesting thing is that my iwlwifi card started to do the same
thing just recently (some weeks ago). However, I do suspend a lot and
it only happens after resuming, but not after every resume (maybe
5-10%). It always came back after restarting except on one day when it
needed three restarts, so maybe mine would be more about needing to
reseat the card.

> On Fri, Jun 14, 2019 at 4:54 AM Balakrishnan Balasubramanian <linux-wireless-list@balki.me> wrote:
>>
>> The issue occured again today. I tried to restart the module
>>
>> > echo 1 > /sys/module/iwlwifi/devices/0000\:02\:00.0/remove
>>
>> There is no folder 'devices'
>>
>> zadesk% ls /sys/module/iwlwifi
>> coresize  drivers  holders  initsize  initstate  notes  parameters  refcnt
>> sections  srcversion  taint  uevent
>>
>> > echo 1 > /sys/bus/pci/rescan
>>
>> Attached the error when trying to rescan.
>>
>> Thanks,
>> Bala
>>
>>
>>
>>

Re: iwlwifi module crash

[Balakrishnan Balasubramanian]

Thanks for the tip. In my system the path to remove was below:

/sys/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/remove
Also symlinked here: 
/sys/module/iwlwifi/drivers/pci:iwlwifi/0000:02:00.0/remove

I am now able to restore internet without system restart. Now I need to find a 
way to do this automatically whenever internet goes down.

Thanks,
Bala

On Sunday, June 23, 2019 5:08:32 AM EDT b.K.il.h.u+tigbuh@gmail.com wrote:
> devices/ is probably just a symlink. Try to find it manually:
> find /sys -iname remove
> lspci
> 
> The interesting thing is that my iwlwifi card started to do the same
> thing just recently (some weeks ago). However, I do suspend a lot and
> it only happens after resuming, but not after every resume (maybe
> 5-10%). It always came back after restarting except on one day when it
> needed three restarts, so maybe mine would be more about needing to
> reseat the card.
> 
> > On Fri, Jun 14, 2019 at 4:54 AM Balakrishnan Balasubramanian <linux-
wireless-list@balki.me> wrote:
> >> The issue occured again today. I tried to restart the module
> >> 
> >> > echo 1 > /sys/module/iwlwifi/devices/0000\:02\:00.0/remove
> >> 
> >> There is no folder 'devices'
> >> 
> >> zadesk% ls /sys/module/iwlwifi
> >> coresize  drivers  holders  initsize  initstate  notes  parameters 
> >> refcnt
> >> sections  srcversion  taint  uevent
> >> 
> >> > echo 1 > /sys/bus/pci/rescan
> >> 
> >> Attached the error when trying to rescan.
> >> 
> >> Thanks,
> >> Bala

Re: iwlwifi module crash

[b.K.il.h.u+tigbuh]

Maybe we don't have the same issue, but today this one solved it for
me: `rmmod iwlmvm; rmmod iwlwifi; modprobe iwlwifi`. Well, kind of. It
couldn't really connect to any network afterwards, but after another
such dance it was finally back to normal.

The device disappears from iwconfig for me when this happens, so it is
easy to detect and correct with few lines of shell. It can also be
gone from lspci as well. Another alternative would be to set up a
syslog-ng program destination triggered by a patterndb sample matching
the noted failures among the kernel messages.

If it gets stuck without a visible sign, you could watch Inactive time
and tx failures in `iw dev wlan0 station dump` increase sharply in
case of issues. If you need a bit more reliability, it is easy to ping
the AP or gateway every second and refresh the connection in case of
too many consecutive missing replies.

On Mon, Jun 24, 2019 at 11:36 PM Balakrishnan Balasubramanian
<linux-wireless-list@balki.me> wrote:
>
> Thanks for the tip. In my system the path to remove was below:
>
> /sys/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/remove
> Also symlinked here:
> /sys/module/iwlwifi/drivers/pci:iwlwifi/0000:02:00.0/remove
>
> I am now able to restore internet without system restart. Now I need to find a
> way to do this automatically whenever internet goes down.
>
> Thanks,
> Bala
>
> On Sunday, June 23, 2019 5:08:32 AM EDT b.K.il.h.u+tigbuh@gmail.com wrote:
> > devices/ is probably just a symlink. Try to find it manually:
> > find /sys -iname remove
> > lspci
> >
> > The interesting thing is that my iwlwifi card started to do the same
> > thing just recently (some weeks ago). However, I do suspend a lot and
> > it only happens after resuming, but not after every resume (maybe
> > 5-10%). It always came back after restarting except on one day when it
> > needed three restarts, so maybe mine would be more about needing to
> > reseat the card.
> >
> > > On Fri, Jun 14, 2019 at 4:54 AM Balakrishnan Balasubramanian <linux-
> wireless-list@balki.me> wrote:
> > >> The issue occured again today. I tried to restart the module
> > >>
> > >> > echo 1 > /sys/module/iwlwifi/devices/0000\:02\:00.0/remove
> > >>
> > >> There is no folder 'devices'
> > >>
> > >> zadesk% ls /sys/module/iwlwifi
> > >> coresize  drivers  holders  initsize  initstate  notes  parameters
> > >> refcnt
> > >> sections  srcversion  taint  uevent
> > >>
> > >> > echo 1 > /sys/bus/pci/rescan
> > >>
> > >> Attached the error when trying to rescan.
> > >>
> > >> Thanks,
> > >> Bala
>
>
>
>