[v2,0/5] cpufreq: intel_pstate: Address some HWP-related oddities
mbox series

Message ID 4169555.5IIHXK4Dsd@kreacher
Headers show
Series
  • cpufreq: intel_pstate: Address some HWP-related oddities
Related show

Message

Rafael J. Wysocki Aug. 24, 2020, 5:39 p.m. UTC
Hi All,

The v2 is here to address feedback from Doug and one issue found by me.

The purpose of this series is to address some peculiarities related to
taking CPUs offline/online and switching between different operation
modes with HWP enabled that have become visible after allowing the
driver to work in the passive mode with HWP enabled in 5.9-rc1 (and
one that was there earlier, but can be addressed easily after the
changes madein 5.9-rc1).

Please refer to the patch changelogs for details.

For easier testing/review, the series is available from the git branch at:

 git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git \
 intel_pstate-testing

Thanks,
Rafael

Comments

Srinivas Pandruvada Aug. 24, 2020, 11:56 p.m. UTC | #1
On Mon, 2020-08-24 at 19:39 +0200, Rafael J. Wysocki wrote:
> Hi All,
> 
> The v2 is here to address feedback from Doug and one issue found by
> me.
> 
> The purpose of this series is to address some peculiarities related
> to
> taking CPUs offline/online and switching between different operation
> modes with HWP enabled that have become visible after allowing the
> driver to work in the passive mode with HWP enabled in 5.9-rc1 (and
> one that was there earlier, but can be addressed easily after the
> changes madein 5.9-rc1).
> 
> Please refer to the patch changelogs for details.
> 
> For easier testing/review, the series is available from the git
> branch at:
> 
>  git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git \
>  intel_pstate-testing
> 

Applied these patches to 5.9-rc2

- After s3  limits got messed up.
 # cat /sys/power/mem_sleep 
s2idle [deep]

- In the dmesg unchecked MSR for HWP register

1.
Before test

sudo rdmsr -a 0x774
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d
7f002b0c



cd /sys/devices/system/cpu/intel_pstate/
[root@otcpl-perf-test-skx-i9 intel_pstate]# grep . *
hwp_dynamic_boost:0
max_perf_pct:100
min_perf_pct:27
no_turbo:0
num_pstates:32
status:active
turbo_pct:32

cd ../cpu1/cpufreq/
[root@otcpl-perf-test-skx-i9 cpufreq]# grep . *
affected_cpus:1
base_frequency:3300000
cpuinfo_max_freq:4300000
cpuinfo_min_freq:1200000
cpuinfo_transition_latency:0
energy_performance_available_preferences:default performance
balance_performance balance_power power 
energy_performance_preference:balance_performance
related_cpus:1
scaling_available_governors:performance powersave
scaling_cur_freq:1200000
scaling_driver:intel_pstate
scaling_governor:powersave
scaling_max_freq:4300000
scaling_min_freq:1200000
scaling_setspeed:<unsupported>


2. Now change the EPP

# echo 127 > energy_performance_preference
sudo rdmsr -a 0x774
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d

Good here

3. Offline/online good

[root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 >
/sys/devices/system/cpu/cpu1/online 
[root@otcpl-perf-test-skx-i9 cpufreq]# echo ` >
/sys/devices/system/cpu/cpu1/online 
> echo ` > /sys/devices/system/cpu/cpu1/online ^C
[root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 >
/sys/devices/system/cpu/cpu1/online 

 sudo rdmsr -a 0x774
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d
7f002b0c

Good. Online restored the setting

4. Now S3

rtcwake -m mem -s 10

All limits are now messed up

sudo rdmsr -a 0x774
80002b0c
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00

5. Now switch to passive
Again bad, some CPU max/min is 0

sudo rdmsr -a 0x774
80002b0d
7f002b0f
80002b0c
80002d0e
80002b0c
80002b0d
80002b0f
80002b2b
80002b0c
80002d1d
80000000
80002b0c
80002b0c
80000000
80000000
80000000
80000000
80000000
80000000
80000000

6.
Switched back to active to restore back

Lost EPP setting but rest are good.

sudo rdmsr -a 0x774
80002b0c
7f002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d

7. S3 again

rtcwake -m mem -s 10

Again messed up

 sudo rdmsr -a 0x774
80002b0c
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00

This time dmesg didn't have unchecked MSR in dmesg (I think because I
didn't change EPP before)

Thanks,
Srinivas

> Thanks,
> Rafael
> 
> 
>
[  416.434555] printk: Suspending console(s) (use no_console_suspend to debug)
[  416.436700] e1000e: EEE TX LPI TIMER: 00000011
[  416.442714] sd 1:0:0:0: [sda] Synchronizing SCSI cache
[  416.444511] sd 6:0:0:0: [sdb] Synchronizing SCSI cache
[  416.444606] sd 6:0:0:0: [sdb] Stopping disk
[  416.625785] sd 1:0:0:0: [sda] Stopping disk
[  416.988913] PM: suspend devices took 0.554 seconds
[  417.017162] ACPI: Preparing to enter system sleep state S3
[  417.017813] PM: Saving platform NVS memory
[  417.018033] Disabling non-boot CPUs ...
[  417.018765] intel_pstate: CPU 1 stopping
[  417.018768] intel_pstate: CPU 1 going offline
[  417.020144] smpboot: CPU 1 is now offline
[  417.022401] intel_pstate: CPU 2 stopping
[  417.022405] intel_pstate: CPU 2 going offline
[  417.023750] smpboot: CPU 2 is now offline
[  417.025089] intel_pstate: CPU 3 stopping
[  417.025093] intel_pstate: CPU 3 going offline
[  417.026481] smpboot: CPU 3 is now offline
[  417.028685] intel_pstate: CPU 4 stopping
[  417.028689] intel_pstate: CPU 4 going offline
[  417.030023] smpboot: CPU 4 is now offline
[  417.032171] intel_pstate: CPU 5 stopping
[  417.032176] intel_pstate: CPU 5 going offline
[  417.033499] smpboot: CPU 5 is now offline
[  417.034809] intel_pstate: CPU 6 stopping
[  417.034814] intel_pstate: CPU 6 going offline
[  417.036235] smpboot: CPU 6 is now offline
[  417.038330] intel_pstate: CPU 7 stopping
[  417.038334] intel_pstate: CPU 7 going offline
[  417.039679] smpboot: CPU 7 is now offline
[  417.041750] intel_pstate: CPU 8 stopping
[  417.041754] intel_pstate: CPU 8 going offline
[  417.044789] smpboot: CPU 8 is now offline
[  417.045953] intel_pstate: CPU 9 stopping
[  417.045956] intel_pstate: CPU 9 going offline
[  417.047253] smpboot: CPU 9 is now offline
[  417.049294] intel_pstate: CPU 10 stopping
[  417.049298] intel_pstate: CPU 10 going offline
[  417.050623] smpboot: CPU 10 is now offline
[  417.053509] intel_pstate: CPU 11 stopping
[  417.053513] intel_pstate: CPU 11 going offline
[  417.054814] smpboot: CPU 11 is now offline
[  417.055746] intel_pstate: CPU 12 stopping
[  417.055750] intel_pstate: CPU 12 going offline
[  417.057095] smpboot: CPU 12 is now offline
[  417.058158] intel_pstate: CPU 13 stopping
[  417.058162] intel_pstate: CPU 13 going offline
[  417.059500] smpboot: CPU 13 is now offline
[  417.060524] intel_pstate: CPU 14 stopping
[  417.060528] intel_pstate: CPU 14 going offline
[  417.061897] smpboot: CPU 14 is now offline
[  417.063000] intel_pstate: CPU 15 stopping
[  417.063003] intel_pstate: CPU 15 going offline
[  417.064369] smpboot: CPU 15 is now offline
[  417.065368] intel_pstate: CPU 16 stopping
[  417.065371] intel_pstate: CPU 16 going offline
[  417.066663] smpboot: CPU 16 is now offline
[  417.067633] intel_pstate: CPU 17 stopping
[  417.067636] intel_pstate: CPU 17 going offline
[  417.068992] smpboot: CPU 17 is now offline
[  417.069995] intel_pstate: CPU 18 stopping
[  417.069999] intel_pstate: CPU 18 going offline
[  417.071347] smpboot: CPU 18 is now offline
[  417.072377] intel_pstate: CPU 19 stopping
[  417.072380] intel_pstate: CPU 19 going offline
[  417.073757] smpboot: CPU 19 is now offline
[18430761567.317871] [Firmware Bug]: TSC ADJUST differs: CPU0 0 --> -3998613504. Restoring
[  417.076929] x86/cpu: VMX (outside TXT) disabled by BIOS

  417.076930] ACPI: Low-level resume complete
[  417.076990] PM: Restoring platform NVS memory
[  417.081736] smpboot: Scheduler frequency invariance went wobbly, disabling!
[  417.081782] Enabling non-boot CPUs ...
[  417.081823] x86: Booting SMP configuration:
[  417.081823] smpboot: Booting Node 0 Processor 1 APIC 0x2
[  417.085503] intel_pstate: CPU 1 going online
[  417.085513] unchecked MSR access error: WRMSR to 0x774 (tried to write 0x000000007f002b0c) at rIP: 0xffffffffa206e334 (native_write_msr+0x4/0x20)
[  417.085513] Call Trace:
[  417.085517]  __wrmsr_on_cpu+0x33/0x40
[  417.085520]  generic_exec_single+0x59/0x90
[  417.085523]  smp_call_function_single+0xde/0x120
[  417.085523]  ? __rdmsr_on_cpu+0x40/0x40
[  417.085526]  ? intel_pstate_init_acpi_perf_limits.isra.10+0x114/0x220
[  417.085527]  wrmsrl_on_cpu+0x58/0x80
[  417.085528]  intel_pstate_cpu_online+0x63/0x70
[  417.085530]  cpufreq_online+0x94/0x8a0
[  417.085532]  ? cpufreq_online+0x8a0/0x8a0
[  417.085533]  cpuhp_cpufreq_online+0xa/0x10
[  417.085535]  cpuhp_invoke_callback+0x94/0x540
[  417.085537]  ? padata_free_shell+0x80/0x80
[  417.085539]  ? sort_range+0x20/0x20
[  417.085540]  cpuhp_thread_fun+0xb0/0x110
[  417.085541]  smpboot_thread_fn+0xc5/0x160
[  417.085543]  kthread+0x116/0x130
[  417.085545]  ? kthread_park+0x80/0x80
[  417.085547]  ret_from_fork+0x22/0x30
[  417.085550] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.085552] unchecked MSR access error: RDMSR from 0x771 at rIP: 0xffffffffa206e253 (native_read_msr+0x3/0x30)
[  417.085552] Call Trace:
[  417.085553]  __rdmsr_on_cpu+0x2f/0x40
[  417.085554]  generic_exec_single+0x59/0x90
[  417.085555]  smp_call_function_single+0xde/0x120
[  417.085556]  ? pldmfw_flash_image+0xb30/0xb30
[  417.085556]  rdmsrl_on_cpu+0x57/0x80
[  417.085557]  intel_pstate_get_hwp_max+0x34/0x90
[  417.085558]  intel_pstate_update_perf_limits+0xe3/0x1f0
[  417.085559]  intel_pstate_set_policy+0x4c/0x175
[  417.085561]  cpufreq_set_policy+0x11c/0x350
[  417.085562]  cpufreq_online+0x7e8/0x8a0
[  417.085563]  ? cpufreq_online+0x8a0/0x8a0
[  417.085564]  cpuhp_cpufreq_online+0xa/0x10
[  417.085565]  cpuhp_invoke_callback+0x94/0x540
[  417.085566]  ? padata_free_shell+0x80/0x80
[  417.085567]  ? sort_range+0x20/0x20
[  417.085567]  cpuhp_thread_fun+0xb0/0x110
[  417.085568]  smpboot_thread_fn+0xc5/0x160
[  417.085569]  kthread+0x116/0x130
[  417.085570]  ? kthread_park+0x80/0x80
[  417.085571]  ret_from_fork+0x22/0x30
[  417.085572] intel_pstate: cpu:1 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.085573] intel_pstate: cpu:1 global_min:0 global_max:0
[  417.085573] intel_pstate: cpu:1 max_perf_ratio:0 min_perf_ratio:0
[  417.085699] CPU1 is up
[  417.085723] smpboot: Booting Node 0 Processor 2 APIC 0x4
[  417.089401] intel_pstate: CPU 2 going online
[  417.089409] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.089411] intel_pstate: cpu:2 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.089411] intel_pstate: cpu:2 global_min:0 global_max:0
[  417.089412] intel_pstate: cpu:2 max_perf_ratio:0 min_perf_ratio:0
[  417.089522] CPU2 is up
[  417.089547] smpboot: Booting Node 0 Processor 3 APIC 0x6
[  417.093177] intel_pstate: CPU 3 going online
[  417.093184] intel_pstate: set_policy cpuinfo.max 4500000 policy->max 4500000
[  417.093186] intel_pstate: cpu:3 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.093186] intel_pstate: cpu:3 global_min:0 global_max:0
[  417.093187] intel_pstate: cpu:3 max_perf_ratio:0 min_perf_ratio:0
[  417.093310] CPU3 is up
[  417.093334] smpboot: Booting Node 0 Processor 4 APIC 0x8
[  417.097093] intel_pstate: CPU 4 going online
[  417.097101] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.097102] intel_pstate: cpu:4 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.097103] intel_pstate: cpu:4 global_min:0 global_max:0
[  417.097103] intel_pstate: cpu:4 max_perf_ratio:0 min_perf_ratio:0
[  417.097234] CPU4 is up
[  417.097263] smpboot: Booting Node 0 Processor 5 APIC 0x10
[  417.101005] intel_pstate: CPU 5 going online
[  417.101014] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.101015] intel_pstate: cpu:5 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.101016] intel_pstate: cpu:5 global_min:0 global_max:0
[  417.101016] intel_pstate: cpu:5 max_perf_ratio:0 min_perf_ratio:0
[  417.101146] CPU5 is up
[  417.101171] smpboot: Booting Node 0 Processor 6 APIC 0x12
[  417.104799] intel_pstate: CPU 6 going online
[  417.104806] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.104808] intel_pstate: cpu:6 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.104809] intel_pstate: cpu:6 global_min:0 global_max:0
[  417.104809] intel_pstate: cpu:6 max_perf_ratio:0 min_perf_ratio:0
[  417.104952] CPU6 is up
[  417.104975] smpboot: Booting Node 0 Processor 7 APIC 0x14
[  417.108606] intel_pstate: CPU 7 going online
[  417.108615] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.108616] intel_pstate: cpu:7 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.108617] intel_pstate: cpu:7 global_min:0 global_max:0
[  417.108617] intel_pstate: cpu:7 max_perf_ratio:0 min_perf_ratio:0
[  417.108760] CPU7 is up
[  417.108784] smpboot: Booting Node 0 Processor 8 APIC 0x16
[  417.112458] intel_pstate: CPU 8 going online
[  417.112465] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.112467] intel_pstate: cpu:8 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.112467] intel_pstate: cpu:8 global_min:0 global_max:0
[  417.112468] intel_pstate: cpu:8 max_perf_ratio:0 min_perf_ratio:0
[  417.112620] CPU8 is up
[  417.112643] smpboot: Booting Node 0 Processor 9 APIC 0x18
[  417.116398] intel_pstate: CPU 9 going online
[  417.116406] intel_pstate: set_policy cpuinfo.max 4500000 policy->max 4500000
[  417.116407] intel_pstate: cpu:9 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.116408] intel_pstate: cpu:9 global_min:0 global_max:0
[  417.116408] intel_pstate: cpu:9 max_perf_ratio:0 min_perf_ratio:0
[  417.116572] CPU9 is up
[  417.116596] smpboot: Booting Node 0 Processor 10 APIC 0x1
[  417.117710] intel_pstate: CPU 10 going online
[  417.117717] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.117719] intel_pstate: cpu:10 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.117719] intel_pstate: cpu:10 global_min:0 global_max:0
[  417.117720] intel_pstate: cpu:10 max_perf_ratio:0 min_perf_ratio:0
[  417.117951] CPU10 is up
[  417.117976] smpboot: Booting Node 0 Processor 11 APIC 0x3
[  417.119177] intel_pstate: CPU 11 going online
[  417.119184] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.119186] intel_pstate: cpu:11 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.119186] intel_pstate: cpu:11 global_min:0 global_max:0
[  417.119187] intel_pstate: cpu:11 max_perf_ratio:0 min_perf_ratio:0
[  417.119383] CPU11 is up
[  417.119407] smpboot: Booting Node 0 Processor 12 APIC 0x5
[  417.120603] intel_pstate: CPU 12 going online
[  417.120610] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.120612] intel_pstate: cpu:12 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.120613] intel_pstate: cpu:12 global_min:0 global_max:0
[  417.120613] intel_pstate: cpu:12 max_perf_ratio:0 min_perf_ratio:0
[  417.120796] CPU12 is up
[  417.120820] smpboot: Booting Node 0 Processor 13 APIC 0x7
[  417.121939] intel_pstate: CPU 13 going online
[  417.121946] intel_pstate: set_policy cpuinfo.max 4500000 policy->max 4500000
[  417.121948] intel_pstate: cpu:13 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.121949] intel_pstate: cpu:13 global_min:0 global_max:0
[  417.121949] intel_pstate: cpu:13 max_perf_ratio:0 min_perf_ratio:0
[  417.122145] CPU13 is up
[  417.122172] smpboot: Booting Node 0 Processor 14 APIC 0x9
[  417.123417] intel_pstate: CPU 14 going online
[  417.123424] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.123426] intel_pstate: cpu:14 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.123426] intel_pstate: cpu:14 global_min:0 global_max:0
[  417.123427] intel_pstate: cpu:14 max_perf_ratio:0 min_perf_ratio:0
[  417.123625] CPU14 is up
[  417.123649] smpboot: Booting Node 0 Processor 15 APIC 0x11
[  417.124883] intel_pstate: CPU 15 going online
[  417.124890] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.124892] intel_pstate: cpu:15 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.124893] intel_pstate: cpu:15 global_min:0 global_max:0
[  417.124893] intel_pstate: cpu:15 max_perf_ratio:0 min_perf_ratio:0
[  417.125108] CPU15 is up
[  417.125132] smpboot: Booting Node 0 Processor 16 APIC 0x13
[  417.126320] intel_pstate: CPU 16 going online
[  417.126327] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.126329] intel_pstate: cpu:16 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.126329] intel_pstate: cpu:16 global_min:0 global_max:0
[  417.126330] intel_pstate: cpu:16 max_perf_ratio:0 min_perf_ratio:0
[  417.126551] CPU16 is up
[  417.126575] smpboot: Booting Node 0 Processor 17 APIC 0x15
[  417.127658] intel_pstate: CPU 17 going online
[  417.127665] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.127667] intel_pstate: cpu:17 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.127668] intel_pstate: cpu:17 global_min:0 global_max:0
[  417.127668] intel_pstate: cpu:17 max_perf_ratio:0 min_perf_ratio:0
[  417.127917] CPU17 is up
[  417.127940] smpboot: Booting Node 0 Processor 18 APIC 0x17
[  417.129146] intel_pstate: CPU 18 going online
[  417.129153] intel_pstate: set_policy cpuinfo.max 4300000 policy->max 4300000
[  417.129155] intel_pstate: cpu:18 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.129155] intel_pstate: cpu:18 global_min:0 global_max:0
[  417.129156] intel_pstate: cpu:18 max_perf_ratio:0 min_perf_ratio:0
[  417.129401] CPU18 is up
[  417.129431] smpboot: Booting Node 0 Processor 19 APIC 0x19
[  417.130681] intel_pstate: CPU 19 going online
[  417.130689] intel_pstate: set_policy cpuinfo.max 4500000 policy->max 4500000
[  417.130691] intel_pstate: cpu:19 max_state 0 min_policy_perf:0 max_policy_perf:0
[  417.130691] intel_pstate: cpu:19 global_min:0 global_max:0
[  417.130692] intel_pstate: cpu:19 max_perf_ratio:0 min_perf_ratio:0
[  417.130935] CPU19 is up
[  417.139360] ACPI: Waking up from system sleep state S3
[  417.176979] pci 0000:16:05.0: disabled boot interrupts on device [8086:2034]
[  417.177008] pci 0000:64:05.0: disabled boot interrupts on device [8086:2034]
[  417.177022] pci 0000:b2:05.0: disabled boot interrupts on device [8086:2034]
[  417.177378] usb usb1: root hub lost power or was reset
[  417.177379] usb usb2: root hub lost power or was reset
[  417.187092] sd 1:0:0:0: [sda] Starting disk
[  417.187147] sd 6:0:0:0: [sdb] Starting disk
[  417.516343] usb 1-2: reset high-speed USB device number 2 using xhci_hcd
[  417.519960] ata3: SATA link down (SStatus 4 SControl 300)
[  417.519974] ata8: SATA link down (SStatus 4 SControl 300)
[  417.520359] ata5: SATA link down (SStatus 4 SControl 300)
[  417.520441] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[  417.520616] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  417.520675] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  417.520698] ata4: SATA link down (SStatus 4 SControl 300)
[  417.520712] ata1: SATA link down (SStatus 4 SControl 300)
[  417.521676] ata7.00: configured for UDMA/133
[  417.521709] ata7.00: Enabling discard_zeroes_data
[  417.524724] ata6.00: configured for UDMA/133
[  417.561597] ata2.00: configured for UDMA/133
[  418.982436] usb 1-2.3: reset high-speed USB device number 3 using xhci_hcd
[  419.072372] PM: resume devices took 1.896 seconds
[  419.072386] OOM killer enabled.
[  419.072386] Restarting tasks ... done.
[  419.072802] PM: suspend exit
Doug Smythies Aug. 25, 2020, 1 a.m. UTC | #2
Hi Srinivas,

I think there is a disconnect between your written
description of what is going on and your supporting MSR reads.

On 2020.08.24 16:56 Srinivas Pandruvada wrote:
> On Mon, 2020-08-24 at 19:39 +0200, Rafael J. Wysocki wrote:
> > Hi All,
> >
> > The v2 is here to address feedback from Doug and one issue found by
> > me.
> >
> > The purpose of this series is to address some peculiarities related
> > to
> > taking CPUs offline/online and switching between different operation
> > modes with HWP enabled that have become visible after allowing the
> > driver to work in the passive mode with HWP enabled in 5.9-rc1 (and
> > one that was there earlier, but can be addressed easily after the
> > changes madein 5.9-rc1).
> >
> > Please refer to the patch changelogs for details.
> >
> > For easier testing/review, the series is available from the git
> > branch at:
> >
> >  git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git \
> >  intel_pstate-testing
> >
> 
> Applied these patches to 5.9-rc2

So did I, and the issues I reported the other day are fine now.
I did try a few of the things you were doing.

> 
> - After s3  limits got messed up.
>  # cat /sys/power/mem_sleep
> s2idle [deep]
> 
> - In the dmesg unchecked MSR for HWP register
> 
> 1.
> Before test
> 
> sudo rdmsr -a 0x774
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0d
> 7f002b0c

?? This looks like the MSR
read for further below, and for 
CPU 19 instead of 1.

> cd /sys/devices/system/cpu/intel_pstate/
> [root@otcpl-perf-test-skx-i9 intel_pstate]# grep . *
> hwp_dynamic_boost:0
> max_perf_pct:100
> min_perf_pct:27
> no_turbo:0
> num_pstates:32
> status:active
> turbo_pct:32
> 
> cd ../cpu1/cpufreq/
> [root@otcpl-perf-test-skx-i9 cpufreq]# grep . *
> affected_cpus:1
> base_frequency:3300000
> cpuinfo_max_freq:4300000
> cpuinfo_min_freq:1200000
> cpuinfo_transition_latency:0
> energy_performance_available_preferences:default performance
> balance_performance balance_power power
> energy_performance_preference:balance_performance
> related_cpus:1
> scaling_available_governors:performance powersave
> scaling_cur_freq:1200000
> scaling_driver:intel_pstate
> scaling_governor:powersave
> scaling_max_freq:4300000
> scaling_min_freq:1200000
> scaling_setspeed:<unsupported>
> 
> 
> 2. Now change the EPP
> 
> # echo 127 > energy_performance_preference
> sudo rdmsr -a 0x774
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0d

This looks like the original MSR read.

> 
> Good here
> 
> 3. Offline/online good
> 
> [root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 >
> /sys/devices/system/cpu/cpu1/online
> [root@otcpl-perf-test-skx-i9 cpufreq]# echo ` >
> /sys/devices/system/cpu/cpu1/online
> > echo ` > /sys/devices/system/cpu/cpu1/online ^C
> [root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 >
> /sys/devices/system/cpu/cpu1/online
> 
>  sudo rdmsr -a 0x774
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0d
> 7f002b0c

O.K.

> 
> Good. Online restored the setting
> 
> 4. Now S3
> 
> rtcwake -m mem -s 10

Cool command. I did not know about it.
I tried it.
> 
> All limits are now messed up
> 
> sudo rdmsr -a 0x774
> 80002b0c
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 

Yes, I got the same:

# /home/doug/c/msr-decoder (edited)
6.) 0x774: IA32_HWP_REQUEST:    CPU 0-5 :
    raw: 80002E08 : 8000FF01 : 8000FF01 : 8000FF01 : 8000FF01 : 8000FF01 :
    min:        8 :        1 :        1 :        1 :        1 :        1 :
    max:       46 :      255 :      255 :      255 :      255 :      255 :
    des:        0 :        0 :        0 :        0 :        0 :        0 :
    epp:      128 :      128 :      128 :      128 :      128 :      128 :
    act:        0 :        0 :        0 :        0 :        0 :        0 :

> 5. Now switch to passive
> Again bad, some CPU max/min is 0
> 
> sudo rdmsr -a 0x774
> 80002b0d
> 7f002b0f

Hmmm... Now seems to be CPU 1

> 80002b0c
> 80002d0e
> 80002b0c
> 80002b0d
> 80002b0f
> 80002b2b
> 80002b0c
> 80002d1d
> 80000000
> 80002b0c
> 80002b0c
> 80000000
> 80000000
> 80000000
> 80000000
> 80000000
> 80000000
> 80000000

MSR 774 was good for me, but in general my decoder was having troubles.

0x774: IA32_HWP_REQUEST:    CPU 0-5 :
sh: 0: getcwd() failed: No such file or directory
    raw: 80002E2E : 7F002E2E : 80002E2E : 80002E2E : 80002E2E : 80002E2E :
    min:       46 :       46 :       46 :       46 :       46 :       46 :
    max:       46 :       46 :       46 :       46 :       46 :       46 :
    des:        0 :        0 :        0 :        0 :        0 :        0 :
    epp:      128 :      127 :      128 :      128 :      128 :      128 :
    act:        0 :        0 :        0 :        0 :        0 :        0 : 

> 
> 6.
> Switched back to active to restore back
> 
> Lost EPP setting but rest are good.
> 
> sudo rdmsr -a 0x774
> 80002b0c
> 7f002b0c

And again, now seems to be CPU1.

> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002b0c
> 80002d0d
> 
> 7. S3 again
> 
> rtcwake -m mem -s 10
> 
> Again messed up
> 
>  sudo rdmsr -a 0x774
> 80002b0c
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 8000ff00
> 
> This time dmesg didn't have unchecked MSR in dmesg (I think because I
> didn't change EPP before)
> 
> Thanks,
> Srinivas
> 
> > Thanks,
> > Rafael
> >
> >
> >
Srinivas Pandruvada Aug. 25, 2020, 3:11 p.m. UTC | #3
Hi Doug,

On Mon, 2020-08-24 at 18:00 -0700, Doug Smythies wrote:
> Hi Srinivas,
> 
> I think there is a disconnect between your written
> description of what is going on and your supporting MSR reads.
> 
I reproduced again.
I see the copy paste individual at the first place swapped.
I pasted the full output by direct copy - paste from the screen.

But the issues are still there.

[labuser@otcpl-perf-test-skx-i9 ~]$ sudo -s
[root@otcpl-perf-test-skx-i9 labuser]# rdmsr -a 0x774
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d
[root@otcpl-perf-test-skx-i9 labuser]# cd /sys/devices/system/cpu/cpu1
[root@otcpl-perf-test-skx-i9 cpu1]# cd cpufreq/
[root@otcpl-perf-test-skx-i9 cpufreq]# echo 127 >
energy_performance_preference 
[root@otcpl-perf-test-skx-i9 cpufreq]# cat
energy_performance_preference 
127
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774
80002b0c
7f002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d
[root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 >
/sys/devices/system/cpu/cpu1/online 
[root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 >
/sys/devices/system/cpu/cpu1/online 
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002d0c
80002b0c
80002b0c
80002b0c
80002b0c
80002b0c
80002d0d
7f002b0c
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 1 0x774
7f002b0c
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 19 0x774
80002d0d
[root@otcpl-perf-test-skx-i9 cpufreq]# rtcwake -m mem -s 10
rtcwake: wakeup from "mem" using /dev/rtc0 at Tue Aug 25 15:04:02 2020
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774
80002b0c
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
8000ff00
[root@otcpl-perf-test-skx-i9 cpufreq]# dmesg > ~/temp/dmesg.txt
[root@otcpl-perf-test-skx-i9 cpufreq]# cat
energy_performance_preference 
127
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 1 0x774
8000ff00
[root@otcpl-perf-test-skx-i9 cpufreq]# echo passive >
/sys/devices/system/
clockevents/  clocksource/  container/    cpu/          edac/         m
achinecheck/ memory/       node/         
[root@otcpl-perf-test-skx-i9 cpufreq]# echo passive >
/sys/devices/system/cpu/intel_pstate/status 
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774
80002b0c
7f000000
80000000
80002d0c
80002b0c
80000000
80000000
80002b0c
80000000
80002d0c
80000000
80002b0c
80002b0d
80002d0c
80000000
80000000
80000000
80002b0d
80000000
80000000
[root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774
80002b0c
7f000000
80000000
80002d0c
80002b0c
80000000
80000000
80002b0c
80000000
80002d0c
80000000
80002b0c
80002b0d
80002d0c
80000000
80002b0c
80000000
80002b0d
80000000
80000000

> On 2020.08.24 16:56 Srinivas Pandruvada wrote:
> > On Mon, 2020-08-24 at 19:39 +0200, Rafael J. Wysocki wrote:
> > > Hi All,
> > > 
> > > The v2 is here to address feedback from Doug and one issue found
> > > by
> > > me.
> > > 
> > > The purpose of this series is to address some peculiarities
> > > related
> > > to
> > > taking CPUs offline/online and switching between different
> > > operation
> > > modes with HWP enabled that have become visible after allowing
> > > the
> > > driver to work in the passive mode with HWP enabled in 5.9-rc1
> > > (and
> > > one that was there earlier, but can be addressed easily after the
> > > changes madein 5.9-rc1).
> > > 
> > > Please refer to the patch changelogs for details.
> > > 
> > > For easier testing/review, the series is available from the git
> > > branch at:
> > > 
> > >  git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-
> > > pm.git \
> > >  intel_pstate-testing
> > > 
> > 
> > Applied these patches to 5.9-rc2
> 
> So did I, and the issues I reported the other day are fine now.
> I did try a few of the things you were doing.
> 
> > - After s3  limits got messed up.
> >  # cat /sys/power/mem_sleep
> > s2idle [deep]
> > 
> > - In the dmesg unchecked MSR for HWP register
> > 
> > 1.
> > Before test
> > 
> > sudo rdmsr -a 0x774
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0d
> > 7f002b0c
> 
> ?? This looks like the MSR
> read for further below, and for 
> CPU 19 instead of 1.
> 
> > cd /sys/devices/system/cpu/intel_pstate/
> > [root@otcpl-perf-test-skx-i9 intel_pstate]# grep . *
> > hwp_dynamic_boost:0
> > max_perf_pct:100
> > min_perf_pct:27
> > no_turbo:0
> > num_pstates:32
> > status:active
> > turbo_pct:32
> > 
> > cd ../cpu1/cpufreq/
> > [root@otcpl-perf-test-skx-i9 cpufreq]# grep . *
> > affected_cpus:1
> > base_frequency:3300000
> > cpuinfo_max_freq:4300000
> > cpuinfo_min_freq:1200000
> > cpuinfo_transition_latency:0
> > energy_performance_available_preferences:default performance
> > balance_performance balance_power power
> > energy_performance_preference:balance_performance
> > related_cpus:1
> > scaling_available_governors:performance powersave
> > scaling_cur_freq:1200000
> > scaling_driver:intel_pstate
> > scaling_governor:powersave
> > scaling_max_freq:4300000
> > scaling_min_freq:1200000
> > scaling_setspeed:<unsupported>
> > 
> > 
> > 2. Now change the EPP
> > 
> > # echo 127 > energy_performance_preference
> > sudo rdmsr -a 0x774
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0d
> 
> This looks like the original MSR read.
> 
> > Good here
> > 
> > 3. Offline/online good
> > 
> > [root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 >
> > /sys/devices/system/cpu/cpu1/online
> > [root@otcpl-perf-test-skx-i9 cpufreq]# echo ` >
> > /sys/devices/system/cpu/cpu1/online
> > > echo ` > /sys/devices/system/cpu/cpu1/online ^C
> > [root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 >
> > /sys/devices/system/cpu/cpu1/online
> > 
> >  sudo rdmsr -a 0x774
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0d
> > 7f002b0c
> 
> O.K.
> 
> > Good. Online restored the setting
> > 
> > 4. Now S3
> > 
> > rtcwake -m mem -s 10
> 
> Cool command. I did not know about it.
> I tried it.
> > All limits are now messed up
> > 
> > sudo rdmsr -a 0x774
> > 80002b0c
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 
> 
> Yes, I got the same:
> 
> # /home/doug/c/msr-decoder (edited)
> 6.) 0x774: IA32_HWP_REQUEST:    CPU 0-5 :
>     raw: 80002E08 : 8000FF01 : 8000FF01 : 8000FF01 : 8000FF01 :
> 8000FF01 :
>     min:        8 :        1 :        1 :        1 :        1
> :        1 :
>     max:       46 :      255 :      255 :      255 :      255
> :      255 :
>     des:        0 :        0 :        0 :        0 :        0
> :        0 :
>     epp:      128 :      128 :      128 :      128 :      128
> :      128 :
>     act:        0 :        0 :        0 :        0 :        0
> :        0 :
> 
> > 5. Now switch to passive
> > Again bad, some CPU max/min is 0
> > 
> > sudo rdmsr -a 0x774
> > 80002b0d
> > 7f002b0f
> 
> Hmmm... Now seems to be CPU 1
> 
> > 80002b0c
> > 80002d0e
> > 80002b0c
> > 80002b0d
> > 80002b0f
> > 80002b2b
> > 80002b0c
> > 80002d1d
> > 80000000
> > 80002b0c
> > 80002b0c
> > 80000000
> > 80000000
> > 80000000
> > 80000000
> > 80000000
> > 80000000
> > 80000000
> 
> MSR 774 was good for me, but in general my decoder was having
> troubles.
> 
> 0x774: IA32_HWP_REQUEST:    CPU 0-5 :
> sh: 0: getcwd() failed: No such file or directory
>     raw: 80002E2E : 7F002E2E : 80002E2E : 80002E2E : 80002E2E :
> 80002E2E :
>     min:       46 :       46 :       46 :       46 :       46
> :       46 :
>     max:       46 :       46 :       46 :       46 :       46
> :       46 :
>     des:        0 :        0 :        0 :        0 :        0
> :        0 :
>     epp:      128 :      127 :      128 :      128 :      128
> :      128 :
>     act:        0 :        0 :        0 :        0 :        0
> :        0 : 
> 
> > 6.
> > Switched back to active to restore back
> > 
> > Lost EPP setting but rest are good.
> > 
> > sudo rdmsr -a 0x774
> > 80002b0c
> > 7f002b0c
> 
> And again, now seems to be CPU1.
> 
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002b0c
> > 80002d0d
> > 
> > 7. S3 again
> > 
> > rtcwake -m mem -s 10
> > 
> > Again messed up
> > 
> >  sudo rdmsr -a 0x774
> > 80002b0c
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 8000ff00
> > 
> > This time dmesg didn't have unchecked MSR in dmesg (I think because
> > I
> > didn't change EPP before)
> > 
> > Thanks,
> > Srinivas
> > 
> > > Thanks,
> > > Rafael
> > > 
> > > 
> > >
Doug Smythies Aug. 25, 2020, 9:07 p.m. UTC | #4
Hi Srinivas,

Thanks for your reply.

On 2020.08.25 08:12 Srinivas Pandruvada wrote:
> On Mon, 2020-08-24 at 18:00 -0700, Doug Smythies wrote:
> > I think there is a disconnect between your written
> > description of what is going on and your supporting MSR reads.
> >
> I reproduced again.
> I see the copy paste individual at the first place swapped.

Yes, and that had me confused, initially.

> I pasted the full output by direct copy - paste from the screen.
> 
> But the issues are still there.

Agreed.
I didn't try your offline/online of CPU 1 part previously,
but did now, and get the same results as you.

I did not know that "rdmsr -a 0x774" lists
stuff in the order that CPU were last brought on-line.
I had assumed the list was in CPU order. Weird.

My example (nothing new here, just me catching up.
The offline/online order was cpu1, then cpu3, then cpu2):

root@s18:/sys/devices/system/cpu# grep . cpu*/cpufreq/energy_performance_preference
cpu0/cpufreq/energy_performance_preference:balance_performance
cpu1/cpufreq/energy_performance_preference:127
cpu2/cpufreq/energy_performance_preference:125
cpu3/cpufreq/energy_performance_preference:126
cpu4/cpufreq/energy_performance_preference:balance_performance
cpu5/cpufreq/energy_performance_preference:balance_performance
root@s18:/sys/devices/system/cpu# rdmsr -p 0 0x774
80002e2e
root@s18:/sys/devices/system/cpu# rdmsr -p 1 0x774
7f002e2e
root@s18:/sys/devices/system/cpu# rdmsr -p 2 0x774
7d002e2e
root@s18:/sys/devices/system/cpu# rdmsr -p 3 0x774
7e002e2e
root@s18:/sys/devices/system/cpu# rdmsr -p 4 0x774
80002e2e
root@s18:/sys/devices/system/cpu# rdmsr -p 5 0x774
80002e2e
root@s18:/sys/devices/system/cpu# rdmsr -a 0x774
80002e2e
80002e2e
80002e2e
7f002e2e
7e002e2e
7d002e2e

... Doug