All of lore.kernel.org
 help / color / mirror / Atom feed
* [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs
@ 2021-01-01 10:11 bugzilla-daemon
  2021-01-01 10:32 ` [Bug 210993] " bugzilla-daemon
                   ` (12 more replies)
  0 siblings, 13 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-01 10:11 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

            Bug ID: 210993
           Summary: Intel frequency scaling causes electrical noise on
                    10th gen CPUs
           Product: Power Management
           Version: 2.5
    Kernel Version: 5.9.16
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: enhancement
          Priority: P1
         Component: cpufreq
          Assignee: linux-pm@vger.kernel.org
          Reporter: karolherbst@gmail.com
        Regression: No

One thing I noticed with my i7-10850H CPU is, that if there are cores with huge
differences in their frequencies (e.g. one at 900 MHz another at 4.9 GHz) some
electrical becomes noticable.

The noise can be reduced by limiting the available range the CPUs frequency can
be scaled to. Sadly the scaling_min_freq property is ignored outright and only
matters as long as the CPU stays idle, but as long as one core gets some load,
another one drops below scaling_min_freq, so I couldn't test if limiting the
lower bound even helps.

Putting one core at max load with stress -c 1 effectively eliminates all noise.

This issue is even more annoying on my laptop as it generally runs with the
fans turned off.

intel_pstate is used in active mode, but putting it in passive mode, disabling
HWP or even using the ACPI freq scaling didn't really change anything in this
regard. The powersave governor and default energy_performance_preference is
used.

Is there anything else which could be done to mitigate the problem? Just
putting a finger on the touchpad makes the noise quite noticeable as cores get
scaled to the max.

Thanks

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
@ 2021-01-01 10:32 ` bugzilla-daemon
  2021-01-03  6:45 ` bugzilla-daemon
                   ` (11 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-01 10:32 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #1 from Karol Herbst (karolherbst@gmail.com) ---
One thing I forgot to mention is, when I limit the max clock, and this just
works, the noise gets quieter the smaller the difference between min and max
is.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
  2021-01-01 10:32 ` [Bug 210993] " bugzilla-daemon
@ 2021-01-03  6:45 ` bugzilla-daemon
  2021-01-04 19:54 ` bugzilla-daemon
                   ` (10 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-03  6:45 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

Zhang Rui (rui.zhang@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO
                 CC|                            |rui.zhang@intel.com

--- Comment #2 from Zhang Rui (rui.zhang@intel.com) ---

(In reply to Karol Herbst from comment #0)
> One thing I noticed with my i7-10850H CPU is, that if there are cores with
> huge differences in their frequencies (e.g. one at 900 MHz another at 4.9
> GHz) some electrical becomes noticable.
> 
> The noise can be reduced by limiting the available range the CPUs frequency
> can be scaled to. Sadly the scaling_min_freq property is ignored outright
> and only matters as long as the CPU stays idle, but as long as one core gets
> some load, another one drops below scaling_min_freq, so I couldn't test if
> limiting the lower bound even helps.

Usually, if we didn't take Pe into account, the CPU frequency should not drop
below scaling_min_freq. This sounds like a bug to me.
please give turbostat output when this happens.

> 
> Putting one core at max load with stress -c 1 effectively eliminates all
> noise.

please give turbostat output when this happens.

> 
> This issue is even more annoying on my laptop as it generally runs with the
> fans turned off.
> 
> intel_pstate is used in active mode, but putting it in passive mode,
> disabling HWP or even using the ACPI freq scaling didn't really change
> anything in this regard. The powersave governor and default
> energy_performance_preference is used.
> 
> Is there anything else which could be done to mitigate the problem? Just
> putting a finger on the touchpad makes the noise quite noticeable as cores
> get scaled to the max.
> 
I'd prefer this is the fan noise rather than CPU electrical noise.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
  2021-01-01 10:32 ` [Bug 210993] " bugzilla-daemon
  2021-01-03  6:45 ` bugzilla-daemon
@ 2021-01-04 19:54 ` bugzilla-daemon
  2021-01-04 19:58 ` bugzilla-daemon
                   ` (9 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-04 19:54 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #3 from Karol Herbst (karolherbst@gmail.com) ---
Created attachment 294493
  --> https://bugzilla.kernel.org/attachment.cgi?id=294493&action=edit
turbostat output

first three data sets are "idle", next two are finger on the touchpad, last
ones are "idle" again.

min clock set to 2.4GHz

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (2 preceding siblings ...)
  2021-01-04 19:54 ` bugzilla-daemon
@ 2021-01-04 19:58 ` bugzilla-daemon
  2021-01-04 21:01 ` bugzilla-daemon
                   ` (8 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-04 19:58 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #4 from Karol Herbst (karolherbst@gmail.com) ---
(In reply to Zhang Rui from comment #2)
> (In reply to Karol Herbst from comment #0)
> > One thing I noticed with my i7-10850H CPU is, that if there are cores with
> > huge differences in their frequencies (e.g. one at 900 MHz another at 4.9
> > GHz) some electrical becomes noticable.
> > 
> > The noise can be reduced by limiting the available range the CPUs frequency
> > can be scaled to. Sadly the scaling_min_freq property is ignored outright
> > and only matters as long as the CPU stays idle, but as long as one core
> gets
> > some load, another one drops below scaling_min_freq, so I couldn't test if
> > limiting the lower bound even helps.
> 
> Usually, if we didn't take Pe into account, the CPU frequency should not
> drop below scaling_min_freq. This sounds like a bug to me.
> please give turbostat output when this happens.
> 
> > 
> > Putting one core at max load with stress -c 1 effectively eliminates all
> > noise.
> 
> please give turbostat output when this happens.
> 

attached below.

> > 
> > This issue is even more annoying on my laptop as it generally runs with the
> > fans turned off.
> > 
> > intel_pstate is used in active mode, but putting it in passive mode,
> > disabling HWP or even using the ACPI freq scaling didn't really change
> > anything in this regard. The powersave governor and default
> > energy_performance_preference is used.
> > 
> > Is there anything else which could be done to mitigate the problem? Just
> > putting a finger on the touchpad makes the noise quite noticeable as cores
> > get scaled to the max.
> > 
> I'd prefer this is the fan noise rather than CPU electrical noise.

Well, the fans are off, so it has to be something else. It might be something
related to the CPU, but the noise it noticeable and doesn't sound like a fan at
all.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (3 preceding siblings ...)
  2021-01-04 19:58 ` bugzilla-daemon
@ 2021-01-04 21:01 ` bugzilla-daemon
  2021-01-04 22:38 ` bugzilla-daemon
                   ` (7 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-04 21:01 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

Francisco Jerez (currojerez@riseup.net) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |currojerez@riseup.net

--- Comment #5 from Francisco Jerez (currojerez@riseup.net) ---
O_o, that's really weird.  I guess this could also be some sort of
electromagnetic interference (E.g. with your soundcard?  Does the problem occur
with the loudspeakers turned off?), or an issue with your PSU.

The apparent frequency dropping below scaling_min_freq might be an artifact of
how the busy frequency of a CPU thread is calculated: It should approximate the
ratio of executed clock cycles to the time it spent in C0 state, which can
deviate from its actual working frequency if the processor enters and exits
some C1+ state repeatedly since the transition may have some significant
latency.  According to your turbostat log the CPU cores that report a busy
frequency below scaling_min_freq seem to be handling quite a few interrupts
which is consistent with that explanation.  It doesn't necessarily indicate a
power management bug.

Does disabling some specific idle state (e.g via
/sys/devices/system/cpu/cpu*/cpuidle/state*/disable) have any effect on the
noise?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (4 preceding siblings ...)
  2021-01-04 21:01 ` bugzilla-daemon
@ 2021-01-04 22:38 ` bugzilla-daemon
  2021-01-04 23:20 ` bugzilla-daemon
                   ` (6 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-04 22:38 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #6 from Karol Herbst (karolherbst@gmail.com) ---
(In reply to Francisco Jerez from comment #5)
> O_o, that's really weird. I guess this could also be some sort of
> electromagnetic interference (E.g. with your soundcard?  Does the problem
> occur with the loudspeakers turned off?), or an issue with your PSU.
> 

It is very quiet. Definitely you can hear it only with slowly spinning fans and
no background noise. Searching through the web it sounds like a common issue
and disabling sleep states is supposed to be the proper "fix" for this.

But at least muting the speakers don't help.

> The apparent frequency dropping below scaling_min_freq might be an artifact
> of how the busy frequency of a CPU thread is calculated: It should
> approximate the ratio of executed clock cycles to the time it spent in C0
> state, which can deviate from its actual working frequency if the processor
> enters and exits some C1+ state repeatedly since the transition may have
> some significant latency.  According to your turbostat log the CPU cores
> that report a busy frequency below scaling_min_freq seem to be handling
> quite a few interrupts which is consistent with that explanation.  It
> doesn't necessarily indicate a power management bug.
> 
> Does disabling some specific idle state (e.g via
> /sys/devices/system/cpu/cpu*/cpuidle/state*/disable) have any effect on the
> noise?

disabling all except state0 stops the noise.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (5 preceding siblings ...)
  2021-01-04 22:38 ` bugzilla-daemon
@ 2021-01-04 23:20 ` bugzilla-daemon
  2021-01-05  2:19 ` bugzilla-daemon
                   ` (5 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-04 23:20 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #7 from Francisco Jerez (currojerez@riseup.net) ---
(In reply to Karol Herbst from comment #6)
>
> > Does disabling some specific idle state (e.g via
> > /sys/devices/system/cpu/cpu*/cpuidle/state*/disable) have any effect on the
> > noise?
> 
> disabling all except state0 stops the noise.

Ah that makes sense, that sounds like the culprit might be a noisy power supply
coupled to the oscillation of the processor's current draw caused by repeated
C-state transitions.  If that's the case you may be able to tweak the latency
parameters to make sure the oscillation frequency is outside the audible range,
though I'm not sure if it would make sense to upstream such a change since it
will likely come at an energy cost (Though I would expect the cost to be minor
as long as the processor can still reach the deepest sleep state in the steady
state while idle, might be a good trade-off in your specific case if this is
particularly annoying to you).

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (6 preceding siblings ...)
  2021-01-04 23:20 ` bugzilla-daemon
@ 2021-01-05  2:19 ` bugzilla-daemon
  2021-01-05  8:00 ` bugzilla-daemon
                   ` (4 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-05  2:19 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #8 from Zhang Rui (rui.zhang@intel.com) ---
(In reply to Karol Herbst from comment #3)
> Created attachment 294493 [details]
> turbostat output
> 
> first three data sets are "idle", next two are finger on the touchpad, last
> ones are "idle" again.
> 
> min clock set to 2.4GHz

"Sadly the scaling_min_freq property is ignored outright and only matters as
long as the CPU stays idle, but as long as one core gets some load, another one
drops below scaling_min_freq"

I didn't observe this from the turbostat output attached, because the system is
idle.
Can you please attach the turbostat output when the cpu frequency drops below
scaling_min_freq?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (7 preceding siblings ...)
  2021-01-05  2:19 ` bugzilla-daemon
@ 2021-01-05  8:00 ` bugzilla-daemon
  2021-01-05  8:10 ` bugzilla-daemon
                   ` (3 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-05  8:00 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #9 from Karol Herbst (karolherbst@gmail.com) ---
(In reply to Zhang Rui from comment #8)
> (In reply to Karol Herbst from comment #3)
> > Created attachment 294493 [details]
> > turbostat output
> > 
> > first three data sets are "idle", next two are finger on the touchpad, last
> > ones are "idle" again.
> > 
> > min clock set to 2.4GHz
> 
> "Sadly the scaling_min_freq property is ignored outright and only matters as
> long as the CPU stays idle, but as long as one core gets some load, another
> one drops below scaling_min_freq"
> 
> I didn't observe this from the turbostat output attached, because the system
> is idle.
> Can you please attach the turbostat output when the cpu frequency drops
> below scaling_min_freq?

it does in the output, just only one core per entry, and then only in the
entries in the middle.

e.g. line 122

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (8 preceding siblings ...)
  2021-01-05  8:00 ` bugzilla-daemon
@ 2021-01-05  8:10 ` bugzilla-daemon
  2021-04-22 13:23 ` bugzilla-daemon
                   ` (2 subsequent siblings)
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-01-05  8:10 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #10 from Karol Herbst (karolherbst@gmail.com) ---
(In reply to Francisco Jerez from comment #7)
> (In reply to Karol Herbst from comment #6)
> >
> > > Does disabling some specific idle state (e.g via
> > > /sys/devices/system/cpu/cpu*/cpuidle/state*/disable) have any effect on
> the
> > > noise?
> > 
> > disabling all except state0 stops the noise.
> 
> Ah that makes sense, that sounds like the culprit might be a noisy power
> supply coupled to the oscillation of the processor's current draw caused by
> repeated C-state transitions.  If that's the case you may be able to tweak
> the latency parameters to make sure the oscillation frequency is outside the
> audible range, though I'm not sure if it would make sense to upstream such a
> change since it will likely come at an energy cost (Though I would expect
> the cost to be minor as long as the processor can still reach the deepest
> sleep state in the steady state while idle, might be a good trade-off in
> your specific case if this is particularly annoying to you).

yeah... might be. Although it felt like this is a more common issue with newer
CPUs, especially 10th gen Intel CPUs and was wondering if there is a way to
mitigate it in software. Limiting the max clock helped to reduce it, so I was
wondering if there could be some nice ideas for people too lazy or not wanting
to bother with replacing the laptop or whatever.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (9 preceding siblings ...)
  2021-01-05  8:10 ` bugzilla-daemon
@ 2021-04-22 13:23 ` bugzilla-daemon
  2021-06-02  5:45 ` bugzilla-daemon
  2021-06-16  7:26 ` bugzilla-daemon
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-04-22 13:23 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

Zhang Rui (rui.zhang@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|cpufreq                     |intel_pstate

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (10 preceding siblings ...)
  2021-04-22 13:23 ` bugzilla-daemon
@ 2021-06-02  5:45 ` bugzilla-daemon
  2021-06-16  7:26 ` bugzilla-daemon
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-06-02  5:45 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

--- Comment #11 from Zhang Rui (rui.zhang@intel.com) ---
We haven't done such things in kernel before.
Compared with the noise, the frequency/performance is one of the indicators
that many people care about and are always measuring/monitoring.

For people who has a strong requirement of limited noise, they can use the tips
you mentioned above.

As this is not a software bug, I tends to close it.
What do you think?

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [Bug 210993] Intel frequency scaling causes electrical noise on 10th gen CPUs
  2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
                   ` (11 preceding siblings ...)
  2021-06-02  5:45 ` bugzilla-daemon
@ 2021-06-16  7:26 ` bugzilla-daemon
  12 siblings, 0 replies; 14+ messages in thread
From: bugzilla-daemon @ 2021-06-16  7:26 UTC (permalink / raw)
  To: linux-pm

https://bugzilla.kernel.org/show_bug.cgi?id=210993

Zhang Rui (rui.zhang@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEEDINFO                    |CLOSED
         Resolution|---                         |DOCUMENTED

--- Comment #12 from Zhang Rui (rui.zhang@intel.com) ---
Bug closed.
Please feel free to reopen if you still have any questions.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2021-06-16  7:26 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-01 10:11 [Bug 210993] New: Intel frequency scaling causes electrical noise on 10th gen CPUs bugzilla-daemon
2021-01-01 10:32 ` [Bug 210993] " bugzilla-daemon
2021-01-03  6:45 ` bugzilla-daemon
2021-01-04 19:54 ` bugzilla-daemon
2021-01-04 19:58 ` bugzilla-daemon
2021-01-04 21:01 ` bugzilla-daemon
2021-01-04 22:38 ` bugzilla-daemon
2021-01-04 23:20 ` bugzilla-daemon
2021-01-05  2:19 ` bugzilla-daemon
2021-01-05  8:00 ` bugzilla-daemon
2021-01-05  8:10 ` bugzilla-daemon
2021-04-22 13:23 ` bugzilla-daemon
2021-06-02  5:45 ` bugzilla-daemon
2021-06-16  7:26 ` bugzilla-daemon

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.