Re: [Intel-gfx] [RFC PATCH 07/11] drm/i915: Expose PMU for Observation Architecture

From: Robert Bragg <robert@sixbynine.org>
To: dri-devel@lists.freedesktop.org, David Airlie <airlied@linux.ie>,
	Daniel Vetter <daniel.vetter@intel.com>,
	Ingo Molnar <mingo@redhat.com>,
	Peter Zijlstra <a.p.zijlstra@chello.nl>,
	Paul Mackerras <paulus@samba.org>,
	Chris Wilson <chris@chris-wilson.co.uk>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	intel-gfx@lists.freedesktop.org, linux-api@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [Intel-gfx] [RFC PATCH 07/11] drm/i915: Expose PMU for Observation Architecture
Date: Mon, 18 May 2015 17:36:52 +0100	[thread overview]
Message-ID: <CAMou1-1QsATpCLMxmzGNTWyTEic4GtihiaRib0GJ55vsLYyaCQ@mail.gmail.com> (raw)
In-Reply-To: <20150507145800.GZ22099@nuc-i3427.alporthouse.com>

On 7 May 2015 15:58, "Chris Wilson" <chris@chris-wilson.co.uk> wrote:
>
> On Thu, May 07, 2015 at 03:15:50PM +0100, Robert Bragg wrote:
> > +     /* We bypass the default perf core perf_paranoid_cpu() ||
> > +      * CAP_SYS_ADMIN check by using the PERF_PMU_CAP_IS_DEVICE
> > +      * flag and instead authenticate based on whether the current
> > +      * pid owns the specified context, or require CAP_SYS_ADMIN
> > +      * when collecting cross-context metrics.
> > +      */
> > +     dev_priv->oa_pmu.specific_ctx = NULL;
> > +     if (oa_attr.single_context) {
> > +             u32 ctx_id = oa_attr.ctx_id;
> > +             unsigned int drm_fd = oa_attr.drm_fd;
> > +             struct fd fd = fdget(drm_fd);
> > +
> > +             if (fd.file) {
>
> Specify a ctx and not providing the right fd should be its own error,
> either EBADF or EINVAL.

Right, I went for both in the end; EBADF if fdget fails and EINVAL if
the fd is ok but we fail to lookup a context with it.

>
> > +                     dev_priv->oa_pmu.specific_ctx =
> > +                             lookup_context(dev_priv, fd.file, ctx_id);
> > +             }
>
> Missing fdput

Ah yes; fixed.

>
> > +     }
> > +
> > +     if (!dev_priv->oa_pmu.specific_ctx && !capable(CAP_SYS_ADMIN))
> > +             return -EACCES;
> > +
> > +     mutex_lock(&dev_priv->dev->struct_mutex);
>
> i915_mutex_interruptible, probably best to couple into the GPU error
> handling here as well especially as init_oa_buffer() will go onto touch
> GPU internals.

Ok, using i915_mutex_interruptible makes sense, I've also moved the
locking into init_oa_buffer.

About the GPU error handling, do you have any thoughts on what could
be most helpful here? I'm thinking a.t.m of extending
i915_capture_reg_state() in i915_gpu_error.c to capture the OACONTROL
+ OASTATUS state and perhaps all the UCGCTL unit clock gating state
too.

>
> > +     ret = init_oa_buffer(event);
> > +     mutex_unlock(&dev_priv->dev->struct_mutex);
> > +
> > +     if (ret)
> > +             return ret;
> > +
> > +     BUG_ON(dev_priv->oa_pmu.exclusive_event);
> > +     dev_priv->oa_pmu.exclusive_event = event;
> > +
> > +     event->destroy = i915_oa_event_destroy;
> > +
> > +     /* PRM - observability performance counters:
> > +      *
> > +      *   OACONTROL, performance counter enable, note:
> > +      *
> > +      *   "When this bit is set, in order to have coherent counts,
> > +      *   RC6 power state and trunk clock gating must be disabled.
> > +      *   This can be achieved by programming MMIO registers as
> > +      *   0xA094=0 and 0xA090[31]=1"
> > +      *
> > +      *   In our case we are expected that taking pm + FORCEWAKE
> > +      *   references will effectively disable RC6 and trunk clock
> > +      *   gating.
> > +      */
> > +     intel_runtime_pm_get(dev_priv);
> > +     intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
>
> That is a nuisance. Aside: Why isn't OA inside the powerctx? Is a subset
> valid with forcewake? It does perturb the system greatly to disable rc6,
> so I wonder if it could be made optional?

Yes, it's a shame.

I probably only really know enough about the OA unit design to be
dangerous and won't try and comment in detail here, but I think
there's more to it than not saving state in a power context. As I
understand it, there were a number of design changes made to enable
OA+RC6 support for BDW+, including having the OA unit automatically
write out reports to the OA buffer when entering RC6.

I think just FORCEWAKE_RENDER would work here, but only say that
because it looks like HSW only has the render forcewake domain from
what I could tell.

I think I need to update the comment above these lines as I don't
think these will affect crclk gating; these just handle disabling RC6.

The WIP patch I sent out basically represents me trying to get to the
bottom of the clock gating constraints we have.

At this point I think I need to disable CS unit gating via UCGCTL1, as
well as DOP gating for the render trunk clock via MISCCPCTL but I'm
not entirely confident about that just yet. At least empirically I see
these fixing some issues in rudimentary testing.

>
> > +
> > +     return 0;
> > +}
> > +
> > +static void update_oacontrol(struct drm_i915_private *dev_priv)
> > +{
> > +     BUG_ON(!spin_is_locked(&dev_priv->oa_pmu.lock));
> > +
> > +     if (dev_priv->oa_pmu.event_active) {
> > +             unsigned long ctx_id = 0;
> > +             bool pinning_ok = false;
> > +
> > +             if (dev_priv->oa_pmu.specific_ctx) {
> > +                     struct intel_context *ctx =
> > +                             dev_priv->oa_pmu.specific_ctx;
> > +                     struct drm_i915_gem_object *obj =
> > +                             ctx->legacy_hw_ctx.rcs_state;
>
> If only there was ctx->legacy_hw_ctx.rcs_vma...

ok, not sure if this is a prod to add that, a heads up that this is
coming or seething because some prior attempt to add this was nack'd.

>
> > +
> > +                     if (i915_gem_obj_is_pinned(obj)) {
> > +                             ctx_id = i915_gem_obj_ggtt_offset(obj);
> > +                             pinning_ok = true;
> > +                     }
> > +             }
> > +
> > +             if ((ctx_id == 0 || pinning_ok)) {
> > +                     bool periodic = dev_priv->oa_pmu.periodic;
> > +                     u32 period_exponent = dev_priv->oa_pmu.period_exponent;
> > +                     u32 report_format = dev_priv->oa_pmu.oa_buffer.format;
> > +
> > +                     I915_WRITE(GEN7_OACONTROL,
> > +                                (ctx_id & GEN7_OACONTROL_CTX_MASK) |
> > +                                (period_exponent <<
> > +                                 GEN7_OACONTROL_TIMER_PERIOD_SHIFT) |
> > +                                (periodic ?
> > +                                 GEN7_OACONTROL_TIMER_ENABLE : 0) |
> > +                                (report_format <<
> > +                                 GEN7_OACONTROL_FORMAT_SHIFT) |
> > +                                (ctx_id ?
> > +                                 GEN7_OACONTROL_PER_CTX_ENABLE : 0) |
> > +                                GEN7_OACONTROL_ENABLE);
>
> I notice you don't use any write barriers...

ok, so I still haven't put write barriers within update_oacontrol()
itself, but I've now added mmiowb()s just before unlocking which is
done outside of the update_oacontrol(). I think a barrier just within
update_oacontrol() could be ok a.t.m while the pinning hooks currently
just use update_oacontol(), but in case we might introduce more
overlapping mmio configuration within these critical sections, waiting
until the unlock might be preferable. On the other hand, a.t.m the
pinning callbacks now have redundant wb()s while there is no specific
context filtering - not sure if that should be a concern.

Looking at this, I also didn't feel happy with the way I reset
oa_pmu->specific_context when destroying an event, considering that
->specific_context being set is what determines whether the pinning
callbacks may call update_oacontrol() asynchronously with respect to
the pmu methods. Although we know oacontrol will be disabled by the
time we come to destroy an event, it didn't seem great that that we
could be continuing to run update_oacontrol() up to the point where we
are resetting all the clock gating, power management and NOA enable
state. I'll attach a separate patch to see if you agree this is worth
changing.

Thanks for the comments.

- Robert

> -Chris
> --
> Chris Wilson, Intel Open Source Technology Centre