[early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

Hi Dave & Daniel,

An early pull for v5.15 (there'll be more coming in a week or two),
consisting of the drm/scheduler conversion and a couple other small
series that one was based one.  Mostly sending this now because IIUC
danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
you disagree then speak up, and I'll instead include this in the main
pull request once that is ready.)

This also has a core patch to drop drm_gem_object_put_locked() now
that the last use of it is removed.

The following changes since commit ff1176468d368232b684f75e82563369208bc371:

  Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)

are available in the Git repository at:

  https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28

for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:

  drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)

----------------------------------------------------------------
Rob Clark (18):
      drm/msm: Let fences read directly from memptrs
      drm/msm: Signal fences sooner
      drm/msm: Split out devfreq handling
      drm/msm: Split out get_freq() helper
      drm/msm: Devfreq tuning
      drm/msm: Docs and misc cleanup
      drm/msm: Small submitqueue creation cleanup
      drm/msm: drop drm_gem_object_put_locked()
      drm: Drop drm_gem_object_put_locked()
      drm/msm/submit: Simplify out-fence-fd handling
      drm/msm: Consolidate submit bo state
      drm/msm: Track "seqno" fences by idr
      drm/msm: Return ERR_PTR() from submit_create()
      drm/msm: Conversion to drm scheduler
      drm/msm: Drop submit bo_list
      drm/msm: Drop struct_mutex in submit path
      drm/msm: Utilize gpu scheduler priorities
      drm/msm/gem: Mark active before pinning

 drivers/gpu/drm/drm_gem.c                   |  22 --
 drivers/gpu/drm/msm/Kconfig                 |   1 +
 drivers/gpu/drm/msm/Makefile                |   1 +
 drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
 drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
 drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
 drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
 drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
 drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
 drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
 drivers/gpu/drm/msm/msm_fence.c             |  53 +----
 drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
 drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
 drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
 drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
 drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
 drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
 drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
 drivers/gpu/drm/msm/msm_rd.c                |   6 +-
 drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
 drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
 drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
 include/drm/drm_gem.h                       |   2 -
 include/uapi/drm/msm_drm.h                  |  14 +-
 26 files changed, 865 insertions(+), 536 deletions(-)
 create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c

[early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

Hi Dave & Daniel,

An early pull for v5.15 (there'll be more coming in a week or two),
consisting of the drm/scheduler conversion and a couple other small
series that one was based one.  Mostly sending this now because IIUC
danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
you disagree then speak up, and I'll instead include this in the main
pull request once that is ready.)

This also has a core patch to drop drm_gem_object_put_locked() now
that the last use of it is removed.

The following changes since commit ff1176468d368232b684f75e82563369208bc371:

  Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)

are available in the Git repository at:

  https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28

for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:

  drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)

----------------------------------------------------------------
Rob Clark (18):
      drm/msm: Let fences read directly from memptrs
      drm/msm: Signal fences sooner
      drm/msm: Split out devfreq handling
      drm/msm: Split out get_freq() helper
      drm/msm: Devfreq tuning
      drm/msm: Docs and misc cleanup
      drm/msm: Small submitqueue creation cleanup
      drm/msm: drop drm_gem_object_put_locked()
      drm: Drop drm_gem_object_put_locked()
      drm/msm/submit: Simplify out-fence-fd handling
      drm/msm: Consolidate submit bo state
      drm/msm: Track "seqno" fences by idr
      drm/msm: Return ERR_PTR() from submit_create()
      drm/msm: Conversion to drm scheduler
      drm/msm: Drop submit bo_list
      drm/msm: Drop struct_mutex in submit path
      drm/msm: Utilize gpu scheduler priorities
      drm/msm/gem: Mark active before pinning

 drivers/gpu/drm/drm_gem.c                   |  22 --
 drivers/gpu/drm/msm/Kconfig                 |   1 +
 drivers/gpu/drm/msm/Makefile                |   1 +
 drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
 drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
 drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
 drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
 drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
 drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
 drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
 drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
 drivers/gpu/drm/msm/msm_fence.c             |  53 +----
 drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
 drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
 drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
 drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
 drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
 drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
 drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
 drivers/gpu/drm/msm/msm_rd.c                |   6 +-
 drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
 drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
 drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
 include/drm/drm_gem.h                       |   2 -
 include/uapi/drm/msm_drm.h                  |  14 +-
 26 files changed, 865 insertions(+), 536 deletions(-)
 create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Caleb Connolly]

Hi Rob,

This series causes a fatal crash on my Oneplus 6, the device goes to 
Qualcomm crashdump mode shortly after reaching UI with the following errors:

https://paste.ubuntu.com/p/HvjmzZYtgw/

I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be 
the cause of the crash, reverting it resolves the issue.


On 28/07/2021 21:52, Rob Clark wrote:
> Hi Dave & Daniel,
> 
> An early pull for v5.15 (there'll be more coming in a week or two),
> consisting of the drm/scheduler conversion and a couple other small
> series that one was based one.  Mostly sending this now because IIUC
> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> you disagree then speak up, and I'll instead include this in the main
> pull request once that is ready.)
> 
> This also has a core patch to drop drm_gem_object_put_locked() now
> that the last use of it is removed.
> 
> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> 
>    Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> 
> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> 
>    drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> 
> ----------------------------------------------------------------
> Rob Clark (18):
>        drm/msm: Let fences read directly from memptrs
>        drm/msm: Signal fences sooner
>        drm/msm: Split out devfreq handling
>        drm/msm: Split out get_freq() helper
>        drm/msm: Devfreq tuning
>        drm/msm: Docs and misc cleanup
>        drm/msm: Small submitqueue creation cleanup
>        drm/msm: drop drm_gem_object_put_locked()
>        drm: Drop drm_gem_object_put_locked()
>        drm/msm/submit: Simplify out-fence-fd handling
>        drm/msm: Consolidate submit bo state
>        drm/msm: Track "seqno" fences by idr
>        drm/msm: Return ERR_PTR() from submit_create()
>        drm/msm: Conversion to drm scheduler
>        drm/msm: Drop submit bo_list
>        drm/msm: Drop struct_mutex in submit path
>        drm/msm: Utilize gpu scheduler priorities
>        drm/msm/gem: Mark active before pinning
> 
>   drivers/gpu/drm/drm_gem.c                   |  22 --
>   drivers/gpu/drm/msm/Kconfig                 |   1 +
>   drivers/gpu/drm/msm/Makefile                |   1 +
>   drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>   drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>   drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>   drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>   drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>   drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>   drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>   drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>   drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>   drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>   drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>   drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>   drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>   drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>   drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>   drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>   drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>   drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>   include/drm/drm_gem.h                       |   2 -
>   include/uapi/drm/msm_drm.h                  |  14 +-
>   26 files changed, 865 insertions(+), 536 deletions(-)
>   create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> 

-- 
Kind Regards,
Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Caleb Connolly]

Hi Rob,

This series causes a fatal crash on my Oneplus 6, the device goes to 
Qualcomm crashdump mode shortly after reaching UI with the following errors:

https://paste.ubuntu.com/p/HvjmzZYtgw/

I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be 
the cause of the crash, reverting it resolves the issue.


On 28/07/2021 21:52, Rob Clark wrote:
> Hi Dave & Daniel,
> 
> An early pull for v5.15 (there'll be more coming in a week or two),
> consisting of the drm/scheduler conversion and a couple other small
> series that one was based one.  Mostly sending this now because IIUC
> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> you disagree then speak up, and I'll instead include this in the main
> pull request once that is ready.)
> 
> This also has a core patch to drop drm_gem_object_put_locked() now
> that the last use of it is removed.
> 
> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> 
>    Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> 
> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> 
>    drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> 
> ----------------------------------------------------------------
> Rob Clark (18):
>        drm/msm: Let fences read directly from memptrs
>        drm/msm: Signal fences sooner
>        drm/msm: Split out devfreq handling
>        drm/msm: Split out get_freq() helper
>        drm/msm: Devfreq tuning
>        drm/msm: Docs and misc cleanup
>        drm/msm: Small submitqueue creation cleanup
>        drm/msm: drop drm_gem_object_put_locked()
>        drm: Drop drm_gem_object_put_locked()
>        drm/msm/submit: Simplify out-fence-fd handling
>        drm/msm: Consolidate submit bo state
>        drm/msm: Track "seqno" fences by idr
>        drm/msm: Return ERR_PTR() from submit_create()
>        drm/msm: Conversion to drm scheduler
>        drm/msm: Drop submit bo_list
>        drm/msm: Drop struct_mutex in submit path
>        drm/msm: Utilize gpu scheduler priorities
>        drm/msm/gem: Mark active before pinning
> 
>   drivers/gpu/drm/drm_gem.c                   |  22 --
>   drivers/gpu/drm/msm/Kconfig                 |   1 +
>   drivers/gpu/drm/msm/Makefile                |   1 +
>   drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>   drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>   drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>   drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>   drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>   drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>   drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>   drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>   drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>   drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>   drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>   drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>   drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>   drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>   drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>   drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>   drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>   drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>   drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>   include/drm/drm_gem.h                       |   2 -
>   include/uapi/drm/msm_drm.h                  |  14 +-
>   26 files changed, 865 insertions(+), 536 deletions(-)
>   create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> 

-- 
Kind Regards,
Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

Jordan, any idea if more frequent frequency changes would for some
reason make a630 grumpy?  I was expecting it should be somewhat
similar to a618 (same GMU fw, etc).  The main result of that patch
should be clamping to min freq when gpu goes idle, and the toggling
back to devfreq provided freq on idle->active transition.  So there
might be more frequent freq transitions.

Caleb, I don't suppose you could somehow delay starting UI and get
some traces?  Something along the lines of:

  localhost ~ # cd /sys/kernel/debug/tracing/
  localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
  localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
  localhost /sys/kernel/debug/tracing # cat trace_pipe

Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
That should bypass the clamping to min freq when the GPU isn't doing
anything and reduce the # of freq transitions.  I suppose we could
opt-in to this behavior on a per-gpu basis..

BR,
-R

On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
<caleb.connolly@linaro.org> wrote:
>
> Hi Rob,
>
> This series causes a fatal crash on my Oneplus 6, the device goes to
> Qualcomm crashdump mode shortly after reaching UI with the following errors:
>
> https://paste.ubuntu.com/p/HvjmzZYtgw/
>
> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
> the cause of the crash, reverting it resolves the issue.
>
>
> On 28/07/2021 21:52, Rob Clark wrote:
> > Hi Dave & Daniel,
> >
> > An early pull for v5.15 (there'll be more coming in a week or two),
> > consisting of the drm/scheduler conversion and a couple other small
> > series that one was based one.  Mostly sending this now because IIUC
> > danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> > you disagree then speak up, and I'll instead include this in the main
> > pull request once that is ready.)
> >
> > This also has a core patch to drop drm_gem_object_put_locked() now
> > that the last use of it is removed.
> >
> > The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> >
> >    Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> >
> > are available in the Git repository at:
> >
> >    https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> >
> > for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> >
> >    drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> >
> > ----------------------------------------------------------------
> > Rob Clark (18):
> >        drm/msm: Let fences read directly from memptrs
> >        drm/msm: Signal fences sooner
> >        drm/msm: Split out devfreq handling
> >        drm/msm: Split out get_freq() helper
> >        drm/msm: Devfreq tuning
> >        drm/msm: Docs and misc cleanup
> >        drm/msm: Small submitqueue creation cleanup
> >        drm/msm: drop drm_gem_object_put_locked()
> >        drm: Drop drm_gem_object_put_locked()
> >        drm/msm/submit: Simplify out-fence-fd handling
> >        drm/msm: Consolidate submit bo state
> >        drm/msm: Track "seqno" fences by idr
> >        drm/msm: Return ERR_PTR() from submit_create()
> >        drm/msm: Conversion to drm scheduler
> >        drm/msm: Drop submit bo_list
> >        drm/msm: Drop struct_mutex in submit path
> >        drm/msm: Utilize gpu scheduler priorities
> >        drm/msm/gem: Mark active before pinning
> >
> >   drivers/gpu/drm/drm_gem.c                   |  22 --
> >   drivers/gpu/drm/msm/Kconfig                 |   1 +
> >   drivers/gpu/drm/msm/Makefile                |   1 +
> >   drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
> >   drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
> >   drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
> >   drivers/gpu/drm/msm/msm_fence.c             |  53 +----
> >   drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
> >   drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
> >   drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
> >   drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
> >   drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
> >   drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
> >   drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
> >   drivers/gpu/drm/msm/msm_rd.c                |   6 +-
> >   drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
> >   drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
> >   drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
> >   include/drm/drm_gem.h                       |   2 -
> >   include/uapi/drm/msm_drm.h                  |  14 +-
> >   26 files changed, 865 insertions(+), 536 deletions(-)
> >   create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> >
>
> --
> Kind Regards,
> Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

Jordan, any idea if more frequent frequency changes would for some
reason make a630 grumpy?  I was expecting it should be somewhat
similar to a618 (same GMU fw, etc).  The main result of that patch
should be clamping to min freq when gpu goes idle, and the toggling
back to devfreq provided freq on idle->active transition.  So there
might be more frequent freq transitions.

Caleb, I don't suppose you could somehow delay starting UI and get
some traces?  Something along the lines of:

  localhost ~ # cd /sys/kernel/debug/tracing/
  localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
  localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
  localhost /sys/kernel/debug/tracing # cat trace_pipe

Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
That should bypass the clamping to min freq when the GPU isn't doing
anything and reduce the # of freq transitions.  I suppose we could
opt-in to this behavior on a per-gpu basis..

BR,
-R

On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
<caleb.connolly@linaro.org> wrote:
>
> Hi Rob,
>
> This series causes a fatal crash on my Oneplus 6, the device goes to
> Qualcomm crashdump mode shortly after reaching UI with the following errors:
>
> https://paste.ubuntu.com/p/HvjmzZYtgw/
>
> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
> the cause of the crash, reverting it resolves the issue.
>
>
> On 28/07/2021 21:52, Rob Clark wrote:
> > Hi Dave & Daniel,
> >
> > An early pull for v5.15 (there'll be more coming in a week or two),
> > consisting of the drm/scheduler conversion and a couple other small
> > series that one was based one.  Mostly sending this now because IIUC
> > danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> > you disagree then speak up, and I'll instead include this in the main
> > pull request once that is ready.)
> >
> > This also has a core patch to drop drm_gem_object_put_locked() now
> > that the last use of it is removed.
> >
> > The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> >
> >    Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> >
> > are available in the Git repository at:
> >
> >    https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> >
> > for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> >
> >    drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> >
> > ----------------------------------------------------------------
> > Rob Clark (18):
> >        drm/msm: Let fences read directly from memptrs
> >        drm/msm: Signal fences sooner
> >        drm/msm: Split out devfreq handling
> >        drm/msm: Split out get_freq() helper
> >        drm/msm: Devfreq tuning
> >        drm/msm: Docs and misc cleanup
> >        drm/msm: Small submitqueue creation cleanup
> >        drm/msm: drop drm_gem_object_put_locked()
> >        drm: Drop drm_gem_object_put_locked()
> >        drm/msm/submit: Simplify out-fence-fd handling
> >        drm/msm: Consolidate submit bo state
> >        drm/msm: Track "seqno" fences by idr
> >        drm/msm: Return ERR_PTR() from submit_create()
> >        drm/msm: Conversion to drm scheduler
> >        drm/msm: Drop submit bo_list
> >        drm/msm: Drop struct_mutex in submit path
> >        drm/msm: Utilize gpu scheduler priorities
> >        drm/msm/gem: Mark active before pinning
> >
> >   drivers/gpu/drm/drm_gem.c                   |  22 --
> >   drivers/gpu/drm/msm/Kconfig                 |   1 +
> >   drivers/gpu/drm/msm/Makefile                |   1 +
> >   drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
> >   drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
> >   drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
> >   drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
> >   drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
> >   drivers/gpu/drm/msm/msm_fence.c             |  53 +----
> >   drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
> >   drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
> >   drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
> >   drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
> >   drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
> >   drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
> >   drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
> >   drivers/gpu/drm/msm/msm_rd.c                |   6 +-
> >   drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
> >   drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
> >   drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
> >   include/drm/drm_gem.h                       |   2 -
> >   include/uapi/drm/msm_drm.h                  |  14 +-
> >   26 files changed, 865 insertions(+), 536 deletions(-)
> >   create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> >
>
> --
> Kind Regards,
> Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Caleb Connolly]

On 29/07/2021 02:02, Rob Clark wrote:
> Jordan, any idea if more frequent frequency changes would for some
> reason make a630 grumpy?  I was expecting it should be somewhat
> similar to a618 (same GMU fw, etc).  The main result of that patch
> should be clamping to min freq when gpu goes idle, and the toggling
> back to devfreq provided freq on idle->active transition.  So there
> might be more frequent freq transitions.
> 
> Caleb, I don't suppose you could somehow delay starting UI and get
> some traces?  Something along the lines of:
> 
>    localhost ~ # cd /sys/kernel/debug/tracing/
>    localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
>    localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
>    localhost /sys/kernel/debug/tracing # cat trace_pipe
Sure, here's the last ~1k lines of the trace logs: 
https://paste.ubuntu.com/p/XMKjKDWxYg/
And what I managed to get from dmesg before the crash (mostly the same 
as before): https://paste.ubuntu.com/p/kGVtRHDWKH/
> 
> Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
> That should bypass the clamping to min freq when the GPU isn't doing
> anything and reduce the # of freq transitions.  I suppose we could
> opt-in to this behavior on a per-gpu basis..
Yeah, that seems to resolve the issue, although I got the following 
probably unrelated (?) error on rebooting the device:
[  134.994449] [drm:dpu_encoder_vsync_event_handler:1749] [dpu 
error]invalid parameters

I wonder if the PocoPhone F1 has the same problem...
> 
> BR,
> -R
> 
> On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
> <caleb.connolly@linaro.org> wrote:
>>
>> Hi Rob,
>>
>> This series causes a fatal crash on my Oneplus 6, the device goes to
>> Qualcomm crashdump mode shortly after reaching UI with the following errors:
>>
>> https://paste.ubuntu.com/p/HvjmzZYtgw/
>>
>> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
>> the cause of the crash, reverting it resolves the issue.
>>
>>
>> On 28/07/2021 21:52, Rob Clark wrote:
>>> Hi Dave & Daniel,
>>>
>>> An early pull for v5.15 (there'll be more coming in a week or two),
>>> consisting of the drm/scheduler conversion and a couple other small
>>> series that one was based one.  Mostly sending this now because IIUC
>>> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
>>> you disagree then speak up, and I'll instead include this in the main
>>> pull request once that is ready.)
>>>
>>> This also has a core patch to drop drm_gem_object_put_locked() now
>>> that the last use of it is removed.
>>>
>>> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
>>>
>>>     Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
>>>
>>> are available in the Git repository at:
>>>
>>>     https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
>>>
>>> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
>>>
>>>     drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
>>>
>>> ----------------------------------------------------------------
>>> Rob Clark (18):
>>>         drm/msm: Let fences read directly from memptrs
>>>         drm/msm: Signal fences sooner
>>>         drm/msm: Split out devfreq handling
>>>         drm/msm: Split out get_freq() helper
>>>         drm/msm: Devfreq tuning
>>>         drm/msm: Docs and misc cleanup
>>>         drm/msm: Small submitqueue creation cleanup
>>>         drm/msm: drop drm_gem_object_put_locked()
>>>         drm: Drop drm_gem_object_put_locked()
>>>         drm/msm/submit: Simplify out-fence-fd handling
>>>         drm/msm: Consolidate submit bo state
>>>         drm/msm: Track "seqno" fences by idr
>>>         drm/msm: Return ERR_PTR() from submit_create()
>>>         drm/msm: Conversion to drm scheduler
>>>         drm/msm: Drop submit bo_list
>>>         drm/msm: Drop struct_mutex in submit path
>>>         drm/msm: Utilize gpu scheduler priorities
>>>         drm/msm/gem: Mark active before pinning
>>>
>>>    drivers/gpu/drm/drm_gem.c                   |  22 --
>>>    drivers/gpu/drm/msm/Kconfig                 |   1 +
>>>    drivers/gpu/drm/msm/Makefile                |   1 +
>>>    drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>>>    drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>>>    drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>>>    drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>>>    drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>>>    drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>>>    drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>>>    drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>>>    drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>>>    drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>>>    drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>>>    drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>>>    drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>>>    drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>>>    drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>>>    include/drm/drm_gem.h                       |   2 -
>>>    include/uapi/drm/msm_drm.h                  |  14 +-
>>>    26 files changed, 865 insertions(+), 536 deletions(-)
>>>    create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
>>>
>>
>> --
>> Kind Regards,
>> Caleb (they/them)

-- 
Kind Regards,
Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Caleb Connolly]

On 29/07/2021 02:02, Rob Clark wrote:
> Jordan, any idea if more frequent frequency changes would for some
> reason make a630 grumpy?  I was expecting it should be somewhat
> similar to a618 (same GMU fw, etc).  The main result of that patch
> should be clamping to min freq when gpu goes idle, and the toggling
> back to devfreq provided freq on idle->active transition.  So there
> might be more frequent freq transitions.
> 
> Caleb, I don't suppose you could somehow delay starting UI and get
> some traces?  Something along the lines of:
> 
>    localhost ~ # cd /sys/kernel/debug/tracing/
>    localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
>    localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
>    localhost /sys/kernel/debug/tracing # cat trace_pipe
Sure, here's the last ~1k lines of the trace logs: 
https://paste.ubuntu.com/p/XMKjKDWxYg/
And what I managed to get from dmesg before the crash (mostly the same 
as before): https://paste.ubuntu.com/p/kGVtRHDWKH/
> 
> Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
> That should bypass the clamping to min freq when the GPU isn't doing
> anything and reduce the # of freq transitions.  I suppose we could
> opt-in to this behavior on a per-gpu basis..
Yeah, that seems to resolve the issue, although I got the following 
probably unrelated (?) error on rebooting the device:
[  134.994449] [drm:dpu_encoder_vsync_event_handler:1749] [dpu 
error]invalid parameters

I wonder if the PocoPhone F1 has the same problem...
> 
> BR,
> -R
> 
> On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
> <caleb.connolly@linaro.org> wrote:
>>
>> Hi Rob,
>>
>> This series causes a fatal crash on my Oneplus 6, the device goes to
>> Qualcomm crashdump mode shortly after reaching UI with the following errors:
>>
>> https://paste.ubuntu.com/p/HvjmzZYtgw/
>>
>> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
>> the cause of the crash, reverting it resolves the issue.
>>
>>
>> On 28/07/2021 21:52, Rob Clark wrote:
>>> Hi Dave & Daniel,
>>>
>>> An early pull for v5.15 (there'll be more coming in a week or two),
>>> consisting of the drm/scheduler conversion and a couple other small
>>> series that one was based one.  Mostly sending this now because IIUC
>>> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
>>> you disagree then speak up, and I'll instead include this in the main
>>> pull request once that is ready.)
>>>
>>> This also has a core patch to drop drm_gem_object_put_locked() now
>>> that the last use of it is removed.
>>>
>>> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
>>>
>>>     Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
>>>
>>> are available in the Git repository at:
>>>
>>>     https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
>>>
>>> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
>>>
>>>     drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
>>>
>>> ----------------------------------------------------------------
>>> Rob Clark (18):
>>>         drm/msm: Let fences read directly from memptrs
>>>         drm/msm: Signal fences sooner
>>>         drm/msm: Split out devfreq handling
>>>         drm/msm: Split out get_freq() helper
>>>         drm/msm: Devfreq tuning
>>>         drm/msm: Docs and misc cleanup
>>>         drm/msm: Small submitqueue creation cleanup
>>>         drm/msm: drop drm_gem_object_put_locked()
>>>         drm: Drop drm_gem_object_put_locked()
>>>         drm/msm/submit: Simplify out-fence-fd handling
>>>         drm/msm: Consolidate submit bo state
>>>         drm/msm: Track "seqno" fences by idr
>>>         drm/msm: Return ERR_PTR() from submit_create()
>>>         drm/msm: Conversion to drm scheduler
>>>         drm/msm: Drop submit bo_list
>>>         drm/msm: Drop struct_mutex in submit path
>>>         drm/msm: Utilize gpu scheduler priorities
>>>         drm/msm/gem: Mark active before pinning
>>>
>>>    drivers/gpu/drm/drm_gem.c                   |  22 --
>>>    drivers/gpu/drm/msm/Kconfig                 |   1 +
>>>    drivers/gpu/drm/msm/Makefile                |   1 +
>>>    drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>>>    drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>>>    drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>>>    drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>>>    drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>>>    drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>>>    drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>>>    drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>>>    drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>>>    drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>>>    drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>>>    drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>>>    drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>>>    drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>>>    drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>>>    drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>>>    drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>>>    include/drm/drm_gem.h                       |   2 -
>>>    include/uapi/drm/msm_drm.h                  |  14 +-
>>>    26 files changed, 865 insertions(+), 536 deletions(-)
>>>    create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
>>>
>>
>> --
>> Kind Regards,
>> Caleb (they/them)

-- 
Kind Regards,
Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

On Wed, Jul 28, 2021 at 7:18 PM Caleb Connolly
<caleb.connolly@linaro.org> wrote:
>
>
>
> On 29/07/2021 02:02, Rob Clark wrote:
> > Jordan, any idea if more frequent frequency changes would for some
> > reason make a630 grumpy?  I was expecting it should be somewhat
> > similar to a618 (same GMU fw, etc).  The main result of that patch
> > should be clamping to min freq when gpu goes idle, and the toggling
> > back to devfreq provided freq on idle->active transition.  So there
> > might be more frequent freq transitions.
> >
> > Caleb, I don't suppose you could somehow delay starting UI and get
> > some traces?  Something along the lines of:
> >
> >    localhost ~ # cd /sys/kernel/debug/tracing/
> >    localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
> >    localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
> >    localhost /sys/kernel/debug/tracing # cat trace_pipe
> Sure, here's the last ~1k lines of the trace logs:
> https://paste.ubuntu.com/p/XMKjKDWxYg/
> And what I managed to get from dmesg before the crash (mostly the same
> as before): https://paste.ubuntu.com/p/kGVtRHDWKH/
> >
> > Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
> > That should bypass the clamping to min freq when the GPU isn't doing
> > anything and reduce the # of freq transitions.  I suppose we could
> > opt-in to this behavior on a per-gpu basis..
> Yeah, that seems to resolve the issue, although I got the following
> probably unrelated (?) error on rebooting the device:
> [  134.994449] [drm:dpu_encoder_vsync_event_handler:1749] [dpu
> error]invalid parameters

I think that should probably be unrelated..

Based on the traces, I'm seeing rapid toggling between idle freq and
non-idle freq.. but no invalid freq's (assuming the dts opp table is
correct) so I *guess* there is maybe some sort of race condition
communicating with GMU or some other issue with rapid freq transition?
 Maybe Jordan has some ideas.

The earlier dmesg you posted look like gpu getting cranky about what
looks like a valid opcode, and then it goes off into the weeds.. when
you start seeing things like "0xDEAFBEAF" I think that means the GPU
has lost context (ie. power collapse and back, and now it is reading
bogus power-on default values).

I think I can put together a patch to make the "clamp to min freq when
gpu is idle" opt-in so we can enable it per-gpu once someone has
confirmed that it doesn't cause problems.  I guess that would at least
work as a short term solution.  But not sure if that is just papering
over some gpu/gmu bug (or maybe gdsc/clk bug), or if it is a legit
workaround for some limitation..

BR,
-R

>
> I wonder if the PocoPhone F1 has the same problem...
> >
> > BR,
> > -R
> >
> > On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
> > <caleb.connolly@linaro.org> wrote:
> >>
> >> Hi Rob,
> >>
> >> This series causes a fatal crash on my Oneplus 6, the device goes to
> >> Qualcomm crashdump mode shortly after reaching UI with the following errors:
> >>
> >> https://paste.ubuntu.com/p/HvjmzZYtgw/
> >>
> >> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
> >> the cause of the crash, reverting it resolves the issue.
> >>
> >>
> >> On 28/07/2021 21:52, Rob Clark wrote:
> >>> Hi Dave & Daniel,
> >>>
> >>> An early pull for v5.15 (there'll be more coming in a week or two),
> >>> consisting of the drm/scheduler conversion and a couple other small
> >>> series that one was based one.  Mostly sending this now because IIUC
> >>> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> >>> you disagree then speak up, and I'll instead include this in the main
> >>> pull request once that is ready.)
> >>>
> >>> This also has a core patch to drop drm_gem_object_put_locked() now
> >>> that the last use of it is removed.
> >>>
> >>> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> >>>
> >>>     Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> >>>
> >>> are available in the Git repository at:
> >>>
> >>>     https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> >>>
> >>> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> >>>
> >>>     drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> >>>
> >>> ----------------------------------------------------------------
> >>> Rob Clark (18):
> >>>         drm/msm: Let fences read directly from memptrs
> >>>         drm/msm: Signal fences sooner
> >>>         drm/msm: Split out devfreq handling
> >>>         drm/msm: Split out get_freq() helper
> >>>         drm/msm: Devfreq tuning
> >>>         drm/msm: Docs and misc cleanup
> >>>         drm/msm: Small submitqueue creation cleanup
> >>>         drm/msm: drop drm_gem_object_put_locked()
> >>>         drm: Drop drm_gem_object_put_locked()
> >>>         drm/msm/submit: Simplify out-fence-fd handling
> >>>         drm/msm: Consolidate submit bo state
> >>>         drm/msm: Track "seqno" fences by idr
> >>>         drm/msm: Return ERR_PTR() from submit_create()
> >>>         drm/msm: Conversion to drm scheduler
> >>>         drm/msm: Drop submit bo_list
> >>>         drm/msm: Drop struct_mutex in submit path
> >>>         drm/msm: Utilize gpu scheduler priorities
> >>>         drm/msm/gem: Mark active before pinning
> >>>
> >>>    drivers/gpu/drm/drm_gem.c                   |  22 --
> >>>    drivers/gpu/drm/msm/Kconfig                 |   1 +
> >>>    drivers/gpu/drm/msm/Makefile                |   1 +
> >>>    drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
> >>>    drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
> >>>    drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
> >>>    drivers/gpu/drm/msm/msm_fence.c             |  53 +----
> >>>    drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
> >>>    drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
> >>>    drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
> >>>    drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
> >>>    drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
> >>>    drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
> >>>    drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
> >>>    drivers/gpu/drm/msm/msm_rd.c                |   6 +-
> >>>    drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
> >>>    drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
> >>>    drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
> >>>    include/drm/drm_gem.h                       |   2 -
> >>>    include/uapi/drm/msm_drm.h                  |  14 +-
> >>>    26 files changed, 865 insertions(+), 536 deletions(-)
> >>>    create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> >>>
> >>
> >> --
> >> Kind Regards,
> >> Caleb (they/them)
>
> --
> Kind Regards,
> Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Rob Clark]

On Wed, Jul 28, 2021 at 7:18 PM Caleb Connolly
<caleb.connolly@linaro.org> wrote:
>
>
>
> On 29/07/2021 02:02, Rob Clark wrote:
> > Jordan, any idea if more frequent frequency changes would for some
> > reason make a630 grumpy?  I was expecting it should be somewhat
> > similar to a618 (same GMU fw, etc).  The main result of that patch
> > should be clamping to min freq when gpu goes idle, and the toggling
> > back to devfreq provided freq on idle->active transition.  So there
> > might be more frequent freq transitions.
> >
> > Caleb, I don't suppose you could somehow delay starting UI and get
> > some traces?  Something along the lines of:
> >
> >    localhost ~ # cd /sys/kernel/debug/tracing/
> >    localhost /sys/kernel/debug/tracing # echo 1 > events/drm_msm_gpu/enable
> >    localhost /sys/kernel/debug/tracing # echo 1 > tracing_on
> >    localhost /sys/kernel/debug/tracing # cat trace_pipe
> Sure, here's the last ~1k lines of the trace logs:
> https://paste.ubuntu.com/p/XMKjKDWxYg/
> And what I managed to get from dmesg before the crash (mostly the same
> as before): https://paste.ubuntu.com/p/kGVtRHDWKH/
> >
> > Does adding an 'if (1) return' at the top of msm_devfreq_idle() help?
> > That should bypass the clamping to min freq when the GPU isn't doing
> > anything and reduce the # of freq transitions.  I suppose we could
> > opt-in to this behavior on a per-gpu basis..
> Yeah, that seems to resolve the issue, although I got the following
> probably unrelated (?) error on rebooting the device:
> [  134.994449] [drm:dpu_encoder_vsync_event_handler:1749] [dpu
> error]invalid parameters

I think that should probably be unrelated..

Based on the traces, I'm seeing rapid toggling between idle freq and
non-idle freq.. but no invalid freq's (assuming the dts opp table is
correct) so I *guess* there is maybe some sort of race condition
communicating with GMU or some other issue with rapid freq transition?
 Maybe Jordan has some ideas.

The earlier dmesg you posted look like gpu getting cranky about what
looks like a valid opcode, and then it goes off into the weeds.. when
you start seeing things like "0xDEAFBEAF" I think that means the GPU
has lost context (ie. power collapse and back, and now it is reading
bogus power-on default values).

I think I can put together a patch to make the "clamp to min freq when
gpu is idle" opt-in so we can enable it per-gpu once someone has
confirmed that it doesn't cause problems.  I guess that would at least
work as a short term solution.  But not sure if that is just papering
over some gpu/gmu bug (or maybe gdsc/clk bug), or if it is a legit
workaround for some limitation..

BR,
-R

>
> I wonder if the PocoPhone F1 has the same problem...
> >
> > BR,
> > -R
> >
> > On Wed, Jul 28, 2021 at 5:35 PM Caleb Connolly
> > <caleb.connolly@linaro.org> wrote:
> >>
> >> Hi Rob,
> >>
> >> This series causes a fatal crash on my Oneplus 6, the device goes to
> >> Qualcomm crashdump mode shortly after reaching UI with the following errors:
> >>
> >> https://paste.ubuntu.com/p/HvjmzZYtgw/
> >>
> >> I did a git bisect and the patch ("drm/msm: Devfreq tuning") seems to be
> >> the cause of the crash, reverting it resolves the issue.
> >>
> >>
> >> On 28/07/2021 21:52, Rob Clark wrote:
> >>> Hi Dave & Daniel,
> >>>
> >>> An early pull for v5.15 (there'll be more coming in a week or two),
> >>> consisting of the drm/scheduler conversion and a couple other small
> >>> series that one was based one.  Mostly sending this now because IIUC
> >>> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> >>> you disagree then speak up, and I'll instead include this in the main
> >>> pull request once that is ready.)
> >>>
> >>> This also has a core patch to drop drm_gem_object_put_locked() now
> >>> that the last use of it is removed.
> >>>
> >>> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> >>>
> >>>     Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> >>>
> >>> are available in the Git repository at:
> >>>
> >>>     https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> >>>
> >>> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> >>>
> >>>     drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> >>>
> >>> ----------------------------------------------------------------
> >>> Rob Clark (18):
> >>>         drm/msm: Let fences read directly from memptrs
> >>>         drm/msm: Signal fences sooner
> >>>         drm/msm: Split out devfreq handling
> >>>         drm/msm: Split out get_freq() helper
> >>>         drm/msm: Devfreq tuning
> >>>         drm/msm: Docs and misc cleanup
> >>>         drm/msm: Small submitqueue creation cleanup
> >>>         drm/msm: drop drm_gem_object_put_locked()
> >>>         drm: Drop drm_gem_object_put_locked()
> >>>         drm/msm/submit: Simplify out-fence-fd handling
> >>>         drm/msm: Consolidate submit bo state
> >>>         drm/msm: Track "seqno" fences by idr
> >>>         drm/msm: Return ERR_PTR() from submit_create()
> >>>         drm/msm: Conversion to drm scheduler
> >>>         drm/msm: Drop submit bo_list
> >>>         drm/msm: Drop struct_mutex in submit path
> >>>         drm/msm: Utilize gpu scheduler priorities
> >>>         drm/msm/gem: Mark active before pinning
> >>>
> >>>    drivers/gpu/drm/drm_gem.c                   |  22 --
> >>>    drivers/gpu/drm/msm/Kconfig                 |   1 +
> >>>    drivers/gpu/drm/msm/Makefile                |   1 +
> >>>    drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
> >>>    drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
> >>>    drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
> >>>    drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
> >>>    drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
> >>>    drivers/gpu/drm/msm/msm_fence.c             |  53 +----
> >>>    drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
> >>>    drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
> >>>    drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
> >>>    drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
> >>>    drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
> >>>    drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
> >>>    drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
> >>>    drivers/gpu/drm/msm/msm_rd.c                |   6 +-
> >>>    drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
> >>>    drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
> >>>    drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
> >>>    include/drm/drm_gem.h                       |   2 -
> >>>    include/uapi/drm/msm_drm.h                  |  14 +-
> >>>    26 files changed, 865 insertions(+), 536 deletions(-)
> >>>    create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c
> >>>
> >>
> >> --
> >> Kind Regards,
> >> Caleb (they/them)
>
> --
> Kind Regards,
> Caleb (they/them)

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Daniel Vetter]

On Wed, Jul 28, 2021 at 01:52:42PM -0700, Rob Clark wrote:
> Hi Dave & Daniel,
> 
> An early pull for v5.15 (there'll be more coming in a week or two),
> consisting of the drm/scheduler conversion and a couple other small
> series that one was based one.  Mostly sending this now because IIUC
> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> you disagree then speak up, and I'll instead include this in the main
> pull request once that is ready.)

I like, but might be some time until I managed to rebase it all and adjust
msm too. Thanks for doing the early pull.
-Daniel

> 
> This also has a core patch to drop drm_gem_object_put_locked() now
> that the last use of it is removed.
> 
> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> 
>   Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> 
> are available in the Git repository at:
> 
>   https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> 
> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> 
>   drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> 
> ----------------------------------------------------------------
> Rob Clark (18):
>       drm/msm: Let fences read directly from memptrs
>       drm/msm: Signal fences sooner
>       drm/msm: Split out devfreq handling
>       drm/msm: Split out get_freq() helper
>       drm/msm: Devfreq tuning
>       drm/msm: Docs and misc cleanup
>       drm/msm: Small submitqueue creation cleanup
>       drm/msm: drop drm_gem_object_put_locked()
>       drm: Drop drm_gem_object_put_locked()
>       drm/msm/submit: Simplify out-fence-fd handling
>       drm/msm: Consolidate submit bo state
>       drm/msm: Track "seqno" fences by idr
>       drm/msm: Return ERR_PTR() from submit_create()
>       drm/msm: Conversion to drm scheduler
>       drm/msm: Drop submit bo_list
>       drm/msm: Drop struct_mutex in submit path
>       drm/msm: Utilize gpu scheduler priorities
>       drm/msm/gem: Mark active before pinning
> 
>  drivers/gpu/drm/drm_gem.c                   |  22 --
>  drivers/gpu/drm/msm/Kconfig                 |   1 +
>  drivers/gpu/drm/msm/Makefile                |   1 +
>  drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>  drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>  drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>  drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>  drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>  drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>  drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>  drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>  drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>  drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>  drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>  drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>  drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>  drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>  drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>  drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>  drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>  drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>  include/drm/drm_gem.h                       |   2 -
>  include/uapi/drm/msm_drm.h                  |  14 +-
>  26 files changed, 865 insertions(+), 536 deletions(-)
>  create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: [early pull] drm/msm: drm-msm-next-2021-07-28 for v5.15

[Daniel Vetter]

On Wed, Jul 28, 2021 at 01:52:42PM -0700, Rob Clark wrote:
> Hi Dave & Daniel,
> 
> An early pull for v5.15 (there'll be more coming in a week or two),
> consisting of the drm/scheduler conversion and a couple other small
> series that one was based one.  Mostly sending this now because IIUC
> danvet wanted it in drm-next so he could rebase on it.  (Daniel, if
> you disagree then speak up, and I'll instead include this in the main
> pull request once that is ready.)

I like, but might be some time until I managed to rebase it all and adjust
msm too. Thanks for doing the early pull.
-Daniel

> 
> This also has a core patch to drop drm_gem_object_put_locked() now
> that the last use of it is removed.
> 
> The following changes since commit ff1176468d368232b684f75e82563369208bc371:
> 
>   Linux 5.14-rc3 (2021-07-25 15:35:14 -0700)
> 
> are available in the Git repository at:
> 
>   https://gitlab.freedesktop.org/drm/msm.git drm-msm-next-2021-07-28
> 
> for you to fetch changes up to 4541e4f2225c30b0e9442be9eb2fb8b7086cdd1f:
> 
>   drm/msm/gem: Mark active before pinning (2021-07-28 09:19:00 -0700)
> 
> ----------------------------------------------------------------
> Rob Clark (18):
>       drm/msm: Let fences read directly from memptrs
>       drm/msm: Signal fences sooner
>       drm/msm: Split out devfreq handling
>       drm/msm: Split out get_freq() helper
>       drm/msm: Devfreq tuning
>       drm/msm: Docs and misc cleanup
>       drm/msm: Small submitqueue creation cleanup
>       drm/msm: drop drm_gem_object_put_locked()
>       drm: Drop drm_gem_object_put_locked()
>       drm/msm/submit: Simplify out-fence-fd handling
>       drm/msm: Consolidate submit bo state
>       drm/msm: Track "seqno" fences by idr
>       drm/msm: Return ERR_PTR() from submit_create()
>       drm/msm: Conversion to drm scheduler
>       drm/msm: Drop submit bo_list
>       drm/msm: Drop struct_mutex in submit path
>       drm/msm: Utilize gpu scheduler priorities
>       drm/msm/gem: Mark active before pinning
> 
>  drivers/gpu/drm/drm_gem.c                   |  22 --
>  drivers/gpu/drm/msm/Kconfig                 |   1 +
>  drivers/gpu/drm/msm/Makefile                |   1 +
>  drivers/gpu/drm/msm/adreno/a5xx_debugfs.c   |   4 +-
>  drivers/gpu/drm/msm/adreno/a5xx_gpu.c       |   6 +-
>  drivers/gpu/drm/msm/adreno/a5xx_power.c     |   2 +-
>  drivers/gpu/drm/msm/adreno/a5xx_preempt.c   |   7 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gmu.c       |  12 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gpu.c       |   6 +-
>  drivers/gpu/drm/msm/adreno/a6xx_gpu_state.c |   4 +-
>  drivers/gpu/drm/msm/adreno/adreno_gpu.c     |   6 +-
>  drivers/gpu/drm/msm/msm_drv.c               |  30 ++-
>  drivers/gpu/drm/msm/msm_fence.c             |  53 +----
>  drivers/gpu/drm/msm/msm_fence.h             |  44 +++-
>  drivers/gpu/drm/msm/msm_gem.c               |  94 +-------
>  drivers/gpu/drm/msm/msm_gem.h               |  47 ++--
>  drivers/gpu/drm/msm/msm_gem_submit.c        | 344 +++++++++++++++++-----------
>  drivers/gpu/drm/msm/msm_gpu.c               | 220 ++++--------------
>  drivers/gpu/drm/msm/msm_gpu.h               | 139 ++++++++++-
>  drivers/gpu/drm/msm/msm_gpu_devfreq.c       | 203 ++++++++++++++++
>  drivers/gpu/drm/msm/msm_rd.c                |   6 +-
>  drivers/gpu/drm/msm/msm_ringbuffer.c        |  69 +++++-
>  drivers/gpu/drm/msm/msm_ringbuffer.h        |  12 +
>  drivers/gpu/drm/msm/msm_submitqueue.c       |  53 +++--
>  include/drm/drm_gem.h                       |   2 -
>  include/uapi/drm/msm_drm.h                  |  14 +-
>  26 files changed, 865 insertions(+), 536 deletions(-)
>  create mode 100644 drivers/gpu/drm/msm/msm_gpu_devfreq.c

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch