[RFC i-g-t] GEM features into feat_profile.json

[RFC i-g-t] GEM features into feat_profile.json

[Joonas Lahtinen]

Hello,

I went through the gem_* tests from intel-gpu-tools and categorized
them into roughly categories "X | X robustness | X performance" ready
to be added to the feat_profile.json.

Lets open a discussion which ones should go where. I tried to place a
single test to under only one category and I'm kind of hopeful that
we'll have the ability to add "depends_on" to create super features in
the future, instead of placing a single test under multiple categories.

I didn't check all the subtests nor wildcard matching with other tests,
this is just all the test names placed under some categories.

Regards, Joonas

PS. I'm still bit unsure where to place gem_spin_batch.c while tests
gem_pipe_control_store_loop, gem_bad_address, gem_bad_batch,
gem_bad_blit, gem_bad_length and gem_bad_reloc seem more like hardware
sanity checks.
---
    "Driver API" : {
        "include_tests": "gem_basic",
    },
    "Driver API robustness" : {
        "include_tests": "gem_close_race|gem_fd_exhaustion|gem_eio",
    },
    "Basic IOCTLs": {
        "include_tests": "gem_create|gem_busy|gem_unref_active_buffers",
    },
    "R/W contention" : {
       "include_tests": "gem_gtt_cpu_tlb|gem_pread_after_blit|gem_pwrite_snooped|gem_readwrite|gem_caching|gem_exec_flush|gem_stress",
    },
    "R/W concurrency" : {
        "include_tests": "gem_concurrent_all|gem_concurrent_blit|gem_exec_parallel|gem_partial_pwrite_pread",
    },
    "Relocations" : {
        "include_tests": "gem_exec_reloc|gem_cpu_reloc|gem_persistent_relocs",
    },
    "Relocations robustness" : {
        "include_tests": "gem_exec_faulting_reloc|gem_exec_bad_domains|gem_reloc_overflow|gem_reloc_vs_gpu",
    },
    "Command streamer robustness" : {
        "include_tests": "gem_cs_tlb|gem_cs_prefetch",
    },
    "Contexts" : {
        "include_tests": "gem_ctx_basic|gem_ctx_exec|gem_ctx_param",
    },
    "Context robustness" : {
       "include_tests": "gem_ctx_create|gem_ctx_bad_destroy|gem_ctx_bad_exec|gem_ctx_switch|gem_ctx_thrash",
    },
    "Interrupts robustness" : {
       "include_tests": "gem_double_irq_loop|gem_ring_sync_loop|gem_sync",
    },
    "Eviction" : {
       "include_tests": "gem_evict|gem_exec_gttfill|gem_shrink",
    },
    "Execbuf IOCTL" : {
       "include_tests": "gem_exec_basic|gem_exec_alignment|gem_exec_lut_handle|gem_lut_handle",
    },
    "Execbuf IOCTL performance" : {
       "include_tests": "gem_exec_blt|gem_exec_create|gem_exec_latency|gem_exec_nop|gem_exec_reuse",
    },
    "Execbuf IOCTL robustness" : {
       "include_tests": "gem_exec_params|gem_fenced_exec_thrash|gem_exec_whisper",
    },
    "Suspend and resume" : {
       "include_tests": "gem_exec_suspend",
    },
    "Scheduler" : {
       "include_tests": "gem_exec_schedule",
    },
    "Asynchronous execution" : {
       "include_tests": "gem_exec_async",
    },
    "Command parser" : {
       "include_tests": "gem_exec_parse",
    },
    "Fences, implicit" : {
       "include_tests": "gem_exec_await",
    },
    "Fences, explicit" : {
       "include_tests": "gem_exec_fence",
    },
    "Tiling" : {
       "include_tests": "gem_fence_upload|gem_set_tiling_vs_blt|gem_set_tiling_vs_gtt|gem_set_tiling_vs_pwrite|gem_tiled_pread_basic|gem_tiled_pread_pwrite|gem_tiled_wb|gem_tiled_wc|gem_tiling_max_stride",
    },
    "Tiling robustness" : {
       "include_tests": "gem_fence_thrash|gem_gtt_hog|gem_render_tiled_blits|gem_threaded_access_tiled|gem_tiled_blits|gem_tiled_fence_blits|gem_tiled_swapping|gem_tiled_partial_pwrite_pread|gem_unfence_active_buffers",
    },
    "GGTT contention" : {
       "include_tests": "gem_gtt_hog|gem_linear_blits",
    },
    "Global objects" : {
       "include_tests": "gem_flink_basic",
    },
    "Global objects robustness" : {
       "include_tests": "gem_flink_race",
    },
    "Workloads" : {
       "include_tests": "gem_storedw_batches_loop|gem_storedw_loop",
    },
    "Workloads, GPGPU" : {
       "include_tests": "gem_gpgpu_fill",
    },
    "Workloads, media" : {
       "include_tests": "gem_media_fill",
    },
    "Workloads, render" : {
       "include_tests": "gem_render_copy",
    },
    "Workloads robustness, render" : {
       "include_tests": "gem_render_copy_redux|gem_render_linear_blits",
    },
    "Workloads robustness" : {
       "include_tests": "gem_ringfill",
    },
    "GGTT performance" : {
       "include_tests": "gem_gtt_speed",
    },
    "Hang recovery" : {
       "include_tests": "gem_hang|gem_hangcheck_forcewake|gem_exec_capture",
    },
    "Large object support" : {
       "include_tests": "gem_exec_big|gem_largeobject",
    },
    "Purging objects" : {
       "include_tests": "gem_madvise",
    },
    "mmap IOCTL" : {
       "include_tests": "gem_mmap|gem_mmap_gtt|gem_mmap_wc",
    },
    "mmap IOCTL robustness" : {
       "include_tests": "gem_mmap_offset_exhaustion",
    },
    "MOCS" : {
       "include_tests": "gem_mocs_settings",
    },
    "Secure batches robustness" : {
       "include_tests": "gem_non_secure_batch",
    },
    "Pinning" : {
       "include_tests": "gem_pin",
    },
    "PPGTT" : {
       "include_tests": "gem_ppgtt",
    },
    "R/W performance" : {
       "include_tests": "gem_pread|gem_pwrite|gem_pwrite_pread|gem_read_read_speed",
    },
    "Register read IOCTL" : {
       "include_tests": "gem_reg_read|gem_workarounds",
    },
    "Request retirement" : {
       "include_tests": "gem_request_retire",
    },
    "Reset statistics IOCTL" : {
       "include_tests": "gem_reset_stats",
    },
    "Sequence number wrapping" : {
       "include_tests": "gem_seqno_wrap",
    },
    "Soft pinning API" : {
       "include_tests": "gem_softpin",
    },
    "Stolen memory" : {
       "include_tests": "gem_stolen",
    },
    "Streaming writes" : {
       "include_tests": "gem_streaming_writes",
    },
    "User pointers API" : {
       "include_tests": "gem_userptr_blits",
    },
    "Wait IOCTL" : {
       "include_tests": "gem_wait",
    },
    "Wait IOCTL" : {
       "include_tests": "gem_wait",
    },
    "Semaphores robustness" : {
       "include_tests": "gem_write_read_ring_switch",
    }
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [RFC i-g-t] GEM features into feat_profile.json

[Chris Wilson]

On Wed, May 31, 2017 at 03:23:12PM +0300, Joonas Lahtinen wrote:
> Hello,
> 
> I went through the gem_* tests from intel-gpu-tools and categorized
> them into roughly categories "X | X robustness | X performance" ready
> to be added to the feat_profile.json.
> 
> Lets open a discussion which ones should go where. I tried to place a
> single test to under only one category and I'm kind of hopeful that
> we'll have the ability to add "depends_on" to create super features in
> the future, instead of placing a single test under multiple categories.
> 
> I didn't check all the subtests nor wildcard matching with other tests,
> this is just all the test names placed under some categories.

You seem to have assigned them exclusively to one category or another,
most tests belong to a few of these categories. More when you consider a
subtest may be targetting a completely different aspect.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [RFC i-g-t] GEM features into feat_profile.json

[Joonas Lahtinen]

On ke, 2017-05-31 at 13:58 +0100, Chris Wilson wrote:
> On Wed, May 31, 2017 at 03:23:12PM +0300, Joonas Lahtinen wrote:
> > 
> > Hello,
> > 
> > I went through the gem_* tests from intel-gpu-tools and categorized
> > them into roughly categories "X | X robustness | X performance" ready
> > to be added to the feat_profile.json.
> > 
> > Lets open a discussion which ones should go where. I tried to place a
> > single test to under only one category and I'm kind of hopeful that
> > we'll have the ability to add "depends_on" to create super features in
> > the future, instead of placing a single test under multiple categories.
> > 
> > I didn't check all the subtests nor wildcard matching with other tests,
> > this is just all the test names placed under some categories.
> 
> You seem to have assigned them exclusively to one category or another,
> most tests belong to a few of these categories. More when you consider a
> subtest may be targetting a completely different aspect.

Yes, that's what I meant to say :) Subtests should probably be matched
by another pattern like "\btiled\b", "\bflink\b" etc.

Ultimately there would be a resolver which would re-assign the
subtests:

"Global objects" would then get:

	"include_subtests": "flink",

Which would steal subtests with /\bflink\b/ from tests. Do we agree
that one subtest would be assigned to one category only, or do you
want to see duplication even at that level?

Regards, Joonas
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [RFC i-g-t] GEM features into feat_profile.json

[Chris Wilson]

On Wed, May 31, 2017 at 04:45:16PM +0300, Joonas Lahtinen wrote:
> On ke, 2017-05-31 at 13:58 +0100, Chris Wilson wrote:
> > On Wed, May 31, 2017 at 03:23:12PM +0300, Joonas Lahtinen wrote:
> > > 
> > > Hello,
> > > 
> > > I went through the gem_* tests from intel-gpu-tools and categorized
> > > them into roughly categories "X | X robustness | X performance" ready
> > > to be added to the feat_profile.json.
> > > 
> > > Lets open a discussion which ones should go where. I tried to place a
> > > single test to under only one category and I'm kind of hopeful that
> > > we'll have the ability to add "depends_on" to create super features in
> > > the future, instead of placing a single test under multiple categories.
> > > 
> > > I didn't check all the subtests nor wildcard matching with other tests,
> > > this is just all the test names placed under some categories.
> > 
> > You seem to have assigned them exclusively to one category or another,
> > most tests belong to a few of these categories. More when you consider a
> > subtest may be targetting a completely different aspect.
> 
> Yes, that's what I meant to say :) Subtests should probably be matched
> by another pattern like "\btiled\b", "\bflink\b" etc.
> 
> Ultimately there would be a resolver which would re-assign the
> subtests:
> 
> "Global objects" would then get:
> 
> 	"include_subtests": "flink",
> 
> Which would steal subtests with /\bflink\b/ from tests. Do we agree
> that one subtest would be assigned to one category only, or do you
> want to see duplication even at that level?

I see duplication everywhere. It's more a concept of tags as opposed to
categories.

The use of such a system would be as
	"give me all the tests that exercise relocation"
	"give me all the tests that use a context"
	"give me all the tests that exercise contention on $mutex"
	"give me all the tests that exercise file.c:line / this patch set"

The last one especially.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [RFC i-g-t] GEM features into feat_profile.json

[Daniel Vetter]

On Wed, May 31, 2017 at 03:02:18PM +0100, Chris Wilson wrote:
> On Wed, May 31, 2017 at 04:45:16PM +0300, Joonas Lahtinen wrote:
> > On ke, 2017-05-31 at 13:58 +0100, Chris Wilson wrote:
> > > On Wed, May 31, 2017 at 03:23:12PM +0300, Joonas Lahtinen wrote:
> > > > 
> > > > Hello,
> > > > 
> > > > I went through the gem_* tests from intel-gpu-tools and categorized
> > > > them into roughly categories "X | X robustness | X performance" ready
> > > > to be added to the feat_profile.json.
> > > > 
> > > > Lets open a discussion which ones should go where. I tried to place a
> > > > single test to under only one category and I'm kind of hopeful that
> > > > we'll have the ability to add "depends_on" to create super features in
> > > > the future, instead of placing a single test under multiple categories.
> > > > 
> > > > I didn't check all the subtests nor wildcard matching with other tests,
> > > > this is just all the test names placed under some categories.
> > > 
> > > You seem to have assigned them exclusively to one category or another,
> > > most tests belong to a few of these categories. More when you consider a
> > > subtest may be targetting a completely different aspect.
> > 
> > Yes, that's what I meant to say :) Subtests should probably be matched
> > by another pattern like "\btiled\b", "\bflink\b" etc.
> > 
> > Ultimately there would be a resolver which would re-assign the
> > subtests:
> > 
> > "Global objects" would then get:
> > 
> > 	"include_subtests": "flink",
> > 
> > Which would steal subtests with /\bflink\b/ from tests. Do we agree
> > that one subtest would be assigned to one category only, or do you
> > want to see duplication even at that level?
> 
> I see duplication everywhere. It's more a concept of tags as opposed to
> categories.
> 
> The use of such a system would be as
> 	"give me all the tests that exercise relocation"
> 	"give me all the tests that use a context"
> 	"give me all the tests that exercise contention on $mutex"
> 	"give me all the tests that exercise file.c:line / this patch set"

file/line is probably out of scope for this, for that we'd need to have
coverage information for each testcase, and then we could map a diff to
the relevant set of testcases. Nifty, but probably an awful lot of work to
automate ...

> The last one especially.

Yeah that was kinda the idea of this, the entries wouldn't be only an
exclusive list of tests only (piglit does that grouping already), but
match patterns for a given feature. And a given test can easily show up in
multiple features (e.g. a testcase that tests gpu hangs vs. suspend/resume
obviously needs to be both a hangcheck test and an s/r test).

I think we can just extend stuff though.

Anyway, thanks a lot for kickstarting this, unfortunately my vacations
start in like 4h so can't really digg into this more before heading back.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx