Craft a list of tests to run in simulation (and tune them)

Craft a list of tests to run in simulation (and tune them)

[Damien Lespiau]

For early bring-up, it's useful to have a list of tests to check everything is
in order. The Makefile file is a great place to keep such a list.

The IGT_SIMULATION environment variable controls the list of tests to run
(either with make test/check or with piglit) and allows us to tune the various
tests we execute to run in an acceptable amount of time in simulation.

[PATCH i-g-t 1/5] README: Reword a few sentences

[Damien Lespiau]

To be a bit more correct (they were understandable though).

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
---
 README | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/README b/README
index be29177..c484a93 100644
--- a/README
+++ b/README
@@ -27,17 +27,17 @@ tests/
 	Run this tests with "make test" as root from this directory. Note that
 	no other drm clients (X server) may run.
 
-	"make test" only runs a default of test usefull for regression testing.
-	Other tests not run are:
+	"make test" only runs a default set of tests and is useful for
+	regression testing. Other tests not run are:
 	- tests that might hang the gpu, see HANG in Makefile.am
 	- gem_stress, a stress test suite. Look at the source for all the
 	  various options.
 	- testdisplay is only run in the default mode. testdisplay has tons of
-	  options to test different kms functionality, again read the source of
+	  options to test different kms functionality, again read the source for
 	  the details.
 
 	The more comfortable way to run tests is with piglit. First grab piglit
-	from
+	from:
 
 	git://anongit.freedesktop.org/piglit
 
-- 
1.7.11.7

[PATCH i-g-t 2/5] lib: Rename IGT_QUICK to IGT_SIMULATION

[Damien Lespiau]

It's more accurate this way as the quick mode is really useful for in
the simulation environment.

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
---
 lib/drmtest.c | 10 +++++-----
 lib/drmtest.h |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/drmtest.c b/lib/drmtest.c
index a60a3a6..6e272d7 100644
--- a/lib/drmtest.c
+++ b/lib/drmtest.c
@@ -587,14 +587,14 @@ static bool env_set(const char *env_var)
 	return atoi(val) != 0;
 }
 
-bool drmtest_run_quick(void)
+bool drmtest_run_in_simulation(void)
 {
-	static int run_quick = -1;
+	static int simulation = -1;
 
-	if (run_quick == -1)
-		run_quick = env_set("IGT_QUICK");
+	if (simulation == -1)
+		simulation = env_set("IGT_SIMULATION");
 
-	return run_quick;
+	return simulation;
 }
 
 /* other helpers */
diff --git a/lib/drmtest.h b/lib/drmtest.h
index 78732a0..07f971c 100644
--- a/lib/drmtest.h
+++ b/lib/drmtest.h
@@ -88,8 +88,8 @@ void drmtest_subtest_init(int argc, char **argv);
 bool drmtest_run_subtest(const char *subtest_name);
 bool drmtest_only_list_subtests(void);
 
-bool drmtest_run_quick(void);
-#define SLOW_QUICK(slow,quick) (drmtest_run_quick() ? (quick) : (slow))
+bool drmtest_run_in_simulation(void);
+#define SLOW_QUICK(slow,quick) (drmtest_run_in_simulation() ? (quick) : (slow))
 
 /* helpers based upon the libdrm buffer manager */
 void drmtest_init_aperture_trashers(drm_intel_bufmgr *bufmgr);
-- 
1.7.11.7

[PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Damien Lespiau]

There are two ways to run tests, directly with make check/test
or through piglit.

When IGT_SIMULATION is set to '1', we substitute the list of tests in
those two code paths with carefully selected tests. The stress tests and
other horrors are left to torture the real hardware as they don't make
too much sense in simulation.

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
---
 tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2fddfe8..33db962 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -137,22 +137,59 @@ kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+# We maintain here a list of tests to run in our simulation environments
+single_simulation_tests = \
+	gem_mmap \
+	gem_pread_after_blit \
+	gem_ring_sync_loop \
+	gem_ctx_basic \
+	gem_pipe_control_store_loop \
+	gem_storedw_loop_render \
+	gem_storedw_loop_blt \
+	gem_storedw_loop_bsd \
+	gem_render_linear_blits \
+	gem_tiled_blits \
+	gem_cpu_reloc \
+	$(NULL)
+
+multi_simulation_tests = \
+	gem_exec_nop \
+	gem_mmap_gtt \
+	$(NULL)
+
+simulation_tests = \
+	$(single_simulation_tests) \
+	$(multi_simulation_tests) \
+	$(NULL)
+
 TESTS = \
 	$(NULL)
 
 test:
 	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
 	@./check_drm_clients
-	@make TESTS="${kernel_tests}" check
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		make TESTS="${simulation_tests}" check; \
+	else \
+		make TESTS="${kernel_tests}" check; \
+	fi
 
 list-single-tests:
 	@echo TESTLIST
-	@echo ${single_kernel_tests}
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		echo ${single_simulation_tests}; \
+	else \
+		echo ${single_kernel_tests}; \
+	fi
 	@echo END TESTLIST
 
 list-multi-tests:
 	@echo TESTLIST
-	@echo ${multi_kernel_tests}
+	@if [ "${IGT_SIMULATION}" == 1 ]; then \
+		echo ${multi_simulation_tests}; \
+	else \
+		echo ${multi_kernel_tests}; \
+	fi
 	@echo END TESTLIST
 
 HANG = \
-- 
1.7.11.7

[PATCH i-g-t 4/5] tests: Instrument tests run in simulation to run quickly

[Damien Lespiau]

We tweak the tests marked as runnable in simulation to run more quickly,
more often then not at the expense of stress testing (which is of an
arguable interest for the initial bring up in simulation). Hopefully the
values chosen still test something, which is not always straightforward.

It does run quickly, the number on an IVB machines are:

$ time sudo IGT_SIMULATION=0 ./piglit-run.py tests/igt.tests foo
[...]
real	2m0.141s
user	0m16.365s
sys	1m33.382s

Vs.

$ time sudo IGT_SIMULATION=1 ./piglit-run.py tests/igt.tests foo
[...]
real	0m0.448s
user	0m0.226s
sys	0m0.183s

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
---
 tests/gem_cpu_reloc.c               | 3 +++
 tests/gem_ctx_basic.c               | 5 +++++
 tests/gem_exec_nop.c                | 2 +-
 tests/gem_mmap_gtt.c                | 5 ++++-
 tests/gem_pipe_control_store_loop.c | 2 +-
 tests/gem_render_linear_blits.c     | 6 ++++++
 tests/gem_ring_sync_loop.c          | 2 +-
 tests/gem_storedw_batches_loop.c    | 8 +++++---
 tests/gem_storedw_loop_blt.c        | 8 +++++---
 tests/gem_storedw_loop_bsd.c        | 8 +++++---
 tests/gem_storedw_loop_render.c     | 8 +++++---
 tests/gem_tiled_blits.c             | 6 ++++++
 12 files changed, 47 insertions(+), 16 deletions(-)

diff --git a/tests/gem_cpu_reloc.c b/tests/gem_cpu_reloc.c
index baf8301..ad70e40 100644
--- a/tests/gem_cpu_reloc.c
+++ b/tests/gem_cpu_reloc.c
@@ -153,6 +153,9 @@ int main(int argc, char **argv)
 	}
 
 	count = aper_size / 4096 * 2;
+	if (drmtest_run_in_simulation())
+		count = 10;
+
 	handles = malloc (count * sizeof(uint32_t));
 	assert(handles);
 
diff --git a/tests/gem_ctx_basic.c b/tests/gem_ctx_basic.c
index 3757d84..729c79c 100644
--- a/tests/gem_ctx_basic.c
+++ b/tests/gem_ctx_basic.c
@@ -143,6 +143,11 @@ int main(int argc, char *argv[])
 	fd = drm_open_any();
 	devid = intel_get_drm_devid(fd);
 
+	if (drmtest_run_in_simulation()) {
+		num_contexts = 2;
+		iter = 4;
+	}
+
 	parse(argc, argv);
 
 	threads = calloc(num_contexts, sizeof(*threads));
diff --git a/tests/gem_exec_nop.c b/tests/gem_exec_nop.c
index 8608592..2307983 100644
--- a/tests/gem_exec_nop.c
+++ b/tests/gem_exec_nop.c
@@ -95,7 +95,7 @@ static void loop(int fd, uint32_t handle, unsigned ring_id, const char *ring_nam
 
 	skipped_all = false;
 
-	for (count = 1; count <= 1<<17; count <<= 1) {
+	for (count = 1; count <= SLOW_QUICK(1<<17, 1<<4); count <<= 1) {
 		struct timeval start, end;
 
 		gettimeofday(&start, NULL);
diff --git a/tests/gem_mmap_gtt.c b/tests/gem_mmap_gtt.c
index d759340..e53a942 100644
--- a/tests/gem_mmap_gtt.c
+++ b/tests/gem_mmap_gtt.c
@@ -40,7 +40,7 @@
 #include "i915_drm.h"
 #include "drmtest.h"
 
-#define OBJECT_SIZE (16*1024*1024)
+static int OBJECT_SIZE = 16*1024*1024;
 
 static void set_domain(int fd, uint32_t handle)
 {
@@ -148,6 +148,9 @@ int main(int argc, char **argv)
 {
 	int fd;
 
+	if (drmtest_run_in_simulation())
+		OBJECT_SIZE = 1 * 1024 * 1024;
+
 	drmtest_subtest_init(argc, argv);
 
 	fd = drm_open_any();
diff --git a/tests/gem_pipe_control_store_loop.c b/tests/gem_pipe_control_store_loop.c
index e03cddd..af6a758 100644
--- a/tests/gem_pipe_control_store_loop.c
+++ b/tests/gem_pipe_control_store_loop.c
@@ -70,7 +70,7 @@ store_pipe_control_loop(void)
 	uint32_t *buf;
 	drm_intel_bo *target_bo;
 
-	for (i = 0; i < 0x10000; i++) {
+	for (i = 0; i < SLOW_QUICK(0x10000, 4); i++) {
 		/* we want to check tlb consistency of the pipe_control target,
 		 * so get a new buffer every time around */
 		target_bo = drm_intel_bo_alloc(bufmgr, "target bo", 4096, 4096);
diff --git a/tests/gem_render_linear_blits.c b/tests/gem_render_linear_blits.c
index a7e0189..c94f451 100644
--- a/tests/gem_render_linear_blits.c
+++ b/tests/gem_render_linear_blits.c
@@ -81,8 +81,11 @@ int main(int argc, char **argv)
 	batch = intel_batchbuffer_alloc(bufmgr, intel_get_drm_devid(fd));
 
 	count = 0;
+	if (drmtest_run_in_simulation())
+		count = 2;
 	if (argc > 1)
 		count = atoi(argv[1]);
+
 	if (count == 0)
 		count = 3 * gem_aperture_size(fd) / SIZE / 2;
 	else if (count < 2) {
@@ -127,6 +130,9 @@ int main(int argc, char **argv)
 	for (i = 0; i < count; i++)
 		check_bo(fd, bo[i]->handle, start_val[i]);
 
+	if (drmtest_run_in_simulation())
+		return 0;
+
 	printf("Cyclic blits, backward...\n");
 	for (i = 0; i < count * 4; i++) {
 		struct scratch_buf src, dst;
diff --git a/tests/gem_ring_sync_loop.c b/tests/gem_ring_sync_loop.c
index b689bcd..6ba1052 100644
--- a/tests/gem_ring_sync_loop.c
+++ b/tests/gem_ring_sync_loop.c
@@ -62,7 +62,7 @@ store_dword_loop(void)
 
 	srandom(0xdeadbeef);
 
-	for (i = 0; i < 0x100000; i++) {
+	for (i = 0; i < SLOW_QUICK(0x100000, 10); i++) {
 		int ring = random() % 3 + 1;
 
 		if (ring == I915_EXEC_RENDER) {
diff --git a/tests/gem_storedw_batches_loop.c b/tests/gem_storedw_batches_loop.c
index 86c3469..7d07c59 100644
--- a/tests/gem_storedw_batches_loop.c
+++ b/tests/gem_storedw_batches_loop.c
@@ -60,7 +60,7 @@ store_dword_loop(int divider)
 	if (!has_ppgtt)
 		cmd |= MI_MEM_VIRTUAL;
 
-	for (i = 0; i < SLOW_QUICK(0x80000, 0x10); i++) {
+	for (i = 0; i < SLOW_QUICK(0x80000, 4); i++) {
 		cmd_bo = drm_intel_bo_alloc(bufmgr, "cmd bo", 4096, 4096);
 		if (!cmd_bo) {
 			fprintf(stderr, "failed to alloc cmd bo\n");
@@ -170,8 +170,10 @@ int main(int argc, char **argv)
 
 	store_dword_loop(1);
 	store_dword_loop(2);
-	store_dword_loop(3);
-	store_dword_loop(5);
+	if (!drmtest_run_in_simulation()) {
+		store_dword_loop(3);
+		store_dword_loop(5);
+	}
 
 	drm_intel_bo_unreference(target_bo);
 	drm_intel_bufmgr_destroy(bufmgr);
diff --git a/tests/gem_storedw_loop_blt.c b/tests/gem_storedw_loop_blt.c
index cacae3a..dfb613b 100644
--- a/tests/gem_storedw_loop_blt.c
+++ b/tests/gem_storedw_loop_blt.c
@@ -63,7 +63,7 @@ store_dword_loop(int divider)
 	if (!has_ppgtt)
 		cmd |= MI_MEM_VIRTUAL;
 
-	for (i = 0; i < SLOW_QUICK(0x100000, 0x10); i++) {
+	for (i = 0; i < SLOW_QUICK(0x100000, 4); i++) {
 		BEGIN_BATCH(4);
 		OUT_BATCH(cmd);
 		OUT_BATCH(0); /* reserved */
@@ -150,8 +150,10 @@ int main(int argc, char **argv)
 
 	store_dword_loop(1);
 	store_dword_loop(2);
-	store_dword_loop(3);
-	store_dword_loop(5);
+	if (!drmtest_run_in_simulation()) {
+		store_dword_loop(3);
+		store_dword_loop(5);
+	}
 
 	drm_intel_bo_unreference(target_buffer);
 	intel_batchbuffer_free(batch);
diff --git a/tests/gem_storedw_loop_bsd.c b/tests/gem_storedw_loop_bsd.c
index 232c3a6..6aefb9a 100644
--- a/tests/gem_storedw_loop_bsd.c
+++ b/tests/gem_storedw_loop_bsd.c
@@ -63,7 +63,7 @@ store_dword_loop(int divider)
 	if (!has_ppgtt)
 		cmd |= MI_MEM_VIRTUAL;
 
-	for (i = 0; i < SLOW_QUICK(0x100000, 0x10); i++) {
+	for (i = 0; i < SLOW_QUICK(0x100000, 4); i++) {
 		BEGIN_BATCH(4);
 		OUT_BATCH(cmd);
 		OUT_BATCH(0); /* reserved */
@@ -156,8 +156,10 @@ int main(int argc, char **argv)
 
 	store_dword_loop(1);
 	store_dword_loop(2);
-	store_dword_loop(3);
-	store_dword_loop(5);
+	if (!drmtest_run_in_simulation()) {
+		store_dword_loop(3);
+		store_dword_loop(5);
+	}
 
 	drm_intel_bo_unreference(target_buffer);
 	intel_batchbuffer_free(batch);
diff --git a/tests/gem_storedw_loop_render.c b/tests/gem_storedw_loop_render.c
index fe09834..e41ecc7 100644
--- a/tests/gem_storedw_loop_render.c
+++ b/tests/gem_storedw_loop_render.c
@@ -63,7 +63,7 @@ store_dword_loop(int divider)
 	if (!has_ppgtt)
 		cmd |= MI_MEM_VIRTUAL;
 
-	for (i = 0; i < SLOW_QUICK(0x100000, 0x10); i++) {
+	for (i = 0; i < SLOW_QUICK(0x100000, 4); i++) {
 		BEGIN_BATCH(4);
 		OUT_BATCH(cmd);
 		OUT_BATCH(0); /* reserved */
@@ -144,8 +144,10 @@ int main(int argc, char **argv)
 
 	store_dword_loop(1);
 	store_dword_loop(2);
-	store_dword_loop(3);
-	store_dword_loop(5);
+	if (!drmtest_run_in_simulation()) {
+		store_dword_loop(3);
+		store_dword_loop(5);
+	}
 
 	drm_intel_bo_unreference(target_buffer);
 	intel_batchbuffer_free(batch);
diff --git a/tests/gem_tiled_blits.c b/tests/gem_tiled_blits.c
index bb43976..86ff745 100644
--- a/tests/gem_tiled_blits.c
+++ b/tests/gem_tiled_blits.c
@@ -127,8 +127,11 @@ int main(int argc, char **argv)
 	fd = drm_open_any();
 
 	count = 0;
+	if (drmtest_run_in_simulation())
+		count = 2;
 	if (argc > 1)
 		count = atoi(argv[1]);
+
 	if (count == 0) {
 		count = 3 * gem_aperture_size(fd) / (1024*1024) / 2;
 		count += (count & 1) == 0;
@@ -175,6 +178,9 @@ int main(int argc, char **argv)
 	for (i = 0; i < count; i++)
 		check_bo(bo[i], bo_start_val[i]);
 
+	if (drmtest_run_in_simulation())
+		return 0;
+
 	printf("Cyclic blits, backward...\n");
 	for (i = 0; i < count * 4; i++) {
 		int src = (i+1) % count;
-- 
1.7.11.7

[PATCH i-g-t 5/5] build: Make grep silent with running make test

[Damien Lespiau]

-q will prevent grep from writing to stdout and print "root" when make
test is running as root.

Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
---
 tests/Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 33db962..d32e944 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -166,7 +166,7 @@ TESTS = \
 	$(NULL)
 
 test:
-	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
+	@whoami | grep -q root || ( echo ERROR: not running as root; exit 1 )
 	@./check_drm_clients
 	@if [ "${IGT_SIMULATION}" == 1 ]; then \
 		make TESTS="${simulation_tests}" check; \
-- 
1.7.11.7

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Daniel Vetter]

On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> There are two ways to run tests, directly with make check/test
> or through piglit.
> 
> When IGT_SIMULATION is set to '1', we substitute the list of tests in
> those two code paths with carefully selected tests. The stress tests and
> other horrors are left to torture the real hardware as they don't make
> too much sense in simulation.
> 
> Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>

This feels a bit fragile since even just now I often fail to put a test
newly converted to the subtest stuff into the right single/multi make
target list. Also, this way we exclude a test by default, which also feels
like the wrong way round (most of the tests we're adding aren't heavy
stress-tests, but more excercise some corner-case).

So what about a sprinkling

drmtest_skip_on_simulation();

over the remaining testcases instead, which just calls exit(77);? Only
caveat is that for subtests we need to put it into the right spot to not
break subtest enumeration in piglit (it's a bit fragile).

Cheers, Daniel

> ---
>  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 40 insertions(+), 3 deletions(-)
> 
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index 2fddfe8..33db962 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -137,22 +137,59 @@ kernel_tests = \
>  	$(multi_kernel_tests) \
>  	$(NULL)
>  
> +# We maintain here a list of tests to run in our simulation environments
> +single_simulation_tests = \
> +	gem_mmap \
> +	gem_pread_after_blit \
> +	gem_ring_sync_loop \
> +	gem_ctx_basic \
> +	gem_pipe_control_store_loop \
> +	gem_storedw_loop_render \
> +	gem_storedw_loop_blt \
> +	gem_storedw_loop_bsd \
> +	gem_render_linear_blits \
> +	gem_tiled_blits \
> +	gem_cpu_reloc \
> +	$(NULL)
> +
> +multi_simulation_tests = \
> +	gem_exec_nop \
> +	gem_mmap_gtt \
> +	$(NULL)
> +
> +simulation_tests = \
> +	$(single_simulation_tests) \
> +	$(multi_simulation_tests) \
> +	$(NULL)
> +
>  TESTS = \
>  	$(NULL)
>  
>  test:
>  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
>  	@./check_drm_clients
> -	@make TESTS="${kernel_tests}" check
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		make TESTS="${simulation_tests}" check; \
> +	else \
> +		make TESTS="${kernel_tests}" check; \
> +	fi
>  
>  list-single-tests:
>  	@echo TESTLIST
> -	@echo ${single_kernel_tests}
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		echo ${single_simulation_tests}; \
> +	else \
> +		echo ${single_kernel_tests}; \
> +	fi
>  	@echo END TESTLIST
>  
>  list-multi-tests:
>  	@echo TESTLIST
> -	@echo ${multi_kernel_tests}
> +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> +		echo ${multi_simulation_tests}; \
> +	else \
> +		echo ${multi_kernel_tests}; \
> +	fi
>  	@echo END TESTLIST
>  
>  HANG = \
> -- 
> 1.7.11.7
> 
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Ben Widawsky]

On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > There are two ways to run tests, directly with make check/test
> > or through piglit.
> > 
> > When IGT_SIMULATION is set to '1', we substitute the list of tests in
> > those two code paths with carefully selected tests. The stress tests and
> > other horrors are left to torture the real hardware as they don't make
> > too much sense in simulation.
> > 
> > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> 
> This feels a bit fragile since even just now I often fail to put a test
> newly converted to the subtest stuff into the right single/multi make
> target list. Also, this way we exclude a test by default, which also feels
> like the wrong way round (most of the tests we're adding aren't heavy
> stress-tests, but more excercise some corner-case).
> 
> So what about a sprinkling
> 
> drmtest_skip_on_simulation();
> 
> over the remaining testcases instead, which just calls exit(77);? Only
> caveat is that for subtests we need to put it into the right spot to not
> break subtest enumeration in piglit (it's a bit fragile).
> 
> Cheers, Daniel

I prefer Damien's explicit definitions in a file. It makes it much
easier to see exactly what's run, and add a test as needed. Since
simulation testing will always be a massive subset of the whole i-g-t
suite, and we'll probably only rarely add or remove a test, I think it's
not a big concern that we might miss a test.

> 
> > ---
> >  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
> >  1 file changed, 40 insertions(+), 3 deletions(-)
> > 
> > diff --git a/tests/Makefile.am b/tests/Makefile.am
> > index 2fddfe8..33db962 100644
> > --- a/tests/Makefile.am
> > +++ b/tests/Makefile.am
> > @@ -137,22 +137,59 @@ kernel_tests = \
> >  	$(multi_kernel_tests) \
> >  	$(NULL)
> >  
> > +# We maintain here a list of tests to run in our simulation environments
> > +single_simulation_tests = \
> > +	gem_mmap \
> > +	gem_pread_after_blit \
> > +	gem_ring_sync_loop \
> > +	gem_ctx_basic \
> > +	gem_pipe_control_store_loop \
> > +	gem_storedw_loop_render \
> > +	gem_storedw_loop_blt \
> > +	gem_storedw_loop_bsd \
> > +	gem_render_linear_blits \
> > +	gem_tiled_blits \
> > +	gem_cpu_reloc \
> > +	$(NULL)
> > +
> > +multi_simulation_tests = \
> > +	gem_exec_nop \
> > +	gem_mmap_gtt \
> > +	$(NULL)
> > +
> > +simulation_tests = \
> > +	$(single_simulation_tests) \
> > +	$(multi_simulation_tests) \
> > +	$(NULL)
> > +
> >  TESTS = \
> >  	$(NULL)
> >  
> >  test:
> >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> >  	@./check_drm_clients
> > -	@make TESTS="${kernel_tests}" check
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		make TESTS="${simulation_tests}" check; \
> > +	else \
> > +		make TESTS="${kernel_tests}" check; \
> > +	fi
> >  
> >  list-single-tests:
> >  	@echo TESTLIST
> > -	@echo ${single_kernel_tests}
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		echo ${single_simulation_tests}; \
> > +	else \
> > +		echo ${single_kernel_tests}; \
> > +	fi
> >  	@echo END TESTLIST
> >  
> >  list-multi-tests:
> >  	@echo TESTLIST
> > -	@echo ${multi_kernel_tests}
> > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > +		echo ${multi_simulation_tests}; \
> > +	else \
> > +		echo ${multi_kernel_tests}; \
> > +	fi
> >  	@echo END TESTLIST
> >  
> >  HANG = \
> > -- 
> > 1.7.11.7
> > 
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Daniel Vetter]

On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > There are two ways to run tests, directly with make check/test
> > > or through piglit.
> > > 
> > > When IGT_SIMULATION is set to '1', we substitute the list of tests in
> > > those two code paths with carefully selected tests. The stress tests and
> > > other horrors are left to torture the real hardware as they don't make
> > > too much sense in simulation.
> > > 
> > > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> > 
> > This feels a bit fragile since even just now I often fail to put a test
> > newly converted to the subtest stuff into the right single/multi make
> > target list. Also, this way we exclude a test by default, which also feels
> > like the wrong way round (most of the tests we're adding aren't heavy
> > stress-tests, but more excercise some corner-case).
> > 
> > So what about a sprinkling
> > 
> > drmtest_skip_on_simulation();
> > 
> > over the remaining testcases instead, which just calls exit(77);? Only
> > caveat is that for subtests we need to put it into the right spot to not
> > break subtest enumeration in piglit (it's a bit fragile).
> > 
> > Cheers, Daniel
> 
> I prefer Damien's explicit definitions in a file. It makes it much
> easier to see exactly what's run, and add a test as needed. Since
> simulation testing will always be a massive subset of the whole i-g-t
> suite, and we'll probably only rarely add or remove a test, I think it's
> not a big concern that we might miss a test.

Imo that's the wrong approach, since most of the tests we've recently
added exercise corner-cases of our code, and in a rather deterministic
way. If we currently have too many tests to get through all of the useful
ones in a day, we need more machines, not fewer tests.

In the end we want to be able to run full piglit on all this stuff in
simulation after all. Cc'ing Yi so he knows where I'm aiming at ;-)
-Daniel

> 
> > 
> > > ---
> > >  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/tests/Makefile.am b/tests/Makefile.am
> > > index 2fddfe8..33db962 100644
> > > --- a/tests/Makefile.am
> > > +++ b/tests/Makefile.am
> > > @@ -137,22 +137,59 @@ kernel_tests = \
> > >  	$(multi_kernel_tests) \
> > >  	$(NULL)
> > >  
> > > +# We maintain here a list of tests to run in our simulation environments
> > > +single_simulation_tests = \
> > > +	gem_mmap \
> > > +	gem_pread_after_blit \
> > > +	gem_ring_sync_loop \
> > > +	gem_ctx_basic \
> > > +	gem_pipe_control_store_loop \
> > > +	gem_storedw_loop_render \
> > > +	gem_storedw_loop_blt \
> > > +	gem_storedw_loop_bsd \
> > > +	gem_render_linear_blits \
> > > +	gem_tiled_blits \
> > > +	gem_cpu_reloc \
> > > +	$(NULL)
> > > +
> > > +multi_simulation_tests = \
> > > +	gem_exec_nop \
> > > +	gem_mmap_gtt \
> > > +	$(NULL)
> > > +
> > > +simulation_tests = \
> > > +	$(single_simulation_tests) \
> > > +	$(multi_simulation_tests) \
> > > +	$(NULL)
> > > +
> > >  TESTS = \
> > >  	$(NULL)
> > >  
> > >  test:
> > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > >  	@./check_drm_clients
> > > -	@make TESTS="${kernel_tests}" check
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		make TESTS="${simulation_tests}" check; \
> > > +	else \
> > > +		make TESTS="${kernel_tests}" check; \
> > > +	fi
> > >  
> > >  list-single-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${single_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${single_simulation_tests}; \
> > > +	else \
> > > +		echo ${single_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  list-multi-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${multi_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${multi_simulation_tests}; \
> > > +	else \
> > > +		echo ${multi_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  HANG = \
> > > -- 
> > > 1.7.11.7
> > > 
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > Intel-gfx@lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > 
> > -- 
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx@lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> -- 
> Ben Widawsky, Intel Open Source Technology Center

-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Sun, Yi]

Hi Daniel, 

Thank you for your notification. This would make it easy to run sub suite on simulation during this stage. I know the goal is to run the whole i-g-t test case. 
So we can run tests/igt.test just like any other real machines. Ok, I'll add it to our nightly ASAP.

Thanks
    --Sun, Yi


> -----Original Message-----
> From: Daniel Vetter [mailto:daniel.vetter@ffwll.ch] On Behalf Of Daniel Vetter
> Sent: Wednesday, March 27, 2013 8:25 AM
> To: Ben Widawsky
> Cc: Daniel Vetter; Lespiau, Damien; intel-gfx@lists.freedesktop.org; Sun, Yi
> Subject: Re: [Intel-gfx] [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune
> the list of tests to run
> 
> On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > There are two ways to run tests, directly with make check/test or
> > > > through piglit.
> > > >
> > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > in those two code paths with carefully selected tests. The stress
> > > > tests and other horrors are left to torture the real hardware as
> > > > they don't make too much sense in simulation.
> > > >
> > > > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> > >
> > > This feels a bit fragile since even just now I often fail to put a
> > > test newly converted to the subtest stuff into the right
> > > single/multi make target list. Also, this way we exclude a test by
> > > default, which also feels like the wrong way round (most of the
> > > tests we're adding aren't heavy stress-tests, but more excercise some
> corner-case).
> > >
> > > So what about a sprinkling
> > >
> > > drmtest_skip_on_simulation();
> > >
> > > over the remaining testcases instead, which just calls exit(77);?
> > > Only caveat is that for subtests we need to put it into the right
> > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > >
> > > Cheers, Daniel
> >
> > I prefer Damien's explicit definitions in a file. It makes it much
> > easier to see exactly what's run, and add a test as needed. Since
> > simulation testing will always be a massive subset of the whole i-g-t
> > suite, and we'll probably only rarely add or remove a test, I think
> > it's not a big concern that we might miss a test.
> 
> Imo that's the wrong approach, since most of the tests we've recently added
> exercise corner-cases of our code, and in a rather deterministic way. If we
> currently have too many tests to get through all of the useful ones in a day, we
> need more machines, not fewer tests.
> 
> In the end we want to be able to run full piglit on all this stuff in simulation after
> all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> 
> >
> > >
> > > > ---
> > > >  tests/Makefile.am | 43
> > > > ++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/tests/Makefile.am b/tests/Makefile.am index
> > > > 2fddfe8..33db962 100644
> > > > --- a/tests/Makefile.am
> > > > +++ b/tests/Makefile.am
> > > > @@ -137,22 +137,59 @@ kernel_tests = \
> > > >  	$(multi_kernel_tests) \
> > > >  	$(NULL)
> > > >
> > > > +# We maintain here a list of tests to run in our simulation
> > > > +environments single_simulation_tests = \
> > > > +	gem_mmap \
> > > > +	gem_pread_after_blit \
> > > > +	gem_ring_sync_loop \
> > > > +	gem_ctx_basic \
> > > > +	gem_pipe_control_store_loop \
> > > > +	gem_storedw_loop_render \
> > > > +	gem_storedw_loop_blt \
> > > > +	gem_storedw_loop_bsd \
> > > > +	gem_render_linear_blits \
> > > > +	gem_tiled_blits \
> > > > +	gem_cpu_reloc \
> > > > +	$(NULL)
> > > > +
> > > > +multi_simulation_tests = \
> > > > +	gem_exec_nop \
> > > > +	gem_mmap_gtt \
> > > > +	$(NULL)
> > > > +
> > > > +simulation_tests = \
> > > > +	$(single_simulation_tests) \
> > > > +	$(multi_simulation_tests) \
> > > > +	$(NULL)
> > > > +
> > > >  TESTS = \
> > > >  	$(NULL)
> > > >
> > > >  test:
> > > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > > >  	@./check_drm_clients
> > > > -	@make TESTS="${kernel_tests}" check
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		make TESTS="${simulation_tests}" check; \
> > > > +	else \
> > > > +		make TESTS="${kernel_tests}" check; \
> > > > +	fi
> > > >
> > > >  list-single-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${single_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${single_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${single_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  list-multi-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${multi_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${multi_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${multi_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  HANG = \
> > > > --
> > > > 1.7.11.7
> > > >
> > > > _______________________________________________
> > > > Intel-gfx mailing list
> > > > Intel-gfx@lists.freedesktop.org
> > > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > >
> > > --
> > > Daniel Vetter
> > > Software Engineer, Intel Corporation
> > > +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > Intel-gfx@lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> >
> > --
> > Ben Widawsky, Intel Open Source Technology Center
> 
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Sun, Yi]

> On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > There are two ways to run tests, directly with make check/test or
> > > > through piglit.
> > > >
> > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > in those two code paths with carefully selected tests. The stress
> > > > tests and other horrors are left to torture the real hardware as
> > > > they don't make too much sense in simulation.
> > > >
> > > > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> > >
> > > This feels a bit fragile since even just now I often fail to put a
> > > test newly converted to the subtest stuff into the right
> > > single/multi make target list. Also, this way we exclude a test by
> > > default, which also feels like the wrong way round (most of the
> > > tests we're adding aren't heavy stress-tests, but more excercise some
> corner-case).
> > >
> > > So what about a sprinkling
> > >
> > > drmtest_skip_on_simulation();
> > >
> > > over the remaining testcases instead, which just calls exit(77);?
> > > Only caveat is that for subtests we need to put it into the right
> > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > >
> > > Cheers, Daniel
> >
> > I prefer Damien's explicit definitions in a file. It makes it much
> > easier to see exactly what's run, and add a test as needed. Since
> > simulation testing will always be a massive subset of the whole i-g-t
> > suite, and we'll probably only rarely add or remove a test, I think
> > it's not a big concern that we might miss a test.
> 
> Imo that's the wrong approach, since most of the tests we've recently added
> exercise corner-cases of our code, and in a rather deterministic way. If we
> currently have too many tests to get through all of the useful ones in a day, we
> need more machines, not fewer tests.
> 
> In the end we want to be able to run full piglit on all this stuff in simulation after
> all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> 
[Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
I noticed Damien's patches aren't on the branch yet.
If we want to split all i-g-t test cases to different HAS, how can we split it?


Thanks
	--Yi Sun

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Ben Widawsky]

On Mon, Apr 01, 2013 at 07:21:59AM +0000, Sun, Yi wrote:
> > On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > > There are two ways to run tests, directly with make check/test or
> > > > > through piglit.
> > > > >
> > > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > > in those two code paths with carefully selected tests. The stress
> > > > > tests and other horrors are left to torture the real hardware as
> > > > > they don't make too much sense in simulation.
> > > > >
> > > > > Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
> > > >
> > > > This feels a bit fragile since even just now I often fail to put a
> > > > test newly converted to the subtest stuff into the right
> > > > single/multi make target list. Also, this way we exclude a test by
> > > > default, which also feels like the wrong way round (most of the
> > > > tests we're adding aren't heavy stress-tests, but more excercise some
> > corner-case).
> > > >
> > > > So what about a sprinkling
> > > >
> > > > drmtest_skip_on_simulation();
> > > >
> > > > over the remaining testcases instead, which just calls exit(77);?
> > > > Only caveat is that for subtests we need to put it into the right
> > > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > > >
> > > > Cheers, Daniel
> > >
> > > I prefer Damien's explicit definitions in a file. It makes it much
> > > easier to see exactly what's run, and add a test as needed. Since
> > > simulation testing will always be a massive subset of the whole i-g-t
> > > suite, and we'll probably only rarely add or remove a test, I think
> > > it's not a big concern that we might miss a test.
> > 
> > Imo that's the wrong approach, since most of the tests we've recently added
> > exercise corner-cases of our code, and in a rather deterministic way. If we
> > currently have too many tests to get through all of the useful ones in a day, we
> > need more machines, not fewer tests.
> > 
> > In the end we want to be able to run full piglit on all this stuff in simulation after
> > all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> > 
> [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> I noticed Damien's patches aren't on the branch yet.
> If we want to split all i-g-t test cases to different HAS, how can we split it?
> 

Daniel, can you please advise Sun Yi since the solution I like has been
shot down?

-- 
Ben Widawsky, Intel Open Source Technology Center

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Damien Lespiau]

On Tue, Apr 02, 2013 at 10:06:37AM -0700, Ben Widawsky wrote:
> > [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> > I noticed Damien's patches aren't on the branch yet.
> > If we want to split all i-g-t test cases to different HAS, how can we split it?
> > 
> 
> Daniel, can you please advise Sun Yi since the solution I like has been
> shot down?

It's not unreasonable to want to run test cases by default and black
list the ones we don't want to run. I'm on holidays soon though and
won't be able to rework this before a little while.

-- 
Damien

Re: [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Daniel Vetter]

On Tue, Apr 02, 2013 at 06:16:59PM +0100, Damien Lespiau wrote:
> On Tue, Apr 02, 2013 at 10:06:37AM -0700, Ben Widawsky wrote:
> > > [Sun, Yi] So what's the conclusion, how can I do to solve the too long time issue?
> > > I noticed Damien's patches aren't on the branch yet.
> > > If we want to split all i-g-t test cases to different HAS, how can we split it?
> > > 
> > 
> > Daniel, can you please advise Sun Yi since the solution I like has been
> > shot down?
> 
> It's not unreasonable to want to run test cases by default and black
> list the ones we don't want to run. I'm on holidays soon though and
> won't be able to rework this before a little while.

tbh I'm not set in stone on this issue, and blocking enabling on fulsim
feels bad, too. So please go ahead with whatever approach you deem best.
Since for both approaches the interface for QA/testing would be just a
magic enviroment variable, we could even switch without much fuzz. So I
don't mind if you go ahead with the current patches.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch