Debug from failing hashequiv builds - server side problem?

Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 3544 bytes --]

Hi Joshua,

I did a little digging on the weird hash equiv failure. I've attached
two logs, one is the simplified console output, the other is the reams
of debug.

What is really odd is this is totally reproducible. If I clean out tmp
and cache from the build, then "bitbake nspr-native", it breaks, every
time and strange as it sounds, I'm not sure it should break
reproducibly.

What is bothering me is it never reuses m4-native from the cache.

At the start of the build we see:

DEBUG: Found unihash 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f in place of 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f for /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/m4/m4-native_1.4.18.bb:do_populate_sysroot from typhoon.yocto.io:8686

i.e. it looked it up and found the hash didn't change. Fine.

Later the task runs:

DEBUG: m4-native-1.4.18-r0 do_populate_sysroot: Task 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f unihash changed 6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f -> a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd by server typhoon.yocto.io:8686

so it was changed to a new hash by the server.

So why when I clean tmp/cache and re-run the build do I not get this
new hash back?

If I don't clean cache, the hash is cached and is used as remapped.

In case its interesting, the "short" console output is:

Sstate summary: Wanted 17 Found 15 Missed 2 Current 0 (88% match, 0% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Task /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/m4/m4-native_1.4.18.bb:do_populate_sysroot unihash changed to a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd
NOTE: Task /home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_populate_sysroot unihash changed to 82d9ac234a96a673b706be0b7ee7651cbca379c41dae0bd8eb8ce2f52da5f67d
NOTE: Reported task virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot as unihash 89b717131d37f2cf8313f0cadc8c0c7ebe9a2229c55e7effcbefe7f0a8013f33 to typhoon.yocto.io:8686 ({'unihash': '3418b83888e28fa9660977f995414241c9c47a1b963ceca8d2a281483be52df9', 'taskhash': 'f0cca9fbe2764fffc26f60e41de01d01d0e60abd447167eb0ef61fb3cef40391', 'method': 'oe.sstatesig.OEOuthashBasic'})
NOTE: Task virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot unihash 3418b83888e28fa9660977f995414241c9c47a1b963ceca8d2a281483be52df9 unchanged by server
NOTE: Already covered setscene for virtual:native:/home/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-support/nspr/nspr_4.24.bb:do_populate_sysroot so ignoring rehash (remap)
NOTE: Setscene tasks completed
ERROR: nspr-native-4.24-r0 do_compile: oe_runmake failed

so there is a second error somewhere in the runqueue code where the
failed remap causes other problems. Not sure why the remap fails but
this first bug is making the second one reproducible...

Thought I'd send in case anything obvious jumps out to you. I've cc'd
the list just so people know where we are with this.

I plan not to disturb the autobuilder too much until we get to the
bottom of these issues whilst we can reproduce them. This will delay
other patches but I think this is important.

Cheers,

Richard



[-- Attachment #2: failing-log.txt.gz --]
[-- Type: application/gzip, Size: 2611 bytes --]

[-- Attachment #3: failing-log-full.txt.gz --]
[-- Type: application/gzip, Size: 14652 bytes --]

Re: Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

On Sun, 2019-12-22 at 10:54 +0000, Richard Purdie wrote:
> I did a little digging on the weird hash equiv failure. I've attached
> two logs, one is the simplified console output, the other is the
> reams of debug.

I dumped out what I think is the relevant bits of the database (which
is nearly 3GB):

richard@jet:~$ cat hashserv.sql | grep 6360e7a9b1945b15f92b70
INSERT INTO tasks_v2 VALUES(3679599,'oe.sstatesig.OEOuthashBasic','ee6b445c02a957ef4b081ee36452bc815402a3a465c5aed0ab3060c3f992c094','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-16 23:58:38.708832',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679606,'oe.sstatesig.OEOuthashBasic','7af7e78f7ede4ea21cd1555f3eb041ecfd747e2d48989abbe20414d0b63e1b65','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-16 23:58:45.951324',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679610,'oe.sstatesig.OEOuthashBasic','17cf76dfd359dc5da2e5931c3372e1f7ccd14f6f7e500c546921a21bb320b7ad','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','9513c21c3f34240d7be0b917b6f0e362ed3be6e6a12e7f4060eb9e3efff47514','2019-12-16 23:58:52.295374',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679629,'oe.sstatesig.OEOuthashBasic','5f3d29da028a75b0a5a060413051d1aa5f0781e1c7860819b34fa237f51e5285','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','b545e7707b4e1847d4927debbceec81ec37622786ce432864e0d3f7a7592ccb8','2019-12-16 23:59:10.861937',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679643,'oe.sstatesig.OEOuthashBasic','5ae9fb99844190c6aadce38246c5ebeb2dc9303d62fcbba5530f37693bb7abc9','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-16 23:59:17.633598',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679645,'oe.sstatesig.OEOuthashBasic','0e1547560936fc0d52a461475c78e12339ff4a75d259411d885d84ccc8020c9e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-16 23:59:24.770771',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679685,'oe.sstatesig.OEOuthashBasic','2877ddc716e5127a0844a906e1e3fefa70b417ea65a5eb533a290539061e4aca','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','9513c21c3f34240d7be0b917b6f0e362ed3be6e6a12e7f4060eb9e3efff47514','2019-12-16 23:59:40.543937',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679799,'oe.sstatesig.OEOuthashBasic','dcfd5b71bf57328e6b90e0824c9b56e72f7a632846c8324e71596dae27e633e6','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:00.767871',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679929,'oe.sstatesig.OEOuthashBasic','1a3647981781233d7a0cd471af5a0adf0934d45e9f7b1182e769c7ae730b9f4b','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:32.584666',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679936,'oe.sstatesig.OEOuthashBasic','412a3cb834d871619dfd23e4577a76322946d889f6262fb1fa5d0c6d5ac31746','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:33.313992',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679958,'oe.sstatesig.OEOuthashBasic','566a75f3055ffe73e4610d97c7629038dd87d958718eaab6196d526d7467533e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:36.950698',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679962,'oe.sstatesig.OEOuthashBasic','fc3136068b3412fd3e4161802a969e3707e927e23173f6e79e3b35f04fb79187','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:37.809560',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679992,'oe.sstatesig.OEOuthashBasic','bde0aaef71c8f251e5ea0c8a1fafcd1cd210c2d427e1835ace677375e4a9346a','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','22a01e6199e47cd0cbbeaf33b68e50f345cdbd83d6a82e14ad1078a62f748e6f','2019-12-17 00:00:43.332100',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679995,'oe.sstatesig.OEOuthashBasic','062037539403510e031936d357929fba38172e89094d355903ca92e334c53926','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:45.211257',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3679996,'oe.sstatesig.OEOuthashBasic','6c04f0f5188614066d97eda86f95b7228f122269aeae25f4fe255bac0cd19b7f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:00:45.297526',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680120,'oe.sstatesig.OEOuthashBasic','d3bc5729937ef6672a1a33ff6713cc8ca5351daf76657dab81405fb711db33dd','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:01:04.179909',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680214,'oe.sstatesig.OEOuthashBasic','891d634846c7f4ae8939e58354380ae458986913044dc5d030ffc9133cc6d75e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:01:24.159781',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680294,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:01:40.804708',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680507,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:02:13.159355',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3682134,'oe.sstatesig.OEOuthashBasic','633d385868d47920c10214d04923e463ae35ac873d836a1f7053000450aae5be','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:06:48.447473',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3682511,'oe.sstatesig.OEOuthashBasic','8832279370b159e91e689009497bfcc7ca5d5217a2285eefeaf5c23c0ddc073e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:07:52.884087',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3682751,'oe.sstatesig.OEOuthashBasic','4fe0e7a0326583c3cf3ed0213ac4cb79ec2f64d15f2a687b5c173ccde81d5c79','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:08:33.534192',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3684205,'oe.sstatesig.OEOuthashBasic','298d9d1d3fd77c545fca07823875cd282b59e914f0a2bd6535629ef723bbe993','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 00:11:19.303968',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3750353,'oe.sstatesig.OEOuthashBasic','0be39bdc7472b2d694fb75283a5174a1b893e5ae1dc39d2cf4aef7412407b1f9','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 01:48:46.907907',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3763068,'oe.sstatesig.OEOuthashBasic','a497a6307a1abaf2d9e7d9c492ffa0660c8148bbecf83f02165f5cc4b293105d','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:07:52.993142',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3765047,'oe.sstatesig.OEOuthashBasic','85f03d784fef9a114d2c88a00670ab7fbda75b6976e8a89a4c59ef937c860939','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:10:58.749872',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3779162,'oe.sstatesig.OEOuthashBasic','869b797eb3eddc9e86527a84946909ccbf36cb0de32cbee843f4791327b6988f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:34:49.808855',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3780230,'oe.sstatesig.OEOuthashBasic','2c2a2ee035ceb8ed907726202f45307fefe07620833fac62634b3bfca80e33ff','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:37:29.967968',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3782881,'oe.sstatesig.OEOuthashBasic','181ea60cee6ef3c4b71b676714325a7abfac1fa975ac8bf9c07c483f00c565e7','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:43:51.347124',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3788180,'oe.sstatesig.OEOuthashBasic','45043704f4c11624c648b23a4909140f9a1d252452b45ffe2e513f63a660b0d2','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 02:54:47.544792',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3805971,'oe.sstatesig.OEOuthashBasic','fbbf9b323763eb81f127ea55fef84b43fc7f47a1a8a5bfa3d02044ddd38db62f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 04:10:20.424618',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3838163,'oe.sstatesig.OEOuthashBasic','6c1f211bfc1780e98e6a7f0572f9bcd66adde0deb272c578d60005ba309f6ef8','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','9513c21c3f34240d7be0b917b6f0e362ed3be6e6a12e7f4060eb9e3efff47514','2019-12-17 13:12:08.206866',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3844297,'oe.sstatesig.OEOuthashBasic','2b918035854599a0f47e4dfa55e6798691c73b99757c7f0490ac64ee69b4ea8f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 13:22:25.248199',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3868111,'oe.sstatesig.OEOuthashBasic','9aea5e17a46c6de81d3a36de9aaa6beeb9d6138dd7f9c92b9478299c45743452','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 14:00:56.443563',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3895278,'oe.sstatesig.OEOuthashBasic','be5105f7dbd196c3a6295e9c3c7ccd5f637f33e8d62505698cc89074d499146b','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 14:52:52.903773',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3918178,'oe.sstatesig.OEOuthashBasic','a6186f282b70b4564586e209231ba9b309ac29715ebbc36e9f98f1507b7417f5','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 17:41:58.741718',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3925112,'oe.sstatesig.OEOuthashBasic','f6d3dac1ae0213bca6f3440fb99e406fb09ca35679ac580111e5a52f7d6e8cbb','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 18:06:55.314078',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3931033,'oe.sstatesig.OEOuthashBasic','248e97e82fc5a8b5b1adf62496e2dd1de1d3e2ccd8c5c71addc0146f60df1d15','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-17 18:41:04.184439',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3941305,'oe.sstatesig.OEOuthashBasic','2f0881d07595b57e17aa7065578bf3dca6ccfc1104de6630b88fcfd839cebb5a','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 00:22:14.468517',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3941372,'oe.sstatesig.OEOuthashBasic','658504f9e9966a761eddcc3327032c2b6a7f3f1fe8f47d1115c8d4664fe9142c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 00:22:29.715743',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3943289,'oe.sstatesig.OEOuthashBasic','c5950f7fd282d6a70577fa125cdc10a813bd4d0160c8cb41aac74635c4aa5140','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 00:27:36.357745',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3943949,'oe.sstatesig.OEOuthashBasic','bb207c72625f4231026c4161b2c75ece5f8544f6548b0e736163a8578a289f9d','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 00:29:05.398551',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4014492,'oe.sstatesig.OEOuthashBasic','48545a8cfab5ad3f02305bfc04d12bd2645ece5b1daeb4cdf3b29a2f7f72bc97','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 02:04:58.749572',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4018111,'oe.sstatesig.OEOuthashBasic','b2c314188632fc20235d15aa6f0362ed4cd988e40974efcbe766eaed53e72ec6','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 02:09:20.985905',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4041803,'oe.sstatesig.OEOuthashBasic','2a9c8158806fcf3cc58bc01c411712a836b44a1047d0d975a85809e7f3b41fb0','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 02:46:57.770385',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4042423,'oe.sstatesig.OEOuthashBasic','41f730982243743a791f9f25e23f68dc1ef91467e64cf0d63c6c9be1484b930d','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 02:48:01.347831',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4049991,'oe.sstatesig.OEOuthashBasic','62cb65f512ca265c20c5104152f3c5425ae62ca05d8a524fe0dfcaa060f97dbe','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 03:03:11.832150',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4055228,'oe.sstatesig.OEOuthashBasic','341c0eb16fc9fc013880759bfbaf565f82fedbdcbbfdfe31b4fb4015b57f75a9','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 03:14:05.719029',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4062553,'oe.sstatesig.OEOuthashBasic','34a9076372ef724d31f3a2d2d0af63aa1582a51baa0d8ace20cf4d2b933095fa','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 03:28:59.496594',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4063273,'oe.sstatesig.OEOuthashBasic','e67ff862264c141fcbbe2538a8cebce640d6b6af4a6a50ef049670ad324e4945','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 03:30:57.520446',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4075611,'oe.sstatesig.OEOuthashBasic','96cfd4715fd1bbbe04056f1530faf29e43d8bac615d2f7470ae75e00645cea84','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 04:02:28.775460',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4082105,'oe.sstatesig.OEOuthashBasic','2302a57dfba80985aa009a39e5cfddbf90cedfc6cbf5ffc728051ed171df1851','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 04:32:16.616630',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4086200,'oe.sstatesig.OEOuthashBasic','d4d1412ed95b9725781e44c49eca249526432ea4e8465251a805818a5b8f801d','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 04:53:51.401902',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4110399,'oe.sstatesig.OEOuthashBasic','990c8852a549cf3ee4fb7ecacb42b6e9e90635ac09b3cab5f554ddc46e2d09ea','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 20:39:36.321312',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4129663,'oe.sstatesig.OEOuthashBasic','982c735cbdf31f2ee969b7520f998b4c2ab521d5c0c7df4f89eada38ee464195','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 21:03:45.428903',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4138506,'oe.sstatesig.OEOuthashBasic','d901b0d39413750538a35f1db5bc1f21ef36eede1ab9bae9e90e1427ba0edb8c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-18 22:00:56.725631',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4142637,'oe.sstatesig.OEOuthashBasic','940a6fe7f9a1d5ba1139cb55f138c57198eeacac0a5d686c3b4bac5ab8c93355','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-19 01:15:01.761885',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4147259,'oe.sstatesig.OEOuthashBasic','18cea44585b8b5bb5139fc4c16c08a838df0f00fb0fc58107d0ace5726bb12ec','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-19 01:26:00.364241',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4151794,'oe.sstatesig.OEOuthashBasic','57abbf15add8ae25f5a1e2d85e08c97184f2622d9e749f3df2f8a8341347031e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-19 02:02:10.422492',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4171259,'oe.sstatesig.OEOuthashBasic','0e1547560936fc0d52a461475c78e12339ff4a75d259411d885d84ccc8020c9e','e0303d04031d7d2d8acd26c2d825799ebf1abe765a919c1262491f1029db5f71','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 00:47:44.156568',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4171309,'oe.sstatesig.OEOuthashBasic','dcfd5b71bf57328e6b90e0824c9b56e72f7a632846c8324e71596dae27e633e6','e0303d04031d7d2d8acd26c2d825799ebf1abe765a919c1262491f1029db5f71','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 00:47:55.567449',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4171456,'oe.sstatesig.OEOuthashBasic','fc3136068b3412fd3e4161802a969e3707e927e23173f6e79e3b35f04fb79187','e0303d04031d7d2d8acd26c2d825799ebf1abe765a919c1262491f1029db5f71','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 00:48:35.823535',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4219766,'oe.sstatesig.OEOuthashBasic','d215ded2c7748fbdab81eb025435a38d542055dd2b280c66526848502499eb4c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 02:01:53.112197',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4258953,'oe.sstatesig.OEOuthashBasic','d7cdacf7a7ce0709f5ea3081820835d77321af92d5a63d79191d71086e4b9db6','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 02:28:38.198437',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4290405,'oe.sstatesig.OEOuthashBasic','6bde91264f173c121def0e539b2eca67d7e40f5a1e86d6097eccd7478bd0253b','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 03:24:10.561579',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4411437,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-20 23:17:19.193566',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4411607,'oe.sstatesig.OEOuthashBasic','e6b31f8ac8b062334460ecb772f5d050f25de6d6bd9c21f6a6db0da1fd35bdb6','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:17:48.575550',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4416443,'oe.sstatesig.OEOuthashBasic','236de7650556cea0af38c2e4e128dfe2cc00630c501df8d281e43e120d4fb595','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:21:18.722664',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4417930,'oe.sstatesig.OEOuthashBasic','b3301a1d5837f7fcf2c5673bd86875705b16e3c5345a224eab17522beec842b8','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:22:18.423365',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4426490,'oe.sstatesig.OEOuthashBasic','7035d8ee2ea362d1d6fe842c35234b5f91b69d710fb6d4ced58632acf4509fda','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:38:29.299874',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4430160,'oe.sstatesig.OEOuthashBasic','6496d68df27a97c04a4f7fbc411abd2142a9349ac91d6a795eab53ea8e63b763','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:49:19.533295',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4430984,'oe.sstatesig.OEOuthashBasic','74ca5104558d1376c077da5cd83c9c3b328aca01e87fc83c5fb3894bca0310a3','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-20 23:51:16.990362',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4437398,'oe.sstatesig.OEOuthashBasic','dd2c8e3f9b3781f0c48056066944dc5a728141d8b5b249db980ebc001fdc45e4','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-21 01:17:22.647222',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4439996,'oe.sstatesig.OEOuthashBasic','798c11d4336410728ac90e2262263689ac95e84f5c740b47eaa1d8b440cfbb51','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-21 01:31:33.326965',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4442520,'oe.sstatesig.OEOuthashBasic','e98174a57574de38b3b85fb98312482adae6ca9e0fae1bae0140b4658b6d318e','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-21 02:22:29.710650',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4445028,'oe.sstatesig.OEOuthashBasic','a22ea62831fc9fec6abc950736e8079f77d4d89d45c7f85dc8a25ea3cda20639','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-22 01:15:37.924462',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4445360,'oe.sstatesig.OEOuthashBasic','f0c1df4541135bdd38340ed346701200f085020045edd8f28216f3e9e7edef10','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-22 01:29:54.268096',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4446351,'oe.sstatesig.OEOuthashBasic','5af04a10a842930ae4f361a3722822592ef5e829ca1b9645347a6bbb252dd9d6','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','2019-12-22 02:21:54.071366',NULL,NULL,NULL,NULL,NULL,NULL);

richard@jet:~$ cat hashserv.sql | grep a884c5002fa7a02111fea

INSERT INTO tasks_v2 VALUES(14,'oe.sstatesig.OEOuthashBasic','40e2bc1f1cc5621b04964946ba78466fafdae7b859cc0411f2d22dad056a3628','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:55:52.911681',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(15,'oe.sstatesig.OEOuthashBasic','41087d86a74b28c71d311c34d2ad8d49f4ab4343f460258c27567cef9f0a4d05','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:55:53.018967',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(53,'oe.sstatesig.OEOuthashBasic','5c5480468a5a915edb6b6e77a4caf3948c17531746cb22706c98d519a4b8b3bc','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:56:27.064619',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(607,'oe.sstatesig.OEOuthashBasic','403694d850fefc870b5cf57becfd4322a927c8a59d5333c832665b34633616dc','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:58:17.529782',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1286,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:59:28.726109',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1533,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 17:59:48.560342',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1875,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-25 18:00:25.830091',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(186466,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','ac9c4dd01c24301e48a2403a3e211dad0275d1339d669249d9991ab7a10aad1e','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-11-27 15:22:12.008478',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1768750,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','0f3cd0305cbba7926b0fc03a35cef1a4c0e79b7bae8ff3c6bdd2a4b31ebe5748','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-10 13:53:15.367311',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1771736,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','0f3cd0305cbba7926b0fc03a35cef1a4c0e79b7bae8ff3c6bdd2a4b31ebe5748','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-10 14:03:25.356854',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1791378,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','b545e7707b4e1847d4927debbceec81ec37622786ce432864e0d3f7a7592ccb8','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-10 15:11:15.418746',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1907114,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','5ef4ca18854dc6350c4a846fddce0c69a5cfd32a043fbb2825cda656b63e0b35','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-10 23:57:36.697793',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1910083,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','5ef4ca18854dc6350c4a846fddce0c69a5cfd32a043fbb2825cda656b63e0b35','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 00:09:42.939024',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(1913130,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','5ef4ca18854dc6350c4a846fddce0c69a5cfd32a043fbb2825cda656b63e0b35','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 00:23:02.958010',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2122900,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','9513c21c3f34240d7be0b917b6f0e362ed3be6e6a12e7f4060eb9e3efff47514','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 09:00:53.371050',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2203360,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','c04e33d8c6a3c7ac328044653d52f005df780c8019611b26459267ccc248b3de','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 11:02:49.537039',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2214463,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','b08f15ea940a52ba7b036cb261558086e46449891a353bbe908b57363e25df49','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 11:46:47.992145',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2221042,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','b08f15ea940a52ba7b036cb261558086e46449891a353bbe908b57363e25df49','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 12:13:04.152462',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2257876,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','b08f15ea940a52ba7b036cb261558086e46449891a353bbe908b57363e25df49','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-11 13:24:12.078067',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2971806,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','22a01e6199e47cd0cbbeaf33b68e50f345cdbd83d6a82e14ad1078a62f748e6f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-14 13:09:54.657729',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2972022,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','22a01e6199e47cd0cbbeaf33b68e50f345cdbd83d6a82e14ad1078a62f748e6f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-14 13:10:30.939892',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(2972242,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','22a01e6199e47cd0cbbeaf33b68e50f345cdbd83d6a82e14ad1078a62f748e6f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-14 13:11:04.919446',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680294,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:01:40.804708',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680507,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:02:13.159355',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4171942,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','e0303d04031d7d2d8acd26c2d825799ebf1abe765a919c1262491f1029db5f71','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-20 00:50:01.714955',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4411437,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-20 23:17:19.193566',NULL,NULL,NULL,NULL,NULL,NULL);

richard@jet:~$ cat hashserv.sql | grep a884c5002fa7a02111fea | grep 6360e7a9b1945b15f92b

INSERT INTO tasks_v2 VALUES(3680294,'oe.sstatesig.OEOuthashBasic','b98e66b13a9edce8ff22faf86fc01df638d73875aaa71b782c97532e8d32c25c','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:01:40.804708',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(3680507,'oe.sstatesig.OEOuthashBasic','5a25f213e1842f9ff63065541e4aad5715272799c1bd02cf9d3dbba010e073a1','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-17 00:02:13.159355',NULL,NULL,NULL,NULL,NULL,NULL);
INSERT INTO tasks_v2 VALUES(4411437,'oe.sstatesig.OEOuthashBasic','abda1f9d7e27db35bb7a2e36a7d90739ea90bd32ed6ee4fb4f9da4cfbabe03ad','6360e7a9b1945b15f92b703b54e4dddba8cf830a22ebce29895b9ce99a11895f','a884c5002fa7a02111fea0f12b37886e2519c9b3c57748c22914588ee4a9adbd','2019-12-20 23:17:19.193566',NULL,NULL,NULL,NULL,NULL,NULL);

Cheers,

Richard

Re: Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

On Sun, 2019-12-22 at 12:08 +0000, Richard Purdie wrote:
> On Sun, 2019-12-22 at 10:54 +0000, Richard Purdie wrote:
> > I did a little digging on the weird hash equiv failure. I've
> > attached two logs, one is the simplified console output, the other
> > is the reams of debug.

I did a bit more thinking about this. These hashes represent m4-native.

We do have a problem that a given input hash for m4-native can
represent two different output states depending on the architecture it
was built on.

For the same input hash, the output hash of these two things will never
match. Nor will there ever be present sstate for them to trigger the
report equiv mechanism.

At query time in a clean build, the hashserver cannot know which of the
two output hashes it needs to return the value for.

I can think of two possible options:

a) When we report after running a task, we check if that input hash
already has a value and then reuse it for the output hash mapping.

b) We start adding some kind of suffix to the reported hashes for
native output which is used within the hash equiv server but not
sstate.

I freely admit this needs much more thought...

Cheers,

Richard

Re: Debug from failing hashequiv builds - server side problem?

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 2254 bytes --]

On Sun, Dec 22, 2019, 6:49 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sun, 2019-12-22 at 12:08 +0000, Richard Purdie wrote:
> > On Sun, 2019-12-22 at 10:54 +0000, Richard Purdie wrote:
> > > I did a little digging on the weird hash equiv failure. I've
> > > attached two logs, one is the simplified console output, the other
> > > is the reams of debug.
>
> I did a bit more thinking about this. These hashes represent m4-native.
>
> We do have a problem that a given input hash for m4-native can
> represent two different output states depending on the architecture it
> was built on.
>
> For the same input hash, the output hash of these two things will never
> match. Nor will there ever be present sstate for them to trigger the
> report equiv mechanism.
>
> At query time in a clean build, the hashserver cannot know which of the
> two output hashes it needs to return the value for.
>

In the case of multiple taskhashes mapping to different output hashes, the
server is supposed to simply return the oldest unihash. If it's not doing
that, there might be a bug.


> I can think of two possible options:
>
> a) When we report after running a task, we check if that input hash
> already has a value and then reuse it for the output hash mapping.
>

I don't quite follow this one, can you be a little more precise with what
hashes you are referring to (e.g. taskhash, unihash, outhash)?




> b) We start adding some kind of suffix to the reported hashes for
> native output which is used within the hash equiv server but not
> sstate.
>

Just for clarification, this is because "native" can either be x86_64 or
aarch64 but the actual arch (HOST_ARCH ?) isn't part of the taskhash
calculation?

This sounds related to the gcc-cross issues we had with the eSDK.

Is there a reason that the host arch isn't part of the taskhash?

I think it might be possible to report additional inputs in the hashes
reported to the server. The server doesn't really care if it's the exact
taskhash, as long as the client is consistent. Perhaps that would help with
the gcc-cross issue as well?



> I freely admit this needs much more thought...
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3609 bytes --]

Re: Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

On Sun, 2019-12-22 at 10:00 -0600, Joshua Watt wrote:
> On Sun, Dec 22, 2019, 6:49 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > On Sun, 2019-12-22 at 12:08 +0000, Richard Purdie wrote:
> > 
> > At query time in a clean build, the hashserver cannot know which of
> > the two output hashes it needs to return the value for.
> 
> In the case of multiple taskhashes mapping to different output
> hashes, the server is supposed to simply return the oldest unihash.
> If it's not doing that, there might be a bug.


The problem here is a single taskhash mapping to two different outputs.

m4-native:do_populate_sysroot (on aarch64)
m4-native:do_populate_sysroot (on x86-64)

When we query based on the input hash we'll get the first built.

We then rebuild as its not in sstate and can match a previous task
output, we then get a new hash.

If we start a new build with no cache, we lookup the input hash and get
the first built task of the wrong arch again.

> > I can think of two possible options:
> > 
> > a) When we report after running a task, we check if that input hash
> > already has a value and then reuse it for the output hash mapping.
> 
> I don't quite follow this one, can you be a little more precise with
> what hashes you are referring to (e.g. taskhash, unihash, outhash)?

We lookup a taskhash, get unihash A but its not in sstate (wrong arch).
We build this thing, send the outhash, currently we get unihash B.

I'm saying we map that outhash to unihash A since the server already
has an entry for it.

> > b) We start adding some kind of suffix to the reported hashes for
> > native output which is used within the hash equiv server but not
> > sstate.

I was thinking about this further and I had a slightly evil idea. What
if we set the method to XXX:<native arch>"? (where XXX is the current
value).

This would namespace the two native arches separately and I think then
avoid the problems?

> Just for clarification, this is because "native" can either be x86_64
> or aarch64 but the actual arch (HOST_ARCH ?) isn't part of the
> taskhash calculation?

Correct.

> This sounds related to the gcc-cross issues we had with the eSDK.

The previous problem was on the boundary between cross/native and
target. This is a pure native (or cross) issue.

> Is there a reason that the host arch isn't part of the taskhash?

Yes. Should the target output depend upon which arch it was built on?
(answer, no it shouldn't so the native hashes have to match).

> I think it might be possible to report additional inputs in the
> hashes reported to the server. The server doesn't really care if it's
> the exact taskhash, as long as the client is consistent. Perhaps that
> would help with the gcc-cross issue as well?

It won't help the other native/cross to target boundary mapping issue.

See above, I'm wondering if we could abuse the method field to make
this work? Certainly we could test that...

Cheers,

Richard

Re: Debug from failing hashequiv builds - server side problem?

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]

On Sun, Dec 22, 2019, 10:09 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sun, 2019-12-22 at 10:00 -0600, Joshua Watt wrote:
> > On Sun, Dec 22, 2019, 6:49 AM Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > > On Sun, 2019-12-22 at 12:08 +0000, Richard Purdie wrote:
> > >
> > > At query time in a clean build, the hashserver cannot know which of
> > > the two output hashes it needs to return the value for.
> >
> > In the case of multiple taskhashes mapping to different output
> > hashes, the server is supposed to simply return the oldest unihash.
> > If it's not doing that, there might be a bug.
>
>
> The problem here is a single taskhash mapping to two different outputs.
>
> m4-native:do_populate_sysroot (on aarch64)
> m4-native:do_populate_sysroot (on x86-64)
>
> When we query based on the input hash we'll get the first built.
>
> We then rebuild as its not in sstate and can match a previous task
> output, we then get a new hash.
>
> If we start a new build with no cache, we lookup the input hash and get
> the first built task of the wrong arch again.
>
> > > I can think of two possible options:
> > >
> > > a) When we report after running a task, we check if that input hash
> > > already has a value and then reuse it for the output hash mapping.
> >
> > I don't quite follow this one, can you be a little more precise with
> > what hashes you are referring to (e.g. taskhash, unihash, outhash)?
>
> We lookup a taskhash, get unihash A but its not in sstate (wrong arch).
> We build this thing, send the outhash, currently we get unihash B.
>
> I'm saying we map that outhash to unihash A since the server already
> has an entry for it.
>
> > > b) We start adding some kind of suffix to the reported hashes for
> > > native output which is used within the hash equiv server but not
> > > sstate.
>
> I was thinking about this further and I had a slightly evil idea. What
> if we set the method to XXX:<native arch>"? (where XXX is the current
> value).
>
> This would namespace the two native arches separately and I think then
> avoid the problems?
>
> > Just for clarification, this is because "native" can either be x86_64
> > or aarch64 but the actual arch (HOST_ARCH ?) isn't part of the
> > taskhash calculation?
>
> Correct.
>
> > This sounds related to the gcc-cross issues we had with the eSDK.
>
> The previous problem was on the boundary between cross/native and
> target. This is a pure native (or cross) issue.
>
> > Is there a reason that the host arch isn't part of the taskhash?
>
> Yes. Should the target output depend upon which arch it was built on?
> (answer, no it shouldn't so the native hashes have to match).
>
> > I think it might be possible to report additional inputs in the
> > hashes reported to the server. The server doesn't really care if it's
> > the exact taskhash, as long as the client is consistent. Perhaps that
> > would help with the gcc-cross issue as well?
>
> It won't help the other native/cross to target boundary mapping issue.
>
> See above, I'm wondering if we could abuse the method field to make
> this work? Certainly we could test that...
>

That seems reasonable for a quick test. The method is supposed to be the
output hash calculation method, so my first instinct would be that it is
better in the long run to add another field e.g. "class" or something.


> Cheers,
>
> Richard
>
>
>

[-- Attachment #2: Type: text/html, Size: 4497 bytes --]

Re: Debug from failing hashequiv builds - server side problem?

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 3807 bytes --]

On Sun, Dec 22, 2019, 10:17 AM Joshua Watt <jpewhacker@gmail.com> wrote:

>
>
> On Sun, Dec 22, 2019, 10:09 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Sun, 2019-12-22 at 10:00 -0600, Joshua Watt wrote:
>> > On Sun, Dec 22, 2019, 6:49 AM Richard Purdie <
>> > richard.purdie@linuxfoundation.org> wrote:
>> > > On Sun, 2019-12-22 at 12:08 +0000, Richard Purdie wrote:
>> > >
>> > > At query time in a clean build, the hashserver cannot know which of
>> > > the two output hashes it needs to return the value for.
>> >
>> > In the case of multiple taskhashes mapping to different output
>> > hashes, the server is supposed to simply return the oldest unihash.
>> > If it's not doing that, there might be a bug.
>>
>>
>> The problem here is a single taskhash mapping to two different outputs.
>>
>> m4-native:do_populate_sysroot (on aarch64)
>> m4-native:do_populate_sysroot (on x86-64)
>>
>> When we query based on the input hash we'll get the first built.
>>
>> We then rebuild as its not in sstate and can match a previous task
>> output, we then get a new hash.
>>
>> If we start a new build with no cache, we lookup the input hash and get
>> the first built task of the wrong arch again.
>>
>> > > I can think of two possible options:
>> > >
>> > > a) When we report after running a task, we check if that input hash
>> > > already has a value and then reuse it for the output hash mapping.
>> >
>> > I don't quite follow this one, can you be a little more precise with
>> > what hashes you are referring to (e.g. taskhash, unihash, outhash)?
>>
>> We lookup a taskhash, get unihash A but its not in sstate (wrong arch).
>> We build this thing, send the outhash, currently we get unihash B.
>>
>> I'm saying we map that outhash to unihash A since the server already
>> has an entry for it.
>>
>> > > b) We start adding some kind of suffix to the reported hashes for
>> > > native output which is used within the hash equiv server but not
>> > > sstate.
>>
>> I was thinking about this further and I had a slightly evil idea. What
>> if we set the method to XXX:<native arch>"? (where XXX is the current
>> value).
>>
>> This would namespace the two native arches separately and I think then
>> avoid the problems?
>>
>> > Just for clarification, this is because "native" can either be x86_64
>> > or aarch64 but the actual arch (HOST_ARCH ?) isn't part of the
>> > taskhash calculation?
>>
>> Correct.
>>
>> > This sounds related to the gcc-cross issues we had with the eSDK.
>>
>> The previous problem was on the boundary between cross/native and
>> target. This is a pure native (or cross) issue.
>>
>> > Is there a reason that the host arch isn't part of the taskhash?
>>
>> Yes. Should the target output depend upon which arch it was built on?
>> (answer, no it shouldn't so the native hashes have to match).
>>
>> > I think it might be possible to report additional inputs in the
>> > hashes reported to the server. The server doesn't really care if it's
>> > the exact taskhash, as long as the client is consistent. Perhaps that
>> > would help with the gcc-cross issue as well?
>>
>> It won't help the other native/cross to target boundary mapping issue.
>>
>> See above, I'm wondering if we could abuse the method field to make
>> this work? Certainly we could test that...
>>
>
> That seems reasonable for a quick test. The method is supposed to be the
> output hash calculation method, so my first instinct would be that it is
> better in the long run to add another field e.g. "class" or something.
>

Perhaps based on ${NATIVELSBSTRING} like sstate.bbclass path? That has nice
congruency with sstate file paths, which I like


>
>> Cheers,
>>
>> Richard
>>
>>
>>

[-- Attachment #2: Type: text/html, Size: 5281 bytes --]

Re: Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

On Sun, 2019-12-22 at 10:22 -0600, Joshua Watt wrote:
> On Sun, Dec 22, 2019, 10:17 AM Joshua Watt <jpewhacker@gmail.com>
> wrote:
> > 
> > On Sun, Dec 22, 2019, 10:09 AM Richard Purdie <
> > richard.purdie@linuxfoundation.org> wrote:
> > > On Sun, 2019-12-22 at 10:00 -0600, Joshua Watt wrote:
> > > It won't help the other native/cross to target boundary mapping
> > > issue.
> > > 
> > > See above, I'm wondering if we could abuse the method field to
> > > make
> > > this work? Certainly we could test that...
> > 
> > That seems reasonable for a quick test. The method is supposed to
> > be the output hash calculation method, so my first instinct would
> > be that it is better in the long run to add another field e.g.
> > "class" or something.
> 
> Perhaps based on ${NATIVELSBSTRING} like sstate.bbclass path? That
> has nice congruency with sstate file paths, which I like

If only that were the right thing.

We need:

    if bb.data.inherits_class('native', d):
        d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
    elif bb.data.inherits_class('cross', d):
        d.setVar('SSTATE_PKGARCH', d.expand("${BUILD_ARCH}_${TARGET_ARCH}"))

i.e. SSTATE_PKGARCH but only for native/cross.

Cheers,

Richard

Re: Debug from failing hashequiv builds - server side problem?

[Joshua Watt]

On Sun, Dec 22, 2019 at 3:25 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2019-12-22 at 10:22 -0600, Joshua Watt wrote:
> > On Sun, Dec 22, 2019, 10:17 AM Joshua Watt <jpewhacker@gmail.com>
> > wrote:
> > >
> > > On Sun, Dec 22, 2019, 10:09 AM Richard Purdie <
> > > richard.purdie@linuxfoundation.org> wrote:
> > > > On Sun, 2019-12-22 at 10:00 -0600, Joshua Watt wrote:
> > > > It won't help the other native/cross to target boundary mapping
> > > > issue.
> > > >
> > > > See above, I'm wondering if we could abuse the method field to
> > > > make
> > > > this work? Certainly we could test that...
> > >
> > > That seems reasonable for a quick test. The method is supposed to
> > > be the output hash calculation method, so my first instinct would
> > > be that it is better in the long run to add another field e.g.
> > > "class" or something.
> >
> > Perhaps based on ${NATIVELSBSTRING} like sstate.bbclass path? That
> > has nice congruency with sstate file paths, which I like
>
> If only that were the right thing.
>
> We need:
>
>     if bb.data.inherits_class('native', d):
>         d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH', False))
>     elif bb.data.inherits_class('cross', d):
>         d.setVar('SSTATE_PKGARCH', d.expand("${BUILD_ARCH}_${TARGET_ARCH}"))
>
> i.e. SSTATE_PKGARCH but only for native/cross.

Ok, so I've had a little bit of time to think about this.

From a more theoretical standpoint, I think that hash equivalence only
works if all of the possible variables that can impact the output hash
are present in the taskhash. Otherwise, you can end up in cases
described above where the same taskhash results in diverging output
hashes that will never converge back together (in this case because
the build host arch differs). For most target recipes this should
already be true because we certainly want to rebuild the recipe if any
of the inputs that could possibly affect the output change. One of the
great parts about hash equivalence is that it sort dynamically and
automatically figures out if these changes don't actually change
anything and accounts for it so we don't have to manually figure out
which variables for which recipes could be whitelisted (not that we
would embark on that madness!).

However, for some recipes (e.g. native and cross), the taskhash is
incomplete in this case because we don't actually care about the
specific binary output from these recipes; we actually care about the
behaviour of these recipes when they are run (e.g. we don't care if
gcc-cross running on x86_64 or aarch is the exact same binary as long
as it produces the exact same cross compiled target binary in both
cases). Hash equivalence sort of falls down a little bit here because
there is no sane way to hash the "behaviour" of a native or cross
recipe. If there were we could set the output hash algorithm for
native and cross recipes to "OEOminiscentBehaviourHash" and all would
be well :) In lieu of that, the next best thing is to make sure that
the inputs we give to the hash equivalence server are the full set
that affect the output hash, just like we do for target builds. This
works because we do expect that on a given build host arch, the output
should be the same from build to build.

So, the real question is how do we make sure that the inputs to the
hash equivalence server are all the ones that affect the output hash?
I think there are a few options:
 1) Make up yet another hash (yahash) that is equivalent to the
taskhash + the hash of variables "unwhitelisted" for the purpose of
hash equivalence. Instead of dealing with taskhashes, the hash
equivalence deals with yahashes
 2) Just like #1, except that the yahash is only the unwhitelisted
variables and hash equivalence uses both the taskhash and the yahash
for lookups.
 3) Implement some sort of "namespace" field in the hash equivalence
tables that is used to filter out entries appropriately (basically,
the solution proposed by Richard).

I don't really have a strong opinion either way.... #3 seems much
easier, but the thing that worries me is if we are going to find other
variables that should be unwhitelisted and do #1 or #2 anyway.

As stated, I think that one of the advantages of hash equivalence is
that we can be a little more aggressive with what we "unwhitelist" and
let the hash equivalence process itself figure out if variable
actually makes a difference in the output.

>
> Cheers,
>
> Richard
>

Re: Debug from failing hashequiv builds - server side problem?

[Richard Purdie]

On Fri, 2019-12-27 at 13:55 -0600, Joshua Watt wrote:
> From a more theoretical standpoint, I think that hash equivalence
> only works if all of the possible variables that can impact the
> output hash are present in the taskhash. Otherwise, you can end up in
> cases described above where the same taskhash results in diverging
> output hashes that will never converge back together (in this case
> because the build host arch differs). For most target recipes this
> should already be true because we certainly want to rebuild the
> recipe if any of the inputs that could possibly affect the output
> change. One of the great parts about hash equivalence is that it sort
> dynamically and automatically figures out if these changes don't
> actually change anything and accounts for it so we don't have to
> manually figure out which variables for which recipes could be
> whitelisted (not that we would embark on that madness!).
> 
> However, for some recipes (e.g. native and cross), the taskhash is
> incomplete in this case because we don't actually care about the
> specific binary output from these recipes; we actually care about the
> behaviour of these recipes when they are run (e.g. we don't care if
> gcc-cross running on x86_64 or aarch is the exact same binary as long
> as it produces the exact same cross compiled target binary in both
> cases). Hash equivalence sort of falls down a little bit here because
> there is no sane way to hash the "behaviour" of a native or cross
> recipe. If there were we could set the output hash algorithm for
> native and cross recipes to "OEOminiscentBehaviourHash" and all would
> be well :) In lieu of that, the next best thing is to make sure that
> the inputs we give to the hash equivalence server are the full set
> that affect the output hash, just like we do for target builds. This
> works because we do expect that on a given build host arch, the
> output should be the same from build to build.
> 
> So, the real question is how do we make sure that the inputs to the
> hash equivalence server are all the ones that affect the output hash?
> I think there are a few options:
>  1) Make up yet another hash (yahash) that is equivalent to the
> taskhash + the hash of variables "unwhitelisted" for the purpose of
> hash equivalence. Instead of dealing with taskhashes, the hash
> equivalence deals with yahashes
>  2) Just like #1, except that the yahash is only the unwhitelisted
> variables and hash equivalence uses both the taskhash and the yahash
> for lookups.
>  3) Implement some sort of "namespace" field in the hash equivalence
> tables that is used to filter out entries appropriately (basically,
> the solution proposed by Richard).
> 
> I don't really have a strong opinion either way.... #3 seems much
> easier, but the thing that worries me is if we are going to find
> other variables that should be unwhitelisted and do #1 or #2 anyway.

We do have quite a number of variables we exclude from the taskhashes.
The most obvious is TMPDIR since we make relocatable binaries and don't
care where something was built. On the most part hashequiv is fairly
tolerant of this as worst case the outhashes don't match if a path
leaks.

I was further thinking about this and another issue is the version of
gcc on the host machine for building native binaries. We're naturally
going to end up with different outhashes for a native binaries built
with different versions of gcc (or other host tools). This is more
problematic as we're back to two outhashes which are treated
equivalently for the same input hash.

I'm not sure its realistic to try and even encode all the permutations
of host tools that may influence the binary output from this
perspective.

> As stated, I think that one of the advantages of hash equivalence is
> that we can be a little more aggressive with what we "unwhitelist"
> and let the hash equivalence process itself figure out if variable
> actually makes a difference in the output.

Having also spent a bit of time thinking about this, I think the
important piece is that we need to match the behaviour of sstate in
hashequiv. The theory is great but in practise they have to work
consistently and together. Where sstate allows two coexisting hashes,
we need to allow for that on the hashequiv side.

For that reason I don't think 1/2 are feasible at implementation and
we'll need to add some namespace field (as in 3) which allows us to
have the two systems work together.

The native arch issue is the big current one and when uninative is
active, I *think* its the only one we have. We may need to account for
the gcc 4.8/4.9/later sstate splitting now I think about it. When
uninative is not active, I think it would have to use the NATIVELSB
string that sstate uses to split the sstate feeds per distro.

I've hacked the method overloading into master-next as an experiment to
see how it works out, just with BUILD_ARCH for now.

I also tracked down the mystery runqueue "not executing tasks" problem
and put hard aborts into runqueue so it does not do that. The challenge
now is the conditionals around that abort aren't right so its aborting
some builds it shouldn't. I'll continue to work on figuring out the
right set.

Cheers,

Richard