sparse-llvm test cases

sparse-llvm test cases

[Dibyendu Majumdar]

Hi

As a measure of the progress made in the past few days (thanks Luc!) -
I have a bunch of test cases that now pass. More tests will be added.

Note that there are fixes in my project related to handling of fpcast,
setval, and local variables that are not yet reflected in sparse code
so these tests will not pass yet in sparse.

The tests that pass are here:

https://github.com/dibyendumajumdar/dmr_c/tree/master/tests

These are not exactly in the format currently used for testing sparse.
It will be nice however to add all the test cases to the official
sparse repository.

Thanks and Regards
Dibyendu

Re: sparse-llvm test cases

[Christopher Li]

On Sun, Mar 12, 2017 at 9:13 PM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:
> Hi
>
> As a measure of the progress made in the past few days (thanks Luc!) -
> I have a bunch of test cases that now pass. More tests will be added.
>
> Note that there are fixes in my project related to handling of fpcast,
> setval, and local variables that are not yet reflected in sparse code
> so these tests will not pass yet in sparse.
>
> The tests that pass are here:
>
> https://github.com/dibyendumajumdar/dmr_c/tree/master/tests

Again, sorry I am not able to response to it earlier. I am in the catching
up backlog mode now.

For those test case. I think it is very welcome to integrate into sparse.

If you want, you can break them down to smaller, digestible piece
and start submitting to sparse as patch series for review, or git pull
request. It will not get pull into this release v0.5.1 for sure. We can
start the discussion now. I can even have a post release branch doing
the merges for the patches after the release.

> These are not exactly in the format currently used for testing sparse.
> It will be nice however to add all the test cases to the official
> sparse repository.

That is fine. If it is all just test cases, that is relative safe.
Luc has a hue pile of pending patches I want to merge. If you touch
some of those core components, which result in Luc need to fix up
his repository a lot, then I would say hold a little bit for Luc's patches
being merged. The test case should be more or less fine.

Thank you

Chris

Re: sparse-llvm test cases

[Christopher Li]

On Sun, Mar 12, 2017 at 9:13 PM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:

>
> https://github.com/dibyendumajumdar/dmr_c/tree/master/tests
>
I take a brief look. I think it has way to many duplicate copy of the
testrun.sh. You might want to think coherently what exactly is need
for your different test case. It is better to have one frame work to
support all the test case.

Another question is that, I don't know how many your test get impact
by the llvm versions. If it get tight to a specific version of llvm, the
test case has less value. People will get false positives due to different
llvm version.

Chris

Re: sparse-llvm test cases

[Dibyendu Majumdar]

Hi Chris,

On 10 August 2017 at 19:38, Christopher Li <sparse@chrisli.org> wrote:
> On Sun, Mar 12, 2017 at 9:13 PM, Dibyendu Majumdar
> <mobile@majumdar.org.uk> wrote:
>> https://github.com/dibyendumajumdar/dmr_c/tree/master/tests
>>
> I take a brief look. I think it has way to many duplicate copy of the
> testrun.sh. You might want to think coherently what exactly is need
> for your different test case. It is better to have one frame work to
> support all the test case.

Yes, the current arrangement is a stop gap. As I develop both on
Windows and Linux, I need a portable solution so I cannot really use
shell scripts.

>
> Another question is that, I don't know how many your test get impact
> by the llvm versions. If it get tight to a specific version of llvm, the
> test case has less value. People will get false positives due to different
> llvm version.
>

The tests are all designed to be runnable - i.e. you can generate the
code and run it either with lli or by generating an executable - so
they are completely independent of LLVM version. But they do need all
the LLVM fixes that are still pending as without those fixes most
tests won't even compile. Nevertheless we can start incorporating
these tests gradually so that when the LLVM fixes are all in then we
will have a good set of tests.

I would prefer to add validation of the Sparse IR as well - at present
I record these but do not try to validate mainly as I think Sparse IR
output is still changing so we need to get to a stable point first.

Regards
Dibyendu

Re: sparse-llvm test cases

[Christopher Li]

On Thu, Aug 10, 2017 at 4:44 PM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:
> Yes, the current arrangement is a stop gap. As I develop both on
> Windows and Linux, I need a portable solution so I cannot really use
> shell scripts.

Can you use python scripts? Just asking.

> The tests are all designed to be runnable - i.e. you can generate the
> code and run it either with lli or by generating an executable - so
> they are completely independent of LLVM version. But they do need all

That is good to know. Thanks for the clarification.

> the LLVM fixes that are still pending as without those fixes most
> tests won't even compile. Nevertheless we can start incorporating
> these tests gradually so that when the LLVM fixes are all in then we
> will have a good set of tests.
>
> I would prefer to add validation of the Sparse IR as well - at present
> I record these but do not try to validate mainly as I think Sparse IR
> output is still changing so we need to get to a stable point first.

I think we should design the test frame work to take into account
for the IR changes. e.g. Ignore the IR that does not recognize.

That being said, I do wish the IR has less changes.

I am not sure how many IR changes in Luc's pending patches.
The OP_PUSH one I assume we can avoid, as I comment on
the original discussion thread.

Chris

Re: sparse-llvm test cases

[Luc Van Oostenryck]

On Thu, Aug 10, 2017 at 8:01 PM, Christopher Li <sparse@chrisli.org> wrote:
> On Sun, Mar 12, 2017 at 9:13 PM, Dibyendu Majumdar wrote:
>
> For those test case. I think it is very welcome to integrate into sparse.

Sure. Very welcome.

> If you want, you can break them down to smaller, digestible piece
> and start submitting to sparse as patch series for review, or git pull
> request. It will not get pull into this release v0.5.1 for sure. We can
> start the discussion now. I can even have a post release branch doing
> the merges for the patches after the release.
>
>> These are not exactly in the format currently used for testing sparse.
>> It will be nice however to add all the test cases to the official
>> sparse repository.
>
> That is fine. If it is all just test cases, that is relative safe.
> Luc has a hue pile of pending patches I want to merge. If you touch
> some of those core components, which result in Luc need to fix up
> his repository a lot, then I would say hold a little bit for Luc's patches
> being merged. The test case should be more or less fine.

Yes, in no ways should additional test cases be a problem.

-- Luc

Re: sparse-llvm test cases

[Luc Van Oostenryck]

On Thu, Aug 10, 2017 at 10:44 PM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:
>
> Yes, the current arrangement is a stop gap. As I develop both on
> Windows and Linux, I need a portable solution so I cannot really use
> shell scripts.

People use bash with mingwin, for example. Would it be something
usable for you?

>> Another question is that, I don't know how many your test get impact
>> by the llvm versions. If it get tight to a specific version of llvm, the
>> test case has less value. People will get false positives due to different
>> llvm version.

If it begin to be the case we'll need to add a new tag telling to ignore
the test if version this or that.

> The tests are all designed to be runnable

I think this is very very important.
Of course, you're interested in LLVM output but not only.

One thing I want to have for testing linearized code is a simulator
for sparse's IR. It's even something quite easy (or at least straightforward)
to do. At least once you can associate to each phi-node's 'argument' the
parent BB they come from. This is something I'm planning to do for several
reasons.

-- Luc

Re: sparse-llvm test cases

[Christopher Li]

On Thu, Aug 10, 2017 at 6:50 PM, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:

> One thing I want to have for testing linearized code is a simulator
> for sparse's IR. It's even something quite easy (or at least straightforward)
> to do. At least once you can associate to each phi-node's 'argument' the
> parent BB they come from. This is something I'm planning to do for several
> reasons.

BTW, I have some python script that parse the show_entry() output.
It is not complete but it is easy enough to extend.  I use the script
to capture some part of the IR for prototyping compiler related algorithms.
e.g. simple fast way to find dominators. Python has very nice pretty printer
and very easy to write verification test. e.g. the dominator I found really
is a dominator.  For complex algorithms, I like to verify it works correctly in
scripts before coding in C.

Chris

Re: sparse-llvm test cases

[Dibyendu Majumdar]

On 10 August 2017 at 23:50, Luc Van Oostenryck
<luc.vanoostenryck@gmail.com> wrote:
> On Thu, Aug 10, 2017 at 10:44 PM, Dibyendu Majumdar
> <mobile@majumdar.org.uk> wrote:
>>
>> Yes, the current arrangement is a stop gap. As I develop both on
>> Windows and Linux, I need a portable solution so I cannot really use
>> shell scripts.
>
> People use bash with mingwin, for example. Would it be something
> usable for you?
>

I have a native MSVC build - I'd rather not have to rely upon mingw /
cygwin. Windows 10 now has a Linux subsystem - I am planning to do my
Linux builds within that. But I would rather have a test harness that
runs natively on all platforms.

>>> Another question is that, I don't know how many your test get impact
>>> by the llvm versions. If it get tight to a specific version of llvm, the
>>> test case has less value. People will get false positives due to different
>>> llvm version.
>
> If it begin to be the case we'll need to add a new tag telling to ignore
> the test if version this or that.
>
>> The tests are all designed to be runnable
>
> I think this is very very important.
> Of course, you're interested in LLVM output but not only.
>

Being able to run it is an easy way to verify that the test works -
because the success / failure condition is part of the test itself /
or output produced by the test. Ideally though I would like to be able
to validate the Sparse IR - but don't know an easy way to do this
other than comparing the output from linearizer.

> One thing I want to have for testing linearized code is a simulator
> for sparse's IR. It's even something quite easy (or at least straightforward)
> to do. At least once you can associate to each phi-node's 'argument' the
> parent BB they come from. This is something I'm planning to do for several
> reasons.
>

Do you mean you will have a VM that runs Sparse IR? That would be nice
but it is a lot of effort I would think.

Regards
Dibyendu

Re: sparse-llvm test cases

[Dibyendu Majumdar]

On 10 August 2017 at 21:59, Christopher Li <sparse@chrisli.org> wrote:
> On Thu, Aug 10, 2017 at 4:44 PM, Dibyendu Majumdar
> <mobile@majumdar.org.uk> wrote:
>> Yes, the current arrangement is a stop gap. As I develop both on
>> Windows and Linux, I need a portable solution so I cannot really use
>> shell scripts.
>
> Can you use python scripts? Just asking.
>

Yes, Python is an option. I guess now-a-days it is available on all platforms.

Regards
Dibyendu

Re: sparse-llvm test cases

[Christopher Li]

On Sat, Aug 12, 2017 at 9:46 AM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:
>
> Yes, Python is an option. I guess now-a-days it is available on all platforms.
>

If python works for you, my current simple python script can be extend
to support more instruction understanding. I did not use it for testing
purpose. I use it to import the IR into python so I can run some experiment
algorithms in python. The function requirement to import the IR is very
similar here so we can reuse some code.

Chris

Re: sparse-llvm test cases

[Luc Van Oostenryck]

On Sat, Aug 12, 2017 at 3:44 PM, Dibyendu Majumdar
<mobile@majumdar.org.uk> wrote:
>
> I have a native MSVC build - I'd rather not have to rely upon mingw /
> cygwin.

I can understand that.

> Windows 10 now has a Linux subsystem - I am planning to do my
> Linux builds within that. But I would rather have a test harness that
> runs natively on all platforms.

I can understand that also but:
- if you want to validate the IR, it's something that is platform independent
  and if validated on your linux platform it will be valid for the others too
- the script used to run the test is something very simple, it shouldn't
  take much work to write a version in some other language.
  Making this new version *the* official version is something else
  and rewritting all the testcases so that this or that is easier for this
  new version would be a totally different question, of course.

>
> Being able to run it is an easy way to verify that the test works -
> because the success / failure condition is part of the test itself /
> or output produced by the test. Ideally though I would like to be able
> to validate the Sparse IR - but don't know an easy way to do this
> other than comparing the output from linearizer.

Slowly, it's what I'm already doing in the current testcases.
It's far from ideal but we can already progress.

>> One thing I want to have for testing linearized code is a simulator
>> for sparse's IR. It's even something quite easy (or at least straightforward)
>> to do. At least once you can associate to each phi-node's 'argument' the
>> parent BB they come from. This is something I'm planning to do for several
>> reasons.
>
> Do you mean you will have a VM that runs Sparse IR? That would be nice
> but it is a lot of effort I would think.

Yes. I call that a simulator but it's all the same.
It would be a program that take some input file
(either directly the C source or the IR in some format yet to defined)
and will 'execute' the IR instruction.
It's in fact something very easy to do, something like a weekend project,
*once* we associate parent BB to phi-node's 'arguments'.

This BB for phi-node's arg is something very needed and it's the top
of my todo list but for the moment, there is more important things to
fix.

-- Luc