[Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[tomba]

Hi.
I'm trying to run quemu-cvs on WinXP SP1 host and MinGW with gcc v3.4.2,
but qemu after start just opens console window and hangs.

After some testing I came to conclusion that this is the offending piece
of code (cpu_exec.c):

#if defined(USE_CODE_COPY)
(...)
#endif
                    spin_unlock(&tb_lock);
                }
                tc_ptr = tb->tc_ptr;
                env->current_tb = tb;
                /* execute the generated code */
                gen_func = (void *)tc_ptr;
		printf("xxx: cpu-exec.c::cpu_exec() AFTER gen_func = (void *)tc_ptr\n");
#if defined(__sparc__)
(...)
#eise
		printf("xxx: cpu-exec.c::cpu_exec() BEFORE gen_func()\n");
                gen_func();
		printf("xxx: cpu-exec.c::cpu_exec() AFTER gen_func()\n");
#endif


The result after executing is like:

xxx: cpu-exec.c::cpu_exec() AFTER spin_unlock(&tb_lock)
xxx: cpu-exec.c::cpu_exec() AFTER gen_func = (void *)tc_ptr
xxx: cpu-exec.c::cpu_exec() BEFORE gen_func()

and then qemu hangs. Debugging this in VisualC gives me:
"Unhandled exception in qemu.exe: 0xC0000005: Access Violation"

Another strange thing is that sometimes after compile qemu makes the whole
first loop:

xxx: vl.c::main_loop() AFTER for(;;)
xxx: vl.c::main_loop() BEFORE ret=cpu_exec()
(...)
xxx: cpu-exec.c::cpu_exec() AFTER spin_unlock(&tb_lock)
xxx: cpu-exec.c::cpu_exec() AFTER gen_func = (void *)tc_ptr
xxx: cpu-exec.c::cpu_exec() BEFORE gen_func()
xxx: cpu-exec.c::cpu_exec() AFTER gen_func()
(...)
xxx: vl.c::main_loop() AFTER ret=cpu_exec()
xxx: vl.c::main_loop() BEFORE main_loop_wait(timeout)
xxx: vl.c::main_loop() AFTER main_loop_wait(timeout)
xxx: vl.c::main_loop() AFTER for(;;)
xxx: vl.c::main_loop() BEFORE ret=cpu_exec()
(...)
xxx: cpu-exec.c::cpu_exec() AFTER spin_unlock(&tb_lock)
xxx: cpu-exec.c::cpu_exec() AFTER gen_func = (void *)tc_ptr
xxx: cpu-exec.c::cpu_exec() BEFORE gen_func()
->hangs


Anyone have an idea what is wrong here?

t.

-- 
  Tomasz Bątor  e-mail: tomba@bartek.tu.kielce.pl  ICQ: 101194886
 ------ ---- -- - -  -    -   -  -  -   -    -  - - -- ---- ------
          "Okay, let's see if I loaded this thing right."
                    -- last words in a RPG game

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

gen_func() is what QEmu does to call the dynamically translated code. As
this is the heart of QEmu and Works Here, I think that it's got to do
something with the compilation: either gcc 3.4.2, or some compiler flags.
Did you try with another gcc version?

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[tomba]

Witam,
Dnia Wed, Dec 15, 2004 at 03:18:03PM +0100 Johannes Schindelin napisal(a):

> this is the heart of QEmu and Works Here, I think that it's got to do
> something with the compilation: either gcc 3.4.2, or some compiler flags.
> Did you try with another gcc version?

Well, I guess that's it. I didn't want to downgrade gcc earlier, but I guess
I'll have no choice as it works with 3.2.3 and 3.3.1 and only hangs on 3.4.x.
Thanks for the hint.

t.

-- 
  Tomasz Bątor  e-mail: tomba@bartek.tu.kielce.pl  ICQ: 101194886
 ------ ---- -- - -  -    -   -  -  -   -    -  - - -- ---- ------
    "The only thing that stops God from sending another flood is
                  that the first one was useless."
                            -- Chamfort

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Wed, Dec 15, 2004 at 03:59:03PM +0100, tomba@100tka.net wrote:
> Witam,
> Dnia Wed, Dec 15, 2004 at 03:18:03PM +0100 Johannes Schindelin napisal(a):
> 
> > this is the heart of QEmu and Works Here, I think that it's got to do
> > something with the compilation: either gcc 3.4.2, or some compiler flags.
> > Did you try with another gcc version?
> 
> Well, I guess that's it. I didn't want to downgrade gcc earlier, but I guess
> I'll have no choice as it works with 3.2.3 and 3.3.1 and only hangs on 3.4.x.
> Thanks for the hint.
> 
> t.
> 

This is due to a change in the way gcc compiles code ... 3.4.0 may work,
but later versions will cause functions to return prematurely (due to the way
qemu handles translated blocks). I suggested that a fix for this would be to
use function pointer calls instead of directly chaining machine code, but
apparently this is too slow to make qemu usable.

> -- 
>   Tomasz B?tor  e-mail: tomba@bartek.tu.kielce.pl  ICQ: 101194886
>  ------ ---- -- - -  -    -   -  -  -   -    -  - - -- ---- ------
>     "The only thing that stops God from sending another flood is
>                   that the first one was useless."
>                             -- Chamfort
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Wed, 15 Dec 2004, Jim C. Brown wrote:

> This is due to a change in the way gcc compiles code ... 3.4.0 may work,
> but later versions will cause functions to return prematurely (due to the way
> qemu handles translated blocks). I suggested that a fix for this would be to
> use function pointer calls instead of directly chaining machine code, but
> apparently this is too slow to make qemu usable.

The Forth way ;-)

No, really. It does not make QEmu too slow to be usable, but it takes
very, very much from the performance improvements QEmu has in comparison
to bochs. Just try the interpreter patch, which is very good to have a
regression test, but otherwise is just slow as compared to FAST!

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Juergen Lock]

On Thu, Dec 16, 2004 at 01:37:53AM +0000, Johannes Schindelin wrote:
> Hi,
> 
> On Wed, 15 Dec 2004, Jim C. Brown wrote:
> 
> > This is due to a change in the way gcc compiles code ... 3.4.0 may work,
> > but later versions will cause functions to return prematurely (due to the way
> > qemu handles translated blocks). I suggested that a fix for this would be to
> > use function pointer calls instead of directly chaining machine code, but
> > apparently this is too slow to make qemu usable.
> 
> The Forth way ;-)
>...

How about instead post-processing the .s, replacing the rets with jumps
to the end of the function?  Of course this part would be architecture
dependant (different assembler syntax), but other parts of qemu
are that as well, right?

 Just a thought...
	Juergen

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Fri, 17 Dec 2004, Juergen Lock wrote:

> How about instead post-processing the .s, replacing the rets with jumps
> to the end of the function?  Of course this part would be architecture
> dependant (different assembler syntax), but other parts of qemu
> are that as well, right?

It would be architecture dependant, right. But it would be better to fix
the compilation of op.c, so that you really can chain the op_* functions.
What I mean: if you work on it, you could as well do it such that you gain
the most (performance) of it.

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Sat, Dec 18, 2004 at 01:09:26AM +0100, Johannes Schindelin wrote:
> Hi,
> 
> it would be better to fix
> the compilation of op.c, so that you really can chain the op_* functions.
> What I mean: if you work on it, you could as well do it such that you gain
> the most (performance) of it.
> 
> Ciao,
> Dscho
> 

>From what I understand, this is outright impossible with later versions of
gcc. In order to do what you want, you would have to write the code purely
in assembler.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Fri, 17 Dec 2004, Jim C. Brown wrote:

> On Sat, Dec 18, 2004 at 01:09:26AM +0100, Johannes Schindelin wrote:
> > Hi,
> >
> > it would be better to fix
> > the compilation of op.c, so that you really can chain the op_* functions.
> > What I mean: if you work on it, you could as well do it such that you gain
> > the most (performance) of it.
> >
> > Ciao,
> > Dscho
> >
>
> >From what I understand, this is outright impossible with later versions of
> gcc. In order to do what you want, you would have to write the code purely
> in assembler.

What do new gcc's do? Why is it impossible to chain the code? Do you have
an example object file, or even better, an assembly snippet?

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Sat, Dec 18, 2004 at 03:59:38PM +0100, Johannes Schindelin wrote:
> Hi,
> 
> On Fri, 17 Dec 2004, Jim C. Brown wrote:
> 
> > On Sat, Dec 18, 2004 at 01:09:26AM +0100, Johannes Schindelin wrote:
> > > Hi,
> > >
> > > it would be better to fix
> > > the compilation of op.c, so that you really can chain the op_* functions.
> > > What I mean: if you work on it, you could as well do it such that you gain
> > > the most (performance) of it.
> > >
> > > Ciao,
> > > Dscho
> > >
> >
> > >From what I understand, this is outright impossible with later versions of
> > gcc. In order to do what you want, you would have to write the code purely
> > in assembler.
> 
> What do new gcc's do? Why is it impossible to chain the code?

Originally, the op_* functions for qemu were written so that they would have the
machine code and then the rets at the very end of the function. qemu could
then just chain them together by stripping the rets.

gcc now rearranges the generated code around, and reserves the right to stick in
extra rets any place it wants, not just the very end, which makes them very hard
to strip off. I asked another developer about the difficulty of doing this, and
was told that it would be easier to write a C compiler just for op.c from
scratch.

> Do you have
> an example object file, or even better, an assembly snippet?

Afraid not.

> 
> Ciao,
> Dscho
> 

On Fri, Dec 17, 2004 at 07:56:27PM +0100, Juergen Lock wrote:
> On Thu, Dec 16, 2004 at 01:37:53AM +0000, Johannes Schindelin wrote:
> > Hi,
> > 
> > On Wed, 15 Dec 2004, Jim C. Brown wrote:
> > 
> > > This is due to a change in the way gcc compiles code ... 3.4.0 may work,
> > > but later versions will cause functions to return prematurely (due to the way
> > > qemu handles translated blocks). I suggested that a fix for this would be to
> > > use function pointer calls instead of directly chaining machine code, but
> > > apparently this is too slow to make qemu usable.
> > 
> > The Forth way ;-)
> >...
> 
> How about instead post-processing the .s, replacing the rets with jumps
> to the end of the function?  Of course this part would be architecture
> dependant (different assembler syntax), but other parts of qemu
> are that as well, right?

This may be possible. I don't know if the change applies to gcc only, in which
case this is the best solution, or if it applies to gas as well, in which case
this would solve nothing.

Well, even if gas was changed, we could probably get around that by using a
newer gcc with an older gas as the backend ...

> 
>  Just a thought...
> 	Juergen
> 
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[John R. Hogerhuis]

On Sat, 2004-12-18 at 06:59, Johannes Schindelin wrote:

> What do new gcc's do? Why is it impossible to chain the code? Do you have
> an example object file, or even better, an assembly snippet?


The op.c primitives are, well, primitive... at some point consideration
should be given to simply giving up.

That is, code the primitives in native assembler for each target CPU.

Certainly the C compiler is being well leveraged at present to do smart
things like allocating registers, but a decent assembly programmer will
make just as good or better decisions than gcc.

The path of least resistance to get there would be to generate op.s
files (using the necessary version of GCC) for all popular CPUs, and
make them part of the build. Then those could be hand optimized over
time as folks have the spare cycles to do it. This would remove specific
gcc version dependencies from the build with little effort.

Unless that's a GPL violation (distributing generated code under a
non-GPL license)... don't know about that angle. If that were the case,
the only option along this line would be coding from scratch unless GPL
is acceptable for the core. [QEMU uses mixed licensing and I don't have
it straight in my head, sorry.]

-- John.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Sat, Dec 18, 2004 at 01:07:36PM -0800, John R. Hogerhuis wrote:
> On Sat, 2004-12-18 at 06:59, Johannes Schindelin wrote:
> 
> The path of least resistance to get there would be to generate op.s
> files (using the necessary version of GCC) for all popular CPUs, and
> make them part of the build. Then those could be hand optimized over
> time as folks have the spare cycles to do it. This would remove specific
> gcc version dependencies from the build with little effort.
> 

That works quite well for now. Of course, if qemu is suddenly ported to a new
CPU arch, then it is likely that the new op.s file needed would have to be
hand coded. Still, a reasonable compromise.

> Unless that's a GPL violation (distributing generated code under a
> non-GPL license)... don't know about that angle.

It is not. GPL only applies to derived code. Generated code is derivied from
what you right, not from the gcc source code, and thus the GPL doesn't apply
to it. So the op.s files would share the same copyright as the original op.c
file.

> If that were the case,
> the only option along this line would be coding from scratch unless GPL
> is acceptable for the core. [QEMU uses mixed licensing and I don't have
> it straight in my head, sorry.]

QEMU uses BSD/MIT license for the system emulator.

It is qemu-user (the linux only one) which is GPL.

> 
> -- John.
> 
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Sat, 18 Dec 2004, Jim C. Brown wrote:

> Originally, the op_* functions for qemu were written so that they would
> have the machine code and then the rets at the very end of the function.
> qemu could then just chain them together by stripping the rets.
>
> gcc now rearranges the generated code around, and reserves the right to
> stick in extra rets any place it wants, not just the very end, which
> makes them very hard to strip off.

IIRC there was a flag to gcc to prevent just this.

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Sat, 18 Dec 2004, John R. Hogerhuis wrote:

> On Sat, 2004-12-18 at 06:59, Johannes Schindelin wrote:
>
> > What do new gcc's do? Why is it impossible to chain the code? Do you have
> > an example object file, or even better, an assembly snippet?
>
>
> The op.c primitives are, well, primitive... at some point consideration
> should be given to simply giving up.
>
> That is, code the primitives in native assembler for each target CPU.

NOOOOO! Did you -- even once -- look at op.s? It is a *big* file. A
quick glance at my reasons for the "NOOOOO!":

- it is a *lot* of work.
- you loose *all* of C's type safety.
- it is much too easy to miss (even unintentional) bugs.
- it is not at all portable.
- it is a maintenance nightmare.

> Certainly the C compiler is being well leveraged at present to do smart
> things like allocating registers, but a decent assembly programmer will
> make just as good or better decisions than gcc.

No. That's just not true. It might have been true with 80x86 up to 80286,
but even the earliest Pentia had several pipeline and caching quirks which
made it virtually impossible to beat very good compilers.

And this is only for Intel platforms. Ask *anybody* with MIPS experience.
There are probably just under 20 people (some claim it's more like 5) on
this planet who could beat the compiler -- and this only in case of short
functions.

One very strong point about QEmu is that it's very portable. Not only is
it easier to find strange bugs using different platforms (if you don't
believe me, I can give you very good examples), but it is important for
the main purpose of an emulator: to run on newer machines.

I agree with you on your point about least resistance. But as a Bazaar
project we have the big advantage that our quality doesn't suffer from
silly deadlines or plain dumb suits (yes, I mean all those business types
who don't have a clue about the subject they're deciding). We should make
it clear to the gcc people that there is a demand for the feature
we're seeking, namely to force the use of exactly one ret.

BTW is there a way to declare an external label? We could replace all
returns in op.c by "jmp exit_label", and parse for this instead of "ret",
if that works.

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Fabrice Bellard]

Juergen Lock wrote:
> On Thu, Dec 16, 2004 at 01:37:53AM +0000, Johannes Schindelin wrote:
> 
>>Hi,
>>
>>On Wed, 15 Dec 2004, Jim C. Brown wrote:
>>
>>
>>>This is due to a change in the way gcc compiles code ... 3.4.0 may work,
>>>but later versions will cause functions to return prematurely (due to the way
>>>qemu handles translated blocks). I suggested that a fix for this would be to
>>>use function pointer calls instead of directly chaining machine code, but
>>>apparently this is too slow to make qemu usable.
>>
>>The Forth way ;-)
>>...
> 
> 
> How about instead post-processing the .s, replacing the rets with jumps
> to the end of the function?  Of course this part would be architecture
> dependant (different assembler syntax), but other parts of qemu
> are that as well, right?

I don't exclude the possibility of using a hand coded code generator. It 
should be possible to do the transition progressively. The first 
candidates could be the code to handle memory access, constant stores 
and helper calls.

Fabrice.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Sun, 19 Dec 2004, Fabrice Bellard wrote:

> I don't exclude the possibility of using a hand coded code generator. It
> should be possible to do the transition progressively. The first
> candidates could be the code to handle memory access, constant stores
> and helper calls.

Is it really that sensible to do that? I mean, up to now porting's really
not much more than writing a dynamic linker.

If you would ask me, I'd rather go for GNU Lightning, which is LGPL, and
has a clean interface. I did not yet play with it, but a friend told me
that it's quite fast, and maybe it's possible to do something Hotspot
like. The downside: Lightning only supports i386, ppc and sparc backends
AFAIK. OTOH this could lead to a synergetic effect.

Ciao,
Dscho

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Sun, Dec 19, 2004 at 03:25:26PM +0100, Johannes Schindelin wrote:
> Hi,
> 
> On Sat, 18 Dec 2004, Jim C. Brown wrote:
> 
> > Originally, the op_* functions for qemu were written so that they would
> > have the machine code and then the rets at the very end of the function.
> > qemu could then just chain them together by stripping the rets.
> >
> > gcc now rearranges the generated code around, and reserves the right to
> > stick in extra rets any place it wants, not just the very end, which
> > makes them very hard to strip off.
> 
> IIRC there was a flag to gcc to prevent just this.
> 
> Ciao,
> Dscho
> 

AFAIK, there was no such flag. Apparently, you can't have such a flag, due to
the redesign of the gcc optimizer.

Of course, that was said to me a while ago ... it may be that someone manged to
nag the gcc team enough to get them to add this flag just for us. If that is so,
we just need to add that flag to the makefiles.

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Rob Browning]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> If you would ask me, I'd rather go for GNU Lightning, which is LGPL,
> and has a clean interface. I did not yet play with it, but a friend
> told me that it's quite fast, and maybe it's possible to do
> something Hotspot like. The downside: Lightning only supports i386,
> ppc and sparc backends AFAIK. OTOH this could lead to a synergetic
> effect.

It might be interesting to note that languages like Scheme have
similar needs, at least when considering direct compilation.  Scheme
requires tail-call optimization, and so compiling Scheme functions
directly to C functions using the normal C ABI won't work because the
caller, rather than the callee, cleans up the stack, and tail-call
optimization requires that there be no stack growth when possible.

I did have a conversation with one of the gcc developers about whether
or not it might be possible to make gcc more amenable to non-C-like
languages, specifically with respect to tail-calls, and he was very
helpful.  In the end he suggested that the most likely way that the
relevant changes would be accepted into gcc would be if they were done
at the RTL level.  So it would be possible to create a front end for a
particular language that took advantage of the new RTL annotations.

For similar reasons the language Mercury actually uses (or has used)
what they call "The Evil Mangler"
(http://www.cse.unsw.edu.au/~chak/haskell/ghc/comm/the-beast/mangler.html)
which post-processes gcc's assembly.

With respect to your comment about GNU Lightning, a while ago there
was also a bit of experimentation with using it in Guile.

In any case, I mention all of this because it suggests that there
might be a non-trivial audience for a good, cross-platform method of
generating code or code-fragments when your needs don't quite match
the C ABI.

Oh, and another possibly relevant project that was mentioned to me,
but I havent investigated much yet, is C--
(http://www.cminusminus.org).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org; previously @cs.utexas.edu
GPG starting 2002-11-03 = 14DD 432F AE39 534D B592  F9A0 25C8 D377 8C7E 73A4

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[John R. Hogerhuis]

On Sun, 2004-12-19 at 06:45, Johannes Schindelin wrote:
> > The op.c primitives are, well, primitive... at some point consideration
> > should be given to simply giving up.
> >
> > That is, code the primitives in native assembler for each target CPU.
> 
> NOOOOO! Did you -- even once -- look at op.s? It is a *big* file. A
> quick glance at my reasons for the "NOOOOO!":
> 

Yes, I've looked at op.s when I was working on a patch some time back.
You're right, it's really big. Hand written though, with macros and
such, I think it would be much much smaller.

> - it is a *lot* of work.

> - you loose *all* of C's type safety.

You have to admit GCC is already being a little abused here. I don't
think any writer of the C compiler would have anticipated the way QEMU
makes use of it. It is designed to generate object modules for a linker,
that's it.

It's a clever hack, but if it didn't work, and no one had done it, I
would have said it likely won't work, you'll end up needing to hack up a
custom compiler just for this purpose.

Turned out I would have been wrong for old versions of GCC. That's
changed with newer ones. I think it will only get worse over time.

> - it is much too easy to miss (even unintentional) bugs.

It is too easy to miss bugs in any language. On the other side of it,
assembly language programmers know they are working without a net and
tend to be much more careful than C programmers.

> - it is not at all portable.

It's just a different level of portablility. Sort of like the
portability of Forth from one architecture to another. You just need to
port the core wordset and you get the high level words for free.

> - it is a maintenance nightmare.

Any sizable piece of software is a maintenance nightmare. I think you
are overstating the negative case (there certainly is a negative
case...) I think the complexity of a core set of primitives in assembler
is manageable, especially since for most CPUs we can start out with
autogenerated versions.

To my mind the status quo is the bigger nightmare. In the current
situation, QEMU breaks through a layer of abstraction/encapsulation that
writers of GCC ought to be able to expect.

I doubt casual programmers would have any necessity to touch op_xxx.s. 
The point is to solve the current maintenance issue (I won't call it a
nightmare yet). QEMU loses its implementation depedencies on GCC in
order that it becomes more easily maintainable given that GCC changes
will no longer break the build.

> 
> > Certainly the C compiler is being well leveraged at present to do smart
> > things like allocating registers, but a decent assembly programmer will
> > make just as good or better decisions than gcc.
> 
> No. That's just not true. It might have been true with 80x86 up to 80286,
> but even the earliest Pentia had several pipeline and caching quirks which
> made it virtually impossible to beat very good compilers.
> 

That's pretty much true. But with primitives we are rearranging in
arbitrary order at runtime, I would *guess* the compiler can't really
help us much.


> And this is only for Intel platforms. Ask *anybody* with MIPS experience.
> There are probably just under 20 people (some claim it's more like 5) on
> this planet who could beat the compiler -- and this only in case of short
> functions.
> 

Well in the case of MIPS, we start out with an autogenerated op.s, the
best that GCC can do. It can only get better with hand tuning and
benchmark testing.

And I would submit that the assembly language programmer will *always*
do the same or better than the compiler. Why? Simply because the
assembly language programmer always has the option of starting with
generated code, hand optimize it, and test the performance to see
whether it got any better.


> One very strong point about QEmu is that it's very portable. Not only is
> it easier to find strange bugs using different platforms (if you don't
> believe me, I can give you very good examples), but it is important for
> the main purpose of an emulator: to run on newer machines.
> 
> I agree with you on your point about least resistance. But as a Bazaar
> project we have the big advantage that our quality doesn't suffer from
> silly deadlines or plain dumb suits (yes, I mean all those business types
> who don't have a clue about the subject they're deciding). We should make
> it clear to the gcc people that there is a demand for the feature
> we're seeking, namely to force the use of exactly one ret.
> 

Yes, but deadlines and suits aren't the issue. The issue is dependency
on another project, gcc. GCC changes are going to continue to break the
build, unless they choose to support the kind of controls on code
generation QEMU and perhaps other projects (interpreters?) needs. QEMU
can't just stay on an old verson of GCC, since new CPUs can and will be
released. Should we fork GCC? Port the existing version ourselves to
each new architecture? I think porting op.s to each new arch will be
much easier. I submit that OSS projects normally take the path of least
resistance, get something working and make improvements/optimizations
later. That "it works" attracts more people to the project, and thus
more bugfixes improvements.

That's philosophy though. The reality is that qemu build doesn't work
with new versions of gcc. You have to remove the direct dependency
altogether: hand code, as I suggest, or use another code generator, c--,
or a generic assembly language as discussed in other threads, that is
actually designed for this purpose.

If we have reason to believe that the GCC folks are genuinely interested
in providing documented code generation facilities of the kind that QEMU
and perhaps other projects really need, that's great.

> BTW is there a way to declare an external label? We could replace all
> returns in op.c by "jmp exit_label", and parse for this instead of "ret",
> if that works.

Perhaps it would make sense to split QEMU into a QEMU Porting Kit and
QEMU. QEMU porting kit could be used to generate op.s files for QEMU.
Then QEMU is no longer dependent on gcc implementation. If gcc breaks
the porting kit it might still generate some .s that could be adjusted
with a text editor or arbitrarily complex evil scripts to make something
useful. But at least that mess would be kept separate from the main
build and no one would feel that pain until a new CPU architecture comes
out to port to.

-- John.

Removing GCC dependency, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

Wow, that was a detailed explanation. I do understand your points now. I
guess that the path QEmu takes will be the path that the first one writing
a proof-of-concept takes...

Ciao,
Dscho

Re: Removing GCC dependency, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[John R. Hogerhuis]

On Mon, 2004-12-20 at 07:16, Johannes Schindelin wrote:
> Hi,
> 
> Wow, that was a detailed explanation. I do understand your points now. I
> guess that the path QEmu takes will be the path that the first one writing
> a proof-of-concept takes...

Well I thought I'd put the idea out there for discussion. The path QEMU
takes will be whichever one Fabrice decides is best, of course.

-- John.

Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[John R. Hogerhuis]

On Sun, 2004-12-19 at 19:31, Rob Browning wrote:

> Oh, and another possibly relevant project that was mentioned to me,
> but I havent investigated much yet, is C--
> (http://www.cminusminus.org).

Poking around there, I found they recommend reading this paper:
http://www.cminusminus.org/download/ppdp.pdf

It doesn't assume a lot of knowledge of the subject, so it is readable
even to me :-)

-- John.

Re: Removing GCC dependency, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Fabrice Bellard]

John R. Hogerhuis wrote:
> On Mon, 2004-12-20 at 07:16, Johannes Schindelin wrote:
> 
>>Hi,
>>
>>Wow, that was a detailed explanation. I do understand your points now. I
>>guess that the path QEmu takes will be the path that the first one writing
>>a proof-of-concept takes...
> 
> 
> Well I thought I'd put the idea out there for discussion. The path QEMU
> takes will be whichever one Fabrice decides is best, of course.

I made tests with QEMU and gcc 3.4.2 and I saw no problems (the macro 
FORCE_RET() in op.c is there to ensure that gcc does not generate ret 
inside a micro op. Moreover, -fno-reorder-blocks is used). Can someone 
give precise examples of bugs with gcc 3.4.2 ?

If we really cannot find a way to trick the next gcc versions, then 
progressively switching to a hand coded code generator is the solution I 
suggest. Using '.s' sources will be more complicated.

I always thought the current 'dyngen' solution as a temporary one 
because it enabled a fast developpement of QEMU as the expense of 
performance (this is relative of course, QEMU is already quite fast and 
usable). But if we want better performances and better... portability 
(in the sense that we are not tied to a specific GCC version), then a 
hand coded code generator is the only way to go.

The priority for the hand coded code generation should be:

1) helper calls with constant parameters or temporary parameters. It is 
very important to get better performances because on some archs (e.g. 
ppc), the current scheme is not efficient at all.

2) conditionnal jumps. These are the points that newer gccs may not 
compile correctly.

3) constant loads.

4) simple operations (all arithmetic and logic between temporaries)

5) memory accesses.

Those points are easy to implement and would give a good speed up. The 
important point is that such a transition can be made progressively by 
using specific micro operations indexes.

Fabrice.

[Qemu-devel] Testing with Win2k SP3 and Win2kSP4

[Elefterios Stamatogiannakis]

  I recently tested qemu. The configuration i used:
  Host: Win2k SP4
  Guests tested: Win2k SP3 and Win2k SP4
  Network: User-net
  Command Line used:
START /MIN qemu.exe -L . -pci -localtime -cirrusvga -user-net -boot c 
-hda image.dsk
  Qemu version: Nightly builds on freeoszoo during the last week.

  Guest win2k was installed using qemu and an image of Win2kSP3 made 
using real hardware. This image has been tested at least 40 times on a 
wide array of hardware and it works without problems.

Summary of problems:
-Windows update doesn't work due to failed Schannel (SSL and others 
channel implementation). Maybe schannel depends on hardware to gather 
entropy?
-Microsoft installer doesn't work.

  Has anybody else tested Qemu with Win2k SP>2 guest and usernet? Any 
similar problems? Any suggestions?

  Linux as a guest works perfectly under qemu.

Detailed description:

-"Antivir Guard" antivirus keeps trying to enable itself. The tray icon 
switches on and off constantly. I disabled it.

-"Microsoft Installer fails". Ms office cannot open as a result of this. 
When the image goes into a new computer, the first time somebody tries 
to open an office application the Installer gets invoked. Installer 
fails under qemu so Word cannot open. An event log for failed 
msiinstaller concerning auto update follows.
Event Log for Installer:
Source MsiInstaller, Type:Error, Event ID:10000
  The description for Event ID ( 10000 ) in Source ( MsiInstaller ) 
cannot be found. The local computer may not have the necessary registry 
information or message DLL files to display messafes from a remote 
computer. The following information is part of the even: Product:
Microsoft Windows Update Auto Update -- Microsoft Windows Update Auto 
Update requires Windows 2000 SP2 or Windows XP., (NULL), (NULL), (NULL), 
(NULL), (NULL), (NULL)

-Windows cannot display "Add/Remove Programs". This seems to be tied to 
the above.

-Schannel fail
For some reason schannel fails. This blocks Windows Update:
Error Log:
--Schannel
Source: Schannel, Type:Error, Event ID:36865
A fatal error occured while opening the system RSA cryptographi module. 
Operations that require the SSL or TLS cryptographi protocols will not 
work correctly. The error code is 0x57.
--Windows Update
Windows Update Error
Windows Update has encountered an error. This may be due to a 
discrepancy in your computer's time setting.

  Hope the above are helpfull.

  teris.

[Qemu-devel] Re: Removing GCC dependency

[Paul Brook]

> I always thought the current 'dyngen' solution as a temporary one 
> because it enabled a fast developpement of QEMU as the expense of 
> performance (this is relative of course, QEMU is already quite fast and 
> usable). But if we want better performances and better... portability 
> (in the sense that we are not tied to a specific GCC version), then a 
> hand coded code generator is the only way to go.

Do we also want to aim towards using the same code generator for all guest 
cpus? We currently have separate op.c for each emulated architecture.

Combining these into a single set of opcodes seems like a good idea.  Once we 
start doing hand coded generator and/or nontrivial optimisations on the 
generated opcodes the duplications is really going to hurt, and the bar for 
adding new architectures will be significantly higher.

I realise there complications with this (different register files and MMU 
types), but I can see large benefits if these can be overcome.

Paul

Re: [Qemu-devel] Re: Removing GCC dependency

[Herbert Poetzl]

On Tue, Dec 21, 2004 at 03:23:11PM +0000, Paul Brook wrote:
> > I always thought the current 'dyngen' solution as a temporary one 
> > because it enabled a fast developpement of QEMU as the expense of 
> > performance (this is relative of course, QEMU is already quite fast and 
> > usable). But if we want better performances and better... portability 
> > (in the sense that we are not tied to a specific GCC version), then a 
> > hand coded code generator is the only way to go.
> 
> Do we also want to aim towards using the same code generator for all guest 
> cpus? We currently have separate op.c for each emulated architecture.
> 
> Combining these into a single set of opcodes seems like a good idea.  Once we 
> start doing hand coded generator and/or nontrivial optimisations on the 
> generated opcodes the duplications is really going to hurt, and the bar for 
> adding new architectures will be significantly higher.
> 
> I realise there complications with this (different register files and MMU 
> types), but I can see large benefits if these can be overcome.
> 

again a look at 'other' binary translators like the
bintrans[1] could provide some good ideas ...

best,
Herbert

-- from the README --

    Machine Descriptions
    --------------------

bintrans uses machine descriptions to automatically generate
interpreters, disassemblers, composer macros, and liveness analysers.
Currently, there are machine descriptions for the i386, the PPC and
the Alpha, residing in the files "i386.lisp", "ppc.lisp", and
"alpha.lisp", respectively.

Should you choose to play around with them, here's how to generate the
various C files from them:

  1. Start CLISP
  2. (load "init.lisp")

then:

  3a. (generate-all-alpha-files)     ; to generate the Alpha files
  3b. (generate-all-ppc-files)       ; to generate the PPC files
  3c. (generate-all-intel-files)     ; to generate the i386 files


> Paul
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

CMinusMins, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Johannes Schindelin]

Hi,

On Mon, 20 Dec 2004, John R. Hogerhuis wrote:

> On Sun, 2004-12-19 at 19:31, Rob Browning wrote:
>
> > Oh, and another possibly relevant project that was mentioned to me,
> > but I havent investigated much yet, is C--
> > (http://www.cminusminus.org).
>
> Poking around there, I found they recommend reading this paper:
> http://www.cminusminus.org/download/ppdp.pdf

The Quick C-- Compiler is written in Objective CAML. I cannot think
of many more ways to turn down developers.

Ciao,
Dscho

Re: CMinusMins, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[malc]

On Mon, 17 Jan 2005, Johannes Schindelin wrote:

> Hi,
>
> On Mon, 20 Dec 2004, John R. Hogerhuis wrote:
>
>> On Sun, 2004-12-19 at 19:31, Rob Browning wrote:
>>
>>> Oh, and another possibly relevant project that was mentioned to me,
>>> but I havent investigated much yet, is C--
>>> (http://www.cminusminus.org).
>>
>> Poking around there, I found they recommend reading this paper:
>> http://www.cminusminus.org/download/ppdp.pdf
>
> The Quick C-- Compiler is written in Objective CAML. I cannot think
> of many more ways to turn down developers.

Please speak for yourself

-- 
mailto:malc@pulsesoft.com

Re: CMinusMins, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Jim C. Brown]

On Mon, Jan 17, 2005 at 06:36:30PM +0300, malc wrote:
> On Mon, 17 Jan 2005, Johannes Schindelin wrote:
> 
> >The Quick C-- Compiler is written in Objective CAML. I cannot think
> >of many more ways to turn down developers.

One could try to re-create a C-- compiler in C, but that seems like a lot of
work. Perhaps someone else has already done this...

I personally like the separation of C into C-- and Cylon (a real high level
language derived from C) on a theoretical level. The fact that neither of these
languages are widely used is telling, though.

> 
> Please speak for yourself

What is Objective CAML?

> 
> -- 
> mailto:malc@pulsesoft.com
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 

-- 
Infinite complexity begets infinite beauty.
Infinite precision begets infinite perfection.

Re: OCAML, was CMinusMins, was Re: [Qemu-devel] get_func() hangs with gcc 3.4.2 on MinGW and WinXP host

[Mark A. Williamson]

Apologies for going slightly OT.  I don't want to start a fight, so I'll just 
give a brief explanation.

> What is Objective CAML?

OCAML is a dialect of ML (Meta Language), the other main dialect being 
Standard ML (SML).  The ML family is a functional language with eager 
evaluation semantics (function arguments are evaluated before calling a 
function).

There is a strong emphasis on immutable data structures although all ML 
dialects that I know of include traditional features (pointers, 
assignment...) as well.  ML languages use a very powerful type-checker with a 
"type inference" algorithm (it figures out what types things should have) and 
powerful support for functions that return other functions or take others as 
arguments (nicer than function pointers).

OCAML adds Object orientation features to the ML dialect CAML.

There are various pros and cons to functional languages and I don't want to 
start a fight over the relative merits of different paradigms.  There's lots 
more information on the web, Google, Foldoc and Wikipedia will probably all 
provide useful information.

HTH,
Mark