Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

Hi James,

 Thanks for your input!

 Cc-ing linux-mips for the discussion about a ptrace(2) kernel API update; 
anyone interested in previous talk about this change please have a look 
at: <https://sourceware.org/ml/gdb-patches/2016-06/msg00441.html> and 
<https://sourceware.org/ml/gdb-patches/2016-10/msg00311.html> for the 
earlier messages.

> >  Hmm, has Linux kernel support for CP0.Config5 accesses gone upstream 
> > already?  Can you give me an upstream commit ID and/or reference to the 
> > discussion where it has been approved if so?
> 
> I don't think it did go upstream yet.

 Good!

> >  More importantly, what do we need CP0.Config5 access for in the first 
> > place?  It looks to me like this bit is irrelevant to GDB as it does not 
> > affect the native (raw) register format.  So the only use would be to let 
> > the user running a debugging session switch between the FRE and NFRE modes 
> > without the need to poke at CP1C.FRE or CP1C.NFRE registers with a CTC1 
> > instruction, which by itself makes sense to me, but needs a further 
> > consideration.
> 
> It allows the FRE bit to be read (I seem to remember this was the only
> bit actually exposed through ptrace by the patch).

 Then I think it makes sense even more not to create this artificial API 
and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
what hardware presents to user software.  Also with CP1C.UFR/CP1C.UNFR vs 
CP0.Status; while we want to retain the latter register in the view for 
historical reasons, it has always been read-only and I think it ought to 
remain such, with any writes to CP0.Status.FR executed via the former CP1C 
registers only.

> FRE simply causes certain instructions (all single precision FP
> arithmetic instructions and FP word loads/stores) to trap to the kernel
> so that it can emulate a variation/subset of FR=0, so the debugger would
> use it to decide how to decode the single precision FP registers based
> on the double precision FP registers (iirc).

 I don't think there is any value in it for GDB, I think all 64-bit FP 
registers ought to remain being presented as doubles and pairs of singles 
regardless of the mode selected (and also possibly fixed-point longs and 
pairs of fixed-point words).  We don't know what's emulated and what's not 
after all, and then the contents of FPRs are not interpreted by GDB itself 
anyhow except in user-supplied expressions or assignment requests, which 
for users' convenience I think should retain the maximum flexibility 
possible.

 So as I say it looks to me like the only, though obviously valid and 
wholeheartedly supported, use for CP1C.FRE/CP1C.NFRE would be for user's 
control of the execution environment.

> >  Additionally exposing CP0.Config5 may have security implications, 
> > especially as parts of the register have not been defined yet in the 
> > architectures and we'd have to force architecture maintainers somehow to 
> > ask us every time they intend to add a bit to this register to check if 
> > this has security implications and has to be avoided and/or explicitly 
> > handled in software.
> 
> yes, as above it explicity only shows certain bits. I'm fine with the
> api changing if necessary though since it isn't upstream.

 It sounds like a plan to me then -- any further questions or comments 
about the kernel API part, anyone?

  Maciej

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

Hi James,

 Thanks for your input!

 Cc-ing linux-mips for the discussion about a ptrace(2) kernel API update; 
anyone interested in previous talk about this change please have a look 
at: <https://sourceware.org/ml/gdb-patches/2016-06/msg00441.html> and 
<https://sourceware.org/ml/gdb-patches/2016-10/msg00311.html> for the 
earlier messages.

> >  Hmm, has Linux kernel support for CP0.Config5 accesses gone upstream 
> > already?  Can you give me an upstream commit ID and/or reference to the 
> > discussion where it has been approved if so?
> 
> I don't think it did go upstream yet.

 Good!

> >  More importantly, what do we need CP0.Config5 access for in the first 
> > place?  It looks to me like this bit is irrelevant to GDB as it does not 
> > affect the native (raw) register format.  So the only use would be to let 
> > the user running a debugging session switch between the FRE and NFRE modes 
> > without the need to poke at CP1C.FRE or CP1C.NFRE registers with a CTC1 
> > instruction, which by itself makes sense to me, but needs a further 
> > consideration.
> 
> It allows the FRE bit to be read (I seem to remember this was the only
> bit actually exposed through ptrace by the patch).

 Then I think it makes sense even more not to create this artificial API 
and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
what hardware presents to user software.  Also with CP1C.UFR/CP1C.UNFR vs 
CP0.Status; while we want to retain the latter register in the view for 
historical reasons, it has always been read-only and I think it ought to 
remain such, with any writes to CP0.Status.FR executed via the former CP1C 
registers only.

> FRE simply causes certain instructions (all single precision FP
> arithmetic instructions and FP word loads/stores) to trap to the kernel
> so that it can emulate a variation/subset of FR=0, so the debugger would
> use it to decide how to decode the single precision FP registers based
> on the double precision FP registers (iirc).

 I don't think there is any value in it for GDB, I think all 64-bit FP 
registers ought to remain being presented as doubles and pairs of singles 
regardless of the mode selected (and also possibly fixed-point longs and 
pairs of fixed-point words).  We don't know what's emulated and what's not 
after all, and then the contents of FPRs are not interpreted by GDB itself 
anyhow except in user-supplied expressions or assignment requests, which 
for users' convenience I think should retain the maximum flexibility 
possible.

 So as I say it looks to me like the only, though obviously valid and 
wholeheartedly supported, use for CP1C.FRE/CP1C.NFRE would be for user's 
control of the execution environment.

> >  Additionally exposing CP0.Config5 may have security implications, 
> > especially as parts of the register have not been defined yet in the 
> > architectures and we'd have to force architecture maintainers somehow to 
> > ask us every time they intend to add a bit to this register to check if 
> > this has security implications and has to be avoided and/or explicitly 
> > handled in software.
> 
> yes, as above it explicity only shows certain bits. I'm fine with the
> api changing if necessary though since it isn't upstream.

 It sounds like a plan to me then -- any further questions or comments 
about the kernel API part, anyone?

  Maciej

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 5915 bytes --]

On Wed, Oct 12, 2016 at 05:29:53PM +0100, Maciej W. Rozycki wrote:
> Hi James,
> 
>  Thanks for your input!
> 
>  Cc-ing linux-mips for the discussion about a ptrace(2) kernel API update; 
> anyone interested in previous talk about this change please have a look 
> at: <https://sourceware.org/ml/gdb-patches/2016-06/msg00441.html> and 
> <https://sourceware.org/ml/gdb-patches/2016-10/msg00311.html> for the 
> earlier messages.
> 
> > >  Hmm, has Linux kernel support for CP0.Config5 accesses gone upstream 
> > > already?  Can you give me an upstream commit ID and/or reference to the 
> > > discussion where it has been approved if so?
> > 
> > I don't think it did go upstream yet.
> 
>  Good!
> 
> > >  More importantly, what do we need CP0.Config5 access for in the first 
> > > place?  It looks to me like this bit is irrelevant to GDB as it does not 
> > > affect the native (raw) register format.  So the only use would be to let 
> > > the user running a debugging session switch between the FRE and NFRE modes 
> > > without the need to poke at CP1C.FRE or CP1C.NFRE registers with a CTC1 
> > > instruction, which by itself makes sense to me, but needs a further 
> > > consideration.
> > 
> > It allows the FRE bit to be read (I seem to remember this was the only
> > bit actually exposed through ptrace by the patch).
> 
>  Then I think it makes sense even more not to create this artificial API 
> and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> what hardware presents to user software.

well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
since FP mode changes must be done for all threads in the process, so
userland can't actually directly access those FCRs either.

> Also with CP1C.UFR/CP1C.UNFR vs 
> CP0.Status; while we want to retain the latter register in the view for 
> historical reasons, it has always been read-only and I think it ought to 
> remain such, with any writes to CP0.Status.FR executed via the former CP1C 
> registers only.
> 
> > FRE simply causes certain instructions (all single precision FP
> > arithmetic instructions and FP word loads/stores) to trap to the kernel
> > so that it can emulate a variation/subset of FR=0, so the debugger would
> > use it to decide how to decode the single precision FP registers based
> > on the double precision FP registers (iirc).
> 
>  I don't think there is any value in it for GDB, I think all 64-bit FP 
> registers ought to remain being presented as doubles and pairs of singles 
> regardless of the mode selected (and also possibly fixed-point longs and 
> pairs of fixed-point words).  We don't know what's emulated and what's not 
> after all, and then the contents of FPRs are not interpreted by GDB itself 
> anyhow except in user-supplied expressions or assignment requests, which 
> for users' convenience I think should retain the maximum flexibility 
> possible.

Well it technically depends on
test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)

So it allows gdb to detect Linux's Hybrid FPU mode, and present the fp
regs (e.g. f0 or f1) more like below to the user depending on the fp
mode, which is surely much more intuitive from an assembly debugging
point of view.

Its also worth noting that "When software changes the value of this bit
[Status.FR], the contents of the floating-point registers are
UNPREDICTABLE". I.e. there is no architectural unifying register layout
between FR=0 / FR=1, only convention. Linux will I presume intentionally
save in old mode and restore in new mode on an fp mode change according
to its own convention (such that the valid mode changes don't clobber
register state).

(disclaimer: I haven't looked at this gdb patchset in detail as to
whether any of below has changed since I worked on it).

(1) Even singles and doubles always overlap one another, as do odd
singles and doubles when FR=1 (and FRE=0):
	/* (little endian) */
	union __gdb_builtin_mips_fp64 {
	  float  f32;
	  double f64;
	  int32  i32;
	  int64  i64;
	};

(2) Odd singles when FR=0 (there are no odd doubles):
	union __gdb_builtin_mips_fp32 {
	  float f32;
	  int32_t i32;
	};

(3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
	struct __gdb_builtin_mips_fp96 {
		union {
			double f64;
			int64  i64;
		};
		union {
			float  f32;
			int32  i32;
		};
	};

i.e.

FR=0:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210
 (2) odd
       single:	FEDCBA98

FR=1, FRE=0:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210
 (1) odd
       double:	                0123456789ABCDEF
       single:	                        89ABCDEF

FR=1, FRE=1:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210   (Hybrid FPR emulation)
 (3) odd
       double:	                0123456789ABCDEF
       single:	FEDCBA98           (Hybrid FPR emulation)
)

Cheers
James

> 
>  So as I say it looks to me like the only, though obviously valid and 
> wholeheartedly supported, use for CP1C.FRE/CP1C.NFRE would be for user's 
> control of the execution environment.
> 
> > >  Additionally exposing CP0.Config5 may have security implications, 
> > > especially as parts of the register have not been defined yet in the 
> > > architectures and we'd have to force architecture maintainers somehow to 
> > > ask us every time they intend to add a bit to this register to check if 
> > > this has security implications and has to be avoided and/or explicitly 
> > > handled in software.
> > 
> > yes, as above it explicity only shows certain bits. I'm fine with the
> > api changing if necessary though since it isn't upstream.
> 
>  It sounds like a plan to me then -- any further questions or comments 
> about the kernel API part, anyone?
> 
>   Maciej

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 5915 bytes --]

On Wed, Oct 12, 2016 at 05:29:53PM +0100, Maciej W. Rozycki wrote:
> Hi James,
> 
>  Thanks for your input!
> 
>  Cc-ing linux-mips for the discussion about a ptrace(2) kernel API update; 
> anyone interested in previous talk about this change please have a look 
> at: <https://sourceware.org/ml/gdb-patches/2016-06/msg00441.html> and 
> <https://sourceware.org/ml/gdb-patches/2016-10/msg00311.html> for the 
> earlier messages.
> 
> > >  Hmm, has Linux kernel support for CP0.Config5 accesses gone upstream 
> > > already?  Can you give me an upstream commit ID and/or reference to the 
> > > discussion where it has been approved if so?
> > 
> > I don't think it did go upstream yet.
> 
>  Good!
> 
> > >  More importantly, what do we need CP0.Config5 access for in the first 
> > > place?  It looks to me like this bit is irrelevant to GDB as it does not 
> > > affect the native (raw) register format.  So the only use would be to let 
> > > the user running a debugging session switch between the FRE and NFRE modes 
> > > without the need to poke at CP1C.FRE or CP1C.NFRE registers with a CTC1 
> > > instruction, which by itself makes sense to me, but needs a further 
> > > consideration.
> > 
> > It allows the FRE bit to be read (I seem to remember this was the only
> > bit actually exposed through ptrace by the patch).
> 
>  Then I think it makes sense even more not to create this artificial API 
> and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> what hardware presents to user software.

well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
since FP mode changes must be done for all threads in the process, so
userland can't actually directly access those FCRs either.

> Also with CP1C.UFR/CP1C.UNFR vs 
> CP0.Status; while we want to retain the latter register in the view for 
> historical reasons, it has always been read-only and I think it ought to 
> remain such, with any writes to CP0.Status.FR executed via the former CP1C 
> registers only.
> 
> > FRE simply causes certain instructions (all single precision FP
> > arithmetic instructions and FP word loads/stores) to trap to the kernel
> > so that it can emulate a variation/subset of FR=0, so the debugger would
> > use it to decide how to decode the single precision FP registers based
> > on the double precision FP registers (iirc).
> 
>  I don't think there is any value in it for GDB, I think all 64-bit FP 
> registers ought to remain being presented as doubles and pairs of singles 
> regardless of the mode selected (and also possibly fixed-point longs and 
> pairs of fixed-point words).  We don't know what's emulated and what's not 
> after all, and then the contents of FPRs are not interpreted by GDB itself 
> anyhow except in user-supplied expressions or assignment requests, which 
> for users' convenience I think should retain the maximum flexibility 
> possible.

Well it technically depends on
test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)

So it allows gdb to detect Linux's Hybrid FPU mode, and present the fp
regs (e.g. f0 or f1) more like below to the user depending on the fp
mode, which is surely much more intuitive from an assembly debugging
point of view.

Its also worth noting that "When software changes the value of this bit
[Status.FR], the contents of the floating-point registers are
UNPREDICTABLE". I.e. there is no architectural unifying register layout
between FR=0 / FR=1, only convention. Linux will I presume intentionally
save in old mode and restore in new mode on an fp mode change according
to its own convention (such that the valid mode changes don't clobber
register state).

(disclaimer: I haven't looked at this gdb patchset in detail as to
whether any of below has changed since I worked on it).

(1) Even singles and doubles always overlap one another, as do odd
singles and doubles when FR=1 (and FRE=0):
	/* (little endian) */
	union __gdb_builtin_mips_fp64 {
	  float  f32;
	  double f64;
	  int32  i32;
	  int64  i64;
	};

(2) Odd singles when FR=0 (there are no odd doubles):
	union __gdb_builtin_mips_fp32 {
	  float f32;
	  int32_t i32;
	};

(3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
	struct __gdb_builtin_mips_fp96 {
		union {
			double f64;
			int64  i64;
		};
		union {
			float  f32;
			int32  i32;
		};
	};

i.e.

FR=0:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210
 (2) odd
       single:	FEDCBA98

FR=1, FRE=0:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210
 (1) odd
       double:	                0123456789ABCDEF
       single:	                        89ABCDEF

FR=1, FRE=1:
 (1) even
       double:	FEDCBA9876543210
       single:	        76543210   (Hybrid FPR emulation)
 (3) odd
       double:	                0123456789ABCDEF
       single:	FEDCBA98           (Hybrid FPR emulation)
)

Cheers
James

> 
>  So as I say it looks to me like the only, though obviously valid and 
> wholeheartedly supported, use for CP1C.FRE/CP1C.NFRE would be for user's 
> control of the execution environment.
> 
> > >  Additionally exposing CP0.Config5 may have security implications, 
> > > especially as parts of the register have not been defined yet in the 
> > > architectures and we'd have to force architecture maintainers somehow to 
> > > ask us every time they intend to add a bit to this register to check if 
> > > this has security implications and has to be avoided and/or explicitly 
> > > handled in software.
> > 
> > yes, as above it explicity only shows certain bits. I'm fine with the
> > api changing if necessary though since it isn't upstream.
> 
>  It sounds like a plan to me then -- any further questions or comments 
> about the kernel API part, anyone?
> 
>   Maciej

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

On Wed, 12 Oct 2016, James Hogan wrote:

> >  Then I think it makes sense even more not to create this artificial API 
> > and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> > what hardware presents to user software.
> 
> well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
> since FP mode changes must be done for all threads in the process, so
> userland can't actually directly access those FCRs either.

 Hmm, I didn't know that -- what was the reason for this design decision?  
Offhand the limitation does not appear necessary to me, each thread has 
its own distinct register set, so it does not appear to me that its mode 
of operation has to be the same across them all.  The current setting 
would still of course be inherited from the parent by any new threads 
created with clone(2).

 Anyway in that case the presented CP1C registers will have to be 
read-only.

> >  I don't think there is any value in it for GDB, I think all 64-bit FP 
> > registers ought to remain being presented as doubles and pairs of singles 
> > regardless of the mode selected (and also possibly fixed-point longs and 
> > pairs of fixed-point words).  We don't know what's emulated and what's not 
> > after all, and then the contents of FPRs are not interpreted by GDB itself 
> > anyhow except in user-supplied expressions or assignment requests, which 
> > for users' convenience I think should retain the maximum flexibility 
> > possible.
> 
> Well it technically depends on
> test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)

 Sure, but the hardware representation is CP0.Config5.FRE/CP1C.FRE.

> So it allows gdb to detect Linux's Hybrid FPU mode, and present the fp
> regs (e.g. f0 or f1) more like below to the user depending on the fp
> mode, which is surely much more intuitive from an assembly debugging
> point of view.
> 
> Its also worth noting that "When software changes the value of this bit
> [Status.FR], the contents of the floating-point registers are
> UNPREDICTABLE". I.e. there is no architectural unifying register layout
> between FR=0 / FR=1, only convention. Linux will I presume intentionally
> save in old mode and restore in new mode on an fp mode change according
> to its own convention (such that the valid mode changes don't clobber
> register state).

 Well the contents might be unpredictable, but there sure will be some and 
GDB is supposed to present it.

> (disclaimer: I haven't looked at this gdb patchset in detail as to
> whether any of below has changed since I worked on it).
> 
> (1) Even singles and doubles always overlap one another, as do odd
> singles and doubles when FR=1 (and FRE=0):
> 	/* (little endian) */
> 	union __gdb_builtin_mips_fp64 {
> 	  float  f32;
> 	  double f64;
> 	  int32  i32;
> 	  int64  i64;
> 	};
> 
> (2) Odd singles when FR=0 (there are no odd doubles):
> 	union __gdb_builtin_mips_fp32 {
> 	  float f32;
> 	  int32_t i32;
> 	};
> 
> (3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
> 	struct __gdb_builtin_mips_fp96 {
> 		union {
> 			double f64;
> 			int64  i64;
> 		};
> 		union {
> 			float  f32;
> 			int32  i32;
> 		};
> 	};
> 
> i.e.
> 
> FR=0:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210
>  (2) odd
>        single:	FEDCBA98
> 
> FR=1, FRE=0:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210
>  (1) odd
>        double:	                0123456789ABCDEF
>        single:	                        89ABCDEF
> 
> FR=1, FRE=1:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210   (Hybrid FPR emulation)
>  (3) odd
>        double:	                0123456789ABCDEF
>        single:	FEDCBA98           (Hybrid FPR emulation)
> )

 I haven't got to this part so far and either way will have to think about 
it yet.  For one as I noted we do want to present vector (paired-single) 
data with FR=1, FRE=0 in addition to what you quoted above.  This was all 
implemented in an old MIPS UK patch originally written by Nigel Stephens 
and included with SDE, which I've never got to upstreaming; have you by 
any chance based your work on that?

 As to FR=1, FRE=1 your quoted representation of singles is a software 
convention only, so I'm not sure offhand how we might represent it in GDB 
to keep it reasonable; the 96-bit cooked FP register structure does not 
appeal to me at all TBH, but maybe it's the best we can do after all.

  Maciej

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

On Wed, 12 Oct 2016, James Hogan wrote:

> >  Then I think it makes sense even more not to create this artificial API 
> > and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> > what hardware presents to user software.
> 
> well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
> since FP mode changes must be done for all threads in the process, so
> userland can't actually directly access those FCRs either.

 Hmm, I didn't know that -- what was the reason for this design decision?  
Offhand the limitation does not appear necessary to me, each thread has 
its own distinct register set, so it does not appear to me that its mode 
of operation has to be the same across them all.  The current setting 
would still of course be inherited from the parent by any new threads 
created with clone(2).

 Anyway in that case the presented CP1C registers will have to be 
read-only.

> >  I don't think there is any value in it for GDB, I think all 64-bit FP 
> > registers ought to remain being presented as doubles and pairs of singles 
> > regardless of the mode selected (and also possibly fixed-point longs and 
> > pairs of fixed-point words).  We don't know what's emulated and what's not 
> > after all, and then the contents of FPRs are not interpreted by GDB itself 
> > anyhow except in user-supplied expressions or assignment requests, which 
> > for users' convenience I think should retain the maximum flexibility 
> > possible.
> 
> Well it technically depends on
> test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)

 Sure, but the hardware representation is CP0.Config5.FRE/CP1C.FRE.

> So it allows gdb to detect Linux's Hybrid FPU mode, and present the fp
> regs (e.g. f0 or f1) more like below to the user depending on the fp
> mode, which is surely much more intuitive from an assembly debugging
> point of view.
> 
> Its also worth noting that "When software changes the value of this bit
> [Status.FR], the contents of the floating-point registers are
> UNPREDICTABLE". I.e. there is no architectural unifying register layout
> between FR=0 / FR=1, only convention. Linux will I presume intentionally
> save in old mode and restore in new mode on an fp mode change according
> to its own convention (such that the valid mode changes don't clobber
> register state).

 Well the contents might be unpredictable, but there sure will be some and 
GDB is supposed to present it.

> (disclaimer: I haven't looked at this gdb patchset in detail as to
> whether any of below has changed since I worked on it).
> 
> (1) Even singles and doubles always overlap one another, as do odd
> singles and doubles when FR=1 (and FRE=0):
> 	/* (little endian) */
> 	union __gdb_builtin_mips_fp64 {
> 	  float  f32;
> 	  double f64;
> 	  int32  i32;
> 	  int64  i64;
> 	};
> 
> (2) Odd singles when FR=0 (there are no odd doubles):
> 	union __gdb_builtin_mips_fp32 {
> 	  float f32;
> 	  int32_t i32;
> 	};
> 
> (3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
> 	struct __gdb_builtin_mips_fp96 {
> 		union {
> 			double f64;
> 			int64  i64;
> 		};
> 		union {
> 			float  f32;
> 			int32  i32;
> 		};
> 	};
> 
> i.e.
> 
> FR=0:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210
>  (2) odd
>        single:	FEDCBA98
> 
> FR=1, FRE=0:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210
>  (1) odd
>        double:	                0123456789ABCDEF
>        single:	                        89ABCDEF
> 
> FR=1, FRE=1:
>  (1) even
>        double:	FEDCBA9876543210
>        single:	        76543210   (Hybrid FPR emulation)
>  (3) odd
>        double:	                0123456789ABCDEF
>        single:	FEDCBA98           (Hybrid FPR emulation)
> )

 I haven't got to this part so far and either way will have to think about 
it yet.  For one as I noted we do want to present vector (paired-single) 
data with FR=1, FRE=0 in addition to what you quoted above.  This was all 
implemented in an old MIPS UK patch originally written by Nigel Stephens 
and included with SDE, which I've never got to upstreaming; have you by 
any chance based your work on that?

 As to FR=1, FRE=1 your quoted representation of singles is a software 
convention only, so I'm not sure offhand how we might represent it in GDB 
to keep it reasonable; the 96-bit cooked FP register structure does not 
appeal to me at all TBH, but maybe it's the best we can do after all.

  Maciej

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 4215 bytes --]

On Wed, Oct 12, 2016 at 11:04:18PM +0100, Maciej W. Rozycki wrote:
> On Wed, 12 Oct 2016, James Hogan wrote:
> 
> > >  Then I think it makes sense even more not to create this artificial API 
> > > and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> > > what hardware presents to user software.
> > 
> > well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
> > since FP mode changes must be done for all threads in the process, so
> > userland can't actually directly access those FCRs either.
> 
>  Hmm, I didn't know that -- what was the reason for this design decision?  
> Offhand the limitation does not appear necessary to me, each thread has 
> its own distinct register set, so it does not appear to me that its mode 
> of operation has to be the same across them all.  The current setting 
> would still of course be inherited from the parent by any new threads 
> created with clone(2).

Paul Burton & Matt Fortune know the details and can correct me if I'm
wrong, but I believe the idea is that the mode change will usually be
initiated by the dynamic loader in repsonse to loading a new shared
library with a more specific FPU ABI (but must of course be compatible
with the current mode). As such you must be careful that all threads in
the process change mode so that they can immediately start using the new
dynamically loaded code using the more specific ABI.

https://dmz-portal.mips.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking#10.5.2._Setting_the_FPU_mode

> > (disclaimer: I haven't looked at this gdb patchset in detail as to
> > whether any of below has changed since I worked on it).
> > 
> > (1) Even singles and doubles always overlap one another, as do odd
> > singles and doubles when FR=1 (and FRE=0):
> > 	/* (little endian) */
> > 	union __gdb_builtin_mips_fp64 {
> > 	  float  f32;
> > 	  double f64;
> > 	  int32  i32;
> > 	  int64  i64;
> > 	};
> > 
> > (2) Odd singles when FR=0 (there are no odd doubles):
> > 	union __gdb_builtin_mips_fp32 {
> > 	  float f32;
> > 	  int32_t i32;
> > 	};
> > 
> > (3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
> > 	struct __gdb_builtin_mips_fp96 {
> > 		union {
> > 			double f64;
> > 			int64  i64;
> > 		};
> > 		union {
> > 			float  f32;
> > 			int32  i32;
> > 		};
> > 	};
> > 
> > i.e.
> > 
> > FR=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (2) odd
> >        single:	FEDCBA98
> > 
> > FR=1, FRE=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (1) odd
> >        double:	                0123456789ABCDEF
> >        single:	                        89ABCDEF
> > 
> > FR=1, FRE=1:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210   (Hybrid FPR emulation)
> >  (3) odd
> >        double:	                0123456789ABCDEF
> >        single:	FEDCBA98           (Hybrid FPR emulation)
> > )
> 
>  I haven't got to this part so far and either way will have to think about 
> it yet.  For one as I noted we do want to present vector (paired-single) 
> data with FR=1, FRE=0 in addition to what you quoted above.

Right, I hadn't looked into that.

> This was all 
> implemented in an old MIPS UK patch originally written by Nigel Stephens 
> and included with SDE, which I've never got to upstreaming; have you by 
> any chance based your work on that?

No.

>  As to FR=1, FRE=1 your quoted representation of singles is a software 
> convention only, so I'm not sure offhand how we might represent it in GDB 
> to keep it reasonable; the 96-bit cooked FP register structure does not 
> appeal to me at all TBH, but maybe it's the best we can do after all.

Me neither, but it at least seems to look reasonable from an assembly
debugging point of view. It took some effort to bend GDB to my will to
be honest, especially around mode changes and the remote protocol (since
the remote could change from FR=0 (32-bit FP registers) to FR=1 (64-bit
FP registers even on 32-bit) at almost any time, but I'm really no gdb
expert.

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 4215 bytes --]

On Wed, Oct 12, 2016 at 11:04:18PM +0100, Maciej W. Rozycki wrote:
> On Wed, 12 Oct 2016, James Hogan wrote:
> 
> > >  Then I think it makes sense even more not to create this artificial API 
> > > and use the CP1C.FRE/CP1C.NFRE registers instead which do correspond to 
> > > what hardware presents to user software.
> > 
> > well, barely. Linux at least doesn't enable Config5.UFE or Config5.UFR,
> > since FP mode changes must be done for all threads in the process, so
> > userland can't actually directly access those FCRs either.
> 
>  Hmm, I didn't know that -- what was the reason for this design decision?  
> Offhand the limitation does not appear necessary to me, each thread has 
> its own distinct register set, so it does not appear to me that its mode 
> of operation has to be the same across them all.  The current setting 
> would still of course be inherited from the parent by any new threads 
> created with clone(2).

Paul Burton & Matt Fortune know the details and can correct me if I'm
wrong, but I believe the idea is that the mode change will usually be
initiated by the dynamic loader in repsonse to loading a new shared
library with a more specific FPU ABI (but must of course be compatible
with the current mode). As such you must be careful that all threads in
the process change mode so that they can immediately start using the new
dynamically loaded code using the more specific ABI.

https://dmz-portal.mips.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking#10.5.2._Setting_the_FPU_mode

> > (disclaimer: I haven't looked at this gdb patchset in detail as to
> > whether any of below has changed since I worked on it).
> > 
> > (1) Even singles and doubles always overlap one another, as do odd
> > singles and doubles when FR=1 (and FRE=0):
> > 	/* (little endian) */
> > 	union __gdb_builtin_mips_fp64 {
> > 	  float  f32;
> > 	  double f64;
> > 	  int32  i32;
> > 	  int64  i64;
> > 	};
> > 
> > (2) Odd singles when FR=0 (there are no odd doubles):
> > 	union __gdb_builtin_mips_fp32 {
> > 	  float f32;
> > 	  int32_t i32;
> > 	};
> > 
> > (3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
> > 	struct __gdb_builtin_mips_fp96 {
> > 		union {
> > 			double f64;
> > 			int64  i64;
> > 		};
> > 		union {
> > 			float  f32;
> > 			int32  i32;
> > 		};
> > 	};
> > 
> > i.e.
> > 
> > FR=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (2) odd
> >        single:	FEDCBA98
> > 
> > FR=1, FRE=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (1) odd
> >        double:	                0123456789ABCDEF
> >        single:	                        89ABCDEF
> > 
> > FR=1, FRE=1:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210   (Hybrid FPR emulation)
> >  (3) odd
> >        double:	                0123456789ABCDEF
> >        single:	FEDCBA98           (Hybrid FPR emulation)
> > )
> 
>  I haven't got to this part so far and either way will have to think about 
> it yet.  For one as I noted we do want to present vector (paired-single) 
> data with FR=1, FRE=0 in addition to what you quoted above.

Right, I hadn't looked into that.

> This was all 
> implemented in an old MIPS UK patch originally written by Nigel Stephens 
> and included with SDE, which I've never got to upstreaming; have you by 
> any chance based your work on that?

No.

>  As to FR=1, FRE=1 your quoted representation of singles is a software 
> convention only, so I'm not sure offhand how we might represent it in GDB 
> to keep it reasonable; the 96-bit cooked FP register structure does not 
> appeal to me at all TBH, but maybe it's the best we can do after all.

Me neither, but it at least seems to look reasonable from an assembly
debugging point of view. It took some effort to bend GDB to my will to
be honest, especially around mode changes and the remote protocol (since
the remote could change from FR=0 (32-bit FP registers) to FR=1 (64-bit
FP registers even on 32-bit) at almost any time, but I'm really no gdb
expert.

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Matthew Fortune]

Maciej Rozycki <Maciej.Rozycki@imgtec.com> writes:
> On Wed, 12 Oct 2016, James Hogan wrote:
> 
> > >  Then I think it makes sense even more not to create this artificial
> > > API and use the CP1C.FRE/CP1C.NFRE registers instead which do
> > > correspond to what hardware presents to user software.
> >
> > well, barely. Linux at least doesn't enable Config5.UFE or
> > Config5.UFR, since FP mode changes must be done for all threads in the
> > process, so userland can't actually directly access those FCRs either.
> 
>  Hmm, I didn't know that -- what was the reason for this design
> decision?
> Offhand the limitation does not appear necessary to me, each thread has
> its own distinct register set, so it does not appear to me that its mode
> of operation has to be the same across them all.  The current setting
> would still of course be inherited from the parent by any new threads
> created with clone(2).
> 
>  Anyway in that case the presented CP1C registers will have to be read-
> only.

There is no need to support the CP1C.FRE/CP1C.NFRE CP1C.FR/CP1C.NFR
registers as they did not form part of the FR compatibility solution in the
end. They were added to the architecture as part of an earlier plan that
would have involved user-mode code switching mode on a per function basis.

They must not be enabled in Linux as use of them will lead to complete
chaos :-).

> > >  I don't think there is any value in it for GDB, I think all 64-bit
> > > FP registers ought to remain being presented as doubles and pairs of
> > > singles regardless of the mode selected (and also possibly
> > > fixed-point longs and pairs of fixed-point words).  We don't know
> > > what's emulated and what's not after all, and then the contents of
> > > FPRs are not interpreted by GDB itself anyhow except in
> > > user-supplied expressions or assignment requests, which for users'
> > > convenience I think should retain the maximum flexibility possible.
> >
> > Well it technically depends on
> > test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)
> 
>  Sure, but the hardware representation is CP0.Config5.FRE/CP1C.FRE.

The FRE compatibility solution does require GDB to both know about and
modify the user-view of registers as the raw register data cannot be
interpreted by a user unassisted. My memory is a little rusty but I think
this already happens for FR=0 vs FR=1 in that GDB is provided with 32
64-bit registers and must present them as either:

FR=0
====
16 doubles by concatenating the low 32-bits of 2 consecutive registers
to form a double.
32* singles by showing the low 32-bits of each register (*odd registers
not being singles in mips V and below in FR=0.)

FR=1
====
32 doubles
32 singles
(32 128-bit)

FRE=1
=====
32 doubles
32 singles which are stored only in even numbered 64-bit registers with
the low 32-bit being an even numbered single and the high 32-bit being
an odd numbered single.
(32 128-bit)

GDB cannot show the FRE=1 state correctly without knowing which mode the
process is running in. I think this matches with comments from James
below.

> > So it allows gdb to detect Linux's Hybrid FPU mode, and present the fp
> > regs (e.g. f0 or f1) more like below to the user depending on the fp
> > mode, which is surely much more intuitive from an assembly debugging
> > point of view.
> >
> > Its also worth noting that "When software changes the value of this
> > bit [Status.FR], the contents of the floating-point registers are
> > UNPREDICTABLE". I.e. there is no architectural unifying register
> > layout between FR=0 / FR=1, only convention. Linux will I presume
> > intentionally save in old mode and restore in new mode on an fp mode
> > change according to its own convention (such that the valid mode
> > changes don't clobber register state).
> 
>  Well the contents might be unpredictable, but there sure will be some
> and GDB is supposed to present it.

The scheme we have guarantees that no FPU mode switch ever leaves the
register state in an unknown state which is another reason why users
cannot change mode directly. The kernel always performs the mode switch
and this happens with the FPU state in soft-context which is then
restored after the mode switch occurs.

> > (disclaimer: I haven't looked at this gdb patchset in detail as to
> > whether any of below has changed since I worked on it).
> >
> > (1) Even singles and doubles always overlap one another, as do odd
> > singles and doubles when FR=1 (and FRE=0):
> > 	/* (little endian) */
> > 	union __gdb_builtin_mips_fp64 {
> > 	  float  f32;
> > 	  double f64;
> > 	  int32  i32;
> > 	  int64  i64;
> > 	};
> >
> > (2) Odd singles when FR=0 (there are no odd doubles):
> > 	union __gdb_builtin_mips_fp32 {
> > 	  float f32;
> > 	  int32_t i32;
> > 	};
> >
> > (3) Odd singles and doubles when FR=1, FRE=1 don't overlap at all:
> > 	struct __gdb_builtin_mips_fp96 {
> > 		union {
> > 			double f64;
> > 			int64  i64;
> > 		};
> > 		union {
> > 			float  f32;
> > 			int32  i32;
> > 		};
> > 	};
> >
> > i.e.
> >
> > FR=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (2) odd
> >        single:	FEDCBA98
> >
> > FR=1, FRE=0:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210
> >  (1) odd
> >        double:	                0123456789ABCDEF
> >        single:	                        89ABCDEF
> >
> > FR=1, FRE=1:
> >  (1) even
> >        double:	FEDCBA9876543210
> >        single:	        76543210   (Hybrid FPR emulation)
> >  (3) odd
> >        double:	                0123456789ABCDEF
> >        single:	FEDCBA98           (Hybrid FPR emulation)
> > )
> 
>  I haven't got to this part so far and either way will have to think
> about it yet.  For one as I noted we do want to present vector (paired-
> single) data with FR=1, FRE=0 in addition to what you quoted above.
> This was all implemented in an old MIPS UK patch originally written by
> Nigel Stephens and included with SDE, which I've never got to
> upstreaming; have you by any chance based your work on that?
> 
>  As to FR=1, FRE=1 your quoted representation of singles is a software
> convention only, so I'm not sure offhand how we might represent it in
> GDB to keep it reasonable; the 96-bit cooked FP register structure does
> not appeal to me at all TBH, but maybe it's the best we can do after
> all.

The whole FR compatibility scheme has some extremely intricate design
especially when it comes to FRE mode and I believe all tools have to play
along in order to get the end result to be seamless for users. If we
can do any simplification of GDB or the kernel interface then I'm open
to ideas.

A reference to the spec in case anyone doesn't know where it is:

https://dmz-portal.ba.imgtec.org/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking

Note that the spec does not include a definition of the ptrace extension
nor core dump extension (possibly not even designed yet).

While I remember the GDB patchset does need at least checking if not
extra work to cope with the way double precision type data is described
in dwarf for the various compile modes.

Matthew

RE: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

On Thu, 13 Oct 2016, Matthew Fortune wrote:

> >  Hmm, I didn't know that -- what was the reason for this design
> > decision?
> > Offhand the limitation does not appear necessary to me, each thread has
> > its own distinct register set, so it does not appear to me that its mode
> > of operation has to be the same across them all.  The current setting
> > would still of course be inherited from the parent by any new threads
> > created with clone(2).
> > 
> >  Anyway in that case the presented CP1C registers will have to be read-
> > only.
> 
> There is no need to support the CP1C.FRE/CP1C.NFRE CP1C.FR/CP1C.NFR
> registers as they did not form part of the FR compatibility solution in the
> end. They were added to the architecture as part of an earlier plan that
> would have involved user-mode code switching mode on a per function basis.

 I wonder in that case whether we shouldn't simply have a virtual $fre 
register represented.  It could be 1-bit, but I suspect this may not play 
well with the RSP, so we can make it wider at the raw level and only 
truncate it at the cooked level, just as we do with some other registers 
including in particular my recent $fcsr/$fir change.

 If we had such a dedicated virtual $fre, and we decided sometime to let 
the user actually write to it and switch the mode process-wide, then we 
could simply invoke the right prctl(2) call in response to the user's 
ptrace(2) request.

 NB the x86-64 target has such support with PTRACE_ARCH_PRCTL already and 
PTRACE_POKEUSR writes to $fs/$gs also invoke prctl(2).  So perhaps we just 
ought to go ahead and do the same.

> They must not be enabled in Linux as use of them will lead to complete
> chaos :-).

 Hmm, that's a bit too vague for me I am afraid to accept as an argument 
in a technical discussion, but James's note about mode switches needed 
process-wide at DSO load time has convinced me, so unless you want 
something to add beyond that, feel free not to expand here.

> > > Well it technically depends on
> > > test_tsk_thread_flag(target, TIF_HYBRID_FPREGS)
> > 
> >  Sure, but the hardware representation is CP0.Config5.FRE/CP1C.FRE.
> 
> The FRE compatibility solution does require GDB to both know about and
> modify the user-view of registers as the raw register data cannot be
> interpreted by a user unassisted. My memory is a little rusty but I think
> this already happens for FR=0 vs FR=1 in that GDB is provided with 32
> 64-bit registers and must present them as either:

 The change between FR=0 and FR=1 is different as there the raw register 
format changes (and consequently the cooked as well).  Whereas for FRE=0 
vs FRE=1 only the cooked format changes.  Strictly speaking we could 
ignore cooked format changes and only pass through the raw register 
contents, however I agree in that providing the user with the available 
numeric formats the contents of an FPR or an FPR pair can be interpreted 
as is appropriate.

> FR=0
> ====
> 16 doubles by concatenating the low 32-bits of 2 consecutive registers
> to form a double.
> 32* singles by showing the low 32-bits of each register (*odd registers
> not being singles in mips V and below in FR=0.)
> 
> FR=1
> ====
> 32 doubles
> 32 singles
> (32 128-bit)
> 
> FRE=1
> =====
> 32 doubles
> 32 singles which are stored only in even numbered 64-bit registers with
> the low 32-bit being an even numbered single and the high 32-bit being
> an odd numbered single.
> (32 128-bit)
> 
> GDB cannot show the FRE=1 state correctly without knowing which mode the
> process is running in. I think this matches with comments from James
> below.

 Fair enough although arguably the native and FRE=1 states could be 
presented both at a time for the user to pick whichever he founds more 
convenient.  For example we present FR=0 singles for both even and odd FPR 
indices even on hardware which does not support single arithmetics on FPRs 
at odd indices, i.e. all legacy ISAs, up to MIPS IV inclusive.

> >  Well the contents might be unpredictable, but there sure will be some
> > and GDB is supposed to present it.
> 
> The scheme we have guarantees that no FPU mode switch ever leaves the
> register state in an unknown state which is another reason why users
> cannot change mode directly. The kernel always performs the mode switch
> and this happens with the FPU state in soft-context which is then
> restored after the mode switch occurs.

 Noted, though GDB does not actually care -- it serves as a pass-through 
really, or at least that's the intent.

> >  I haven't got to this part so far and either way will have to think
> > about it yet.  For one as I noted we do want to present vector (paired-
> > single) data with FR=1, FRE=0 in addition to what you quoted above.
> > This was all implemented in an old MIPS UK patch originally written by
> > Nigel Stephens and included with SDE, which I've never got to
> > upstreaming; have you by any chance based your work on that?
> > 
> >  As to FR=1, FRE=1 your quoted representation of singles is a software
> > convention only, so I'm not sure offhand how we might represent it in
> > GDB to keep it reasonable; the 96-bit cooked FP register structure does
> > not appeal to me at all TBH, but maybe it's the best we can do after
> > all.
> 
> The whole FR compatibility scheme has some extremely intricate design
> especially when it comes to FRE mode and I believe all tools have to play
> along in order to get the end result to be seamless for users. If we
> can do any simplification of GDB or the kernel interface then I'm open
> to ideas.

 I'll give it some thought yet and encourage everyone interested to do so 
as well.

> A reference to the spec in case anyone doesn't know where it is:
> 
> https://dmz-portal.ba.imgtec.org/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking
> 
> Note that the spec does not include a definition of the ptrace extension
> nor core dump extension (possibly not even designed yet).

 Thanks for the clarification.

> While I remember the GDB patchset does need at least checking if not
> extra work to cope with the way double precision type data is described
> in dwarf for the various compile modes.

 Noted, though with the cooked register view as outlined above it looks 
to me like register references are supposed to match with no extra effort.

 I'll see if it's worthwhile to push the original MIPS UK work first 
though.  It's been in use for over 10 years now, with our own SDE and then 
CodeSourcery toolchains, although regrettably serving non-XML-described 
targets only:

(gdb) show mips abi
The MIPS ABI is set automatically (currently "n64").
(gdb) show endian
The target endianness is set automatically (currently big endian)
(gdb) ptype $f0
type = union __gdb_builtin_type_mips_double_float_reg {
    double d;
    int64_t l;
    float s __attribute__ ((vector_size(2)));
    int32_t w __attribute__ ((vector_size(2)));
}
(gdb) info registers $f0
f0:  0xffffffffffffffff flt: -nan              dbl: -nan
(gdb) print $f0
$2 = {d = -nan(0xfffffffffffff), l = -1, s = {-nan(0x7fffff), -nan(0x7fffff)},
  w = {-1, -1}}
(gdb) set $f0.s[1] = -1
(gdb) info registers $f0
f0:  0xffffffffbf800000 flt: -1                dbl: -nan
(gdb) print $f0
$5 = {d = -nan(0xfffffbf800000), l = -1082130432, s = {-nan(0x7fffff), -1},
  w = {-1, -1082130432}}
(gdb) 

 Questions or comments?

  Maciej

RE: [PATCH 02/24] Add MIPS32 FPU64 GDB target descriptions

[Maciej W. Rozycki]

On Fri, 21 Oct 2016, Maciej W. Rozycki wrote:

>  If we had such a dedicated virtual $fre, and we decided sometime to let 
> the user actually write to it and switch the mode process-wide, then we 
> could simply invoke the right prctl(2) call in response to the user's 
> ptrace(2) request.

 Or we could call it $fp_mode and map it directly to prctl(PR_GET_FP_MODE) 
and prctl(PR_SET_FP_MODE, ...).

  Maciej