Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Mathieu Desnoyers]

* Yao Qi (yao@codesourcery.com) wrote:
> Hi,
> We wrote byte_order in metadata either "le" or "be", according to the
> host endianness.  Today, when I check the ctf specification, I find
> "native" fits our needs well.  However, after I replace "le" with
> "native" in metadata, I got the following error message:
> 
>   [error] get_trace_byte_order: unexpected string "native". Should be "native", "network", "be" or "le".
> 
> This patch teaches babeltrace understand "native" and "network".  With
> this patch applied, the output looks right now.  Let me know if I need
> run testsuite, and how.

your interpretation of what the byte order should accept is based on
section 4.1.3 of the CTF spec, which discusses what byte order can be
accepted by a type.

However, here you are proposing to change the byte order recognised at
the trace declaration level. Unfortunately, we cannot assume anything
about native endianness of the traced machine until we have the
endianness information at the trace declaration level, as the trace can
come from any source.

Possibly that the CTF spec would need to be clarified. Currently, we
need to look at:

7.4 TSDL Examples

trace {
  ...
  byte_order = be OR le;                        /* Endianness (required) */
  ...
};

to understand that only be and le should be expected in the trace
description.

Thoughts ?

Thanks,

Mathieu

> 
> Signed-off-by: Yao Qi <yao@codesourcery.com>
> ---
>  .../ctf/metadata/ctf-visitor-generate-io-struct.c  |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> index 9bcc170..cea2877 100644
> --- a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> +++ b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> @@ -1227,6 +1227,10 @@ int get_trace_byte_order(FILE *fd, int depth, struct ctf_node *unary_expression)
>  		byte_order = BIG_ENDIAN;
>  	else if (!strcmp(unary_expression->u.unary_expression.u.string, "le"))
>  		byte_order = LITTLE_ENDIAN;
> +	else if (!strcmp(unary_expression->u.unary_expression.u.string, "native"))
> +		byte_order = BYTE_ORDER;
> +	else if (!strcmp(unary_expression->u.unary_expression.u.string, "network"))
> +		byte_order = BIG_ENDIAN;
>  	else {
>  		fprintf(fd, "[error] %s: unexpected string \"%s\". Should be \"be\" or \"le\".\n",
>  			__func__, unary_expression->u.unary_expression.u.string);
> -- 
> 1.7.7.6
> 
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Mathieu Desnoyers]

* Mathieu Desnoyers (mathieu.desnoyers@efficios.com) wrote:
> * Yao Qi (yao@codesourcery.com) wrote:
> > Hi,
> > We wrote byte_order in metadata either "le" or "be", according to the
> > host endianness.  Today, when I check the ctf specification, I find
> > "native" fits our needs well.  However, after I replace "le" with
> > "native" in metadata, I got the following error message:
> > 
> >   [error] get_trace_byte_order: unexpected string "native". Should be "native", "network", "be" or "le".
> > 
> > This patch teaches babeltrace understand "native" and "network".  With
> > this patch applied, the output looks right now.  Let me know if I need
> > run testsuite, and how.
> 
> your interpretation of what the byte order should accept is based on
> section 4.1.3 of the CTF spec, which discusses what byte order can be
> accepted by a type.
> 
> However, here you are proposing to change the byte order recognised at
> the trace declaration level. Unfortunately, we cannot assume anything
> about native endianness of the traced machine until we have the
> endianness information at the trace declaration level, as the trace can
> come from any source.
> 
> Possibly that the CTF spec would need to be clarified. Currently, we
> need to look at:
> 
> 7.4 TSDL Examples
> 
> trace {
>   ...
>   byte_order = be OR le;                        /* Endianness (required) */
>   ...
> };
> 
> to understand that only be and le should be expected in the trace
> description.
> 
> Thoughts ?

Clarified with CTF commit:

commit 7e58b8bfd86eea7b2929f2ee37dad7e167e11500
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Fri Mar 15 10:31:37 2013 -0400

    Clarify trace byte_order
    
    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

diff --git a/common-trace-format-specification.txt b/common-trace-format-specification.txt
index 1cd3c3d..4ab3ed9 100644
--- a/common-trace-format-specification.txt
+++ b/common-trace-format-specification.txt
@@ -181,6 +181,11 @@ TSDL meta-data representation:
 
   byte_order = native OR network OR be OR le;	/* network and be are aliases */
 
+Even though the trace description section is not per se a type, for sake
+of clarity, it should be noted that native and network byte orders are
+only allowed within type declaration. The byte_order specified in the
+trace description section only accepts be OR le values.
+
 4.1.4 Size
 
 Type size, in bits, for integers and floats is that returned by "sizeof()" in C

> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Signed-off-by: Yao Qi <yao@codesourcery.com>
> > ---
> >  .../ctf/metadata/ctf-visitor-generate-io-struct.c  |    4 ++++
> >  1 files changed, 4 insertions(+), 0 deletions(-)
> > 
> > diff --git a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> > index 9bcc170..cea2877 100644
> > --- a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> > +++ b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
> > @@ -1227,6 +1227,10 @@ int get_trace_byte_order(FILE *fd, int depth, struct ctf_node *unary_expression)
> >  		byte_order = BIG_ENDIAN;
> >  	else if (!strcmp(unary_expression->u.unary_expression.u.string, "le"))
> >  		byte_order = LITTLE_ENDIAN;
> > +	else if (!strcmp(unary_expression->u.unary_expression.u.string, "native"))
> > +		byte_order = BYTE_ORDER;
> > +	else if (!strcmp(unary_expression->u.unary_expression.u.string, "network"))
> > +		byte_order = BIG_ENDIAN;
> >  	else {
> >  		fprintf(fd, "[error] %s: unexpected string \"%s\". Should be \"be\" or \"le\".\n",
> >  			__func__, unary_expression->u.unary_expression.u.string);
> > -- 
> > 1.7.7.6
> > 
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Yao Qi]

On 03/15/2013 08:07 PM, Mathieu Desnoyers wrote:
> your interpretation of what the byte order should accept is based on
> section 4.1.3 of the CTF spec, which discusses what byte order can be
> accepted by a type.
>
> However, here you are proposing to change the byte order recognised at
> the trace declaration level. Unfortunately, we cannot assume anything
> about native endianness of the traced machine until we have the
> endianness information at the trace declaration level, as the trace can
> come from any source.

It is clear to me now.  Thanks for the clarification.

>
> Possibly that the CTF spec would need to be clarified. Currently, we
> need to look at:

The doc part is fine, but this error message is confusing,

[error] get_trace_byte_order: unexpected string "native". Should be 
"native", "network", "be" or "le".

Looks it was fixed.  When I update my tree, the error message is updated.

I'll give a comment in your comment to CTF spec.

-- 
Yao (齐尧)

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Yao Qi]

On 03/15/2013 10:33 PM, Mathieu Desnoyers wrote:
>     byte_order = native OR network OR be OR le;	/* network and be are aliases */
>
> +Even though the trace description section is not per se a type, for sake
> +of clarity, it should be noted that native and network byte orders are
> +only allowed within type declaration. The byte_order specified in the
> +trace description section only accepts be OR le values.
> +

The first and last sentence below is a little bit confusing to me.

4.1.3 Byte order

By default, the native endianness of the source architecture is used.
....
If not specified, the byte order is native.


1.  "native endianness" appear in the spec for the first time, without a 
precise explanation.  After your explanation in the last mail, I got to 
know it is the endianess described in the trace description.

2.  "source architecture" is not clear to me.

Maybe, we can say something like:

"By default, byte order of a basic type is the byte order described in 
the trace description.  It can be overridden by specifying a 
"byte_order" attribute for a basic type.  Typical use-case is to specify 
the network byte order (big endian: "be") to save data captured from the 
network into the trace without conversion."

I also suggest that we get rid of the term "native" here.  IIUC, we have 
four options for byte order, default, network, le and be.  The latter 
three are easy to understand, and if attribute "byte_order" is not 
specified, it is the default.  On the other hand, "native endianness" 
means the endianness of underlying machine, which is a little different.

I am not good at documentation, nor a native English speaker.  My 
suggestion can be wrong.

-- 
Yao (齐尧)

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Mathieu Desnoyers]

* Yao Qi (yao@codesourcery.com) wrote:
> On 03/15/2013 10:33 PM, Mathieu Desnoyers wrote:
>>     byte_order = native OR network OR be OR le;	/* network and be are aliases */
>>
>> +Even though the trace description section is not per se a type, for sake
>> +of clarity, it should be noted that native and network byte orders are
>> +only allowed within type declaration. The byte_order specified in the
>> +trace description section only accepts be OR le values.
>> +
>
> The first and last sentence below is a little bit confusing to me.
>
> 4.1.3 Byte order
>
> By default, the native endianness of the source architecture is used.
> ....
> If not specified, the byte order is native.
>
>
> 1.  "native endianness" appear in the spec for the first time, without a  
> precise explanation.  After your explanation in the last mail, I got to  
> know it is the endianess described in the trace description.
>
> 2.  "source architecture" is not clear to me.
>
> Maybe, we can say something like:
>
> "By default, byte order of a basic type is the byte order described in  
> the trace description.  It can be overridden by specifying a  
> "byte_order" attribute for a basic type.  Typical use-case is to specify  
> the network byte order (big endian: "be") to save data captured from the  
> network into the trace without conversion."
>
> I also suggest that we get rid of the term "native" here.  IIUC, we have  
> four options for byte order, default, network, le and be.  The latter  
> three are easy to understand, and if attribute "byte_order" is not  
> specified, it is the default.  On the other hand, "native endianness"  
> means the endianness of underlying machine, which is a little different.

I added an explanatory paragraph for "native" in addition to your
recommended changes. We cannot remove an keyword from the specification
lightly, because this would break compatibility.

>
> I am not good at documentation, nor a native English speaker.  My  
> suggestion can be wrong.

Here is the update. Comments are welcome!

Thanks,

Mathieu

commit 237324a20398c5c8eaf92aff0eace177ef8a747f
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Fri Mar 15 11:32:03 2013 -0400

    Clarify byte order section
    
    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

diff --git a/common-trace-format-specification.txt b/common-trace-format-specification.txt
index 4ab3ed9..797dec1 100644
--- a/common-trace-format-specification.txt
+++ b/common-trace-format-specification.txt
@@ -171,20 +171,23 @@ TSDL meta-data attribute representation of a specific alignment:
 
 4.1.3 Byte order
 
-By default, the native endianness of the source architecture is used.
-Byte order can be overridden for a basic type by specifying a "byte_order"
-attribute. Typical use-case is to specify the network byte order (big endian:
-"be") to save data captured from the network into the trace without conversion.
-If not specified, the byte order is native.
+By default, byte order of a basic type is the byte order described in
+the trace description.  It can be overridden by specifying a
+"byte_order" attribute for a basic type.  Typical use-case is to specify
+the network byte order (big endian: "be") to save data captured from the
+network into the trace without conversion.
 
 TSDL meta-data representation:
 
   byte_order = native OR network OR be OR le;	/* network and be are aliases */
 
+The "native" keyword selects the byte order described in the trace
+description. The "network" byte order is an alias for big endian.
+
 Even though the trace description section is not per se a type, for sake
-of clarity, it should be noted that native and network byte orders are
-only allowed within type declaration. The byte_order specified in the
-trace description section only accepts be OR le values.
+of clarity, it should be noted that "native" and "network" byte orders
+are only allowed within type declaration. The byte_order specified in
+the trace description section only accepts "be" or "le" values.
 
 4.1.4 Size
 


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Yao Qi]

On 03/15/2013 11:33 PM, Mathieu Desnoyers wrote:
> I added an explanatory paragraph for "native" in addition to your
> recommended changes. We cannot remove an keyword from the specification
> lightly, because this would break compatibility.

It looks pretty good.  Thanks for doing this.

-- 
Yao (齐尧)

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[PATCH] Teach babeltrace understand "native" and "network" for byte_order

[Yao Qi]

Hi,
We wrote byte_order in metadata either "le" or "be", according to the
host endianness.  Today, when I check the ctf specification, I find
"native" fits our needs well.  However, after I replace "le" with
"native" in metadata, I got the following error message:

  [error] get_trace_byte_order: unexpected string "native". Should be "native", "network", "be" or "le".

This patch teaches babeltrace understand "native" and "network".  With
this patch applied, the output looks right now.  Let me know if I need
run testsuite, and how.

Signed-off-by: Yao Qi <yao@codesourcery.com>
---
 .../ctf/metadata/ctf-visitor-generate-io-struct.c  |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
index 9bcc170..cea2877 100644
--- a/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
+++ b/formats/ctf/metadata/ctf-visitor-generate-io-struct.c
@@ -1227,6 +1227,10 @@ int get_trace_byte_order(FILE *fd, int depth, struct ctf_node *unary_expression)
 		byte_order = BIG_ENDIAN;
 	else if (!strcmp(unary_expression->u.unary_expression.u.string, "le"))
 		byte_order = LITTLE_ENDIAN;
+	else if (!strcmp(unary_expression->u.unary_expression.u.string, "native"))
+		byte_order = BYTE_ORDER;
+	else if (!strcmp(unary_expression->u.unary_expression.u.string, "network"))
+		byte_order = BIG_ENDIAN;
 	else {
 		fprintf(fd, "[error] %s: unexpected string \"%s\". Should be \"be\" or \"le\".\n",
 			__func__, unary_expression->u.unary_expression.u.string);
-- 
1.7.7.6