[Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

Signed-off-by: Leon Alrae <leon.alrae@imgtec.com>
---
Hi,

Continuing the discussion related to extending QEMU's command line with new
argument allowing to pass semi-hosting input arguments to the guest program:
https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg05960.html

This simple patch adds "arg" sub-argument which in my opinion is flexible
enough to satisfy semi-hosting interfaces which allocate buffers at runtime
(so it is possible to pass any number of strings of any length) as well as the
ones which have limited buffer for a single input string.

Highlights:

* arg can appear multiple times in -semihosting-config group of sub-arguments
  and is used to create argc/argv.

* Single string cmdline can be still easily created:
    static void concat_semihosting_args(char *buffer, int buffer_size,
                                        const char *separator)
    {
        int i;
        buffer[0] = '\0';
        for (i = 0; i < semihosting_argc - 1; i++) {
            pstrcat(buffer, buffer_size, semihosting_argv[i]);
            pstrcat(buffer, buffer_size, separator);
        }
        pstrcat(buffer, buffer_size, semihosting_argv[i]);
    }

* argv[0] is set by the first occurence of "arg" rather than taking the same
  string from "-kernel". This might be required in cases where the path is too
  long and semi-hosting app's buffer for the cmdline is fixed and small.

* No altering the parser itself, just traversing the -semihosting-config opts
  and picking all arg strings.

* This patch doesn't fix the need of using double-commas if user want to have
  a comma in the string.

If we agree on this new option then I would like to follow up with a clean up
where we've got a semi-hosting specific structure rather than keep generating
more semihosting_* variables (I skipped it in this patch to avoid generating
clean up related noise in the diff).

Any comments?

Thanks,
Leon
---
 include/sysemu/sysemu.h |  2 ++
 vl.c                    | 18 ++++++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 8a52934..651863f 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -126,6 +126,8 @@ extern int graphic_rotate;
 extern int no_quit;
 extern int no_shutdown;
 extern int semihosting_enabled;
+extern const char **semihosting_argv;
+extern int semihosting_argc;
 extern int old_param;
 extern int boot_menu;
 extern bool boot_strict;
diff --git a/vl.c b/vl.c
index 74c2681..640d9da 100644
--- a/vl.c
+++ b/vl.c
@@ -171,6 +171,8 @@ const char *watchdog;
 QEMUOptionRom option_rom[MAX_OPTION_ROMS];
 int nb_option_roms;
 int semihosting_enabled = 0;
+const char **semihosting_argv;
+int semihosting_argc;
 int old_param = 0;
 const char *qemu_name;
 int alt_grab = 0;
@@ -485,6 +487,9 @@ static QemuOptsList qemu_semihosting_config_opts = {
         }, {
             .name = "target",
             .type = QEMU_OPT_STRING,
+        }, {
+            .name = "arg",
+            .type = QEMU_OPT_STRING,
         },
         { /* end of list */ }
     },
@@ -2727,6 +2732,17 @@ static void set_memory_options(uint64_t *ram_slots, ram_addr_t *maxram_size)
     }
 }
 
+static int add_semihosting_arg(const char *name, const char *val, void *opaque)
+{
+    if (strcmp(name, "arg") == 0) {
+        semihosting_argc++;
+        semihosting_argv = g_realloc(semihosting_argv,
+                                     semihosting_argc * sizeof(void *));
+        semihosting_argv[semihosting_argc - 1] = val;
+    }
+    return 0;
+}
+
 int main(int argc, char **argv, char **envp)
 {
     int i;
@@ -3562,6 +3578,8 @@ int main(int argc, char **argv, char **envp)
                     } else {
                         semihosting_target = SEMIHOSTING_TARGET_AUTO;
                     }
+                    /* Set semihosting argument count and vector */
+                    qemu_opt_foreach(opts, add_semihosting_arg, NULL, 0);
                 } else {
                     fprintf(stderr, "Unsupported semihosting-config %s\n",
                             optarg);
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 01 Apr 2015, at 16:18, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... This simple patch adds "arg" sub-argument which in my opinion is flexible
> enough to satisfy semi-hosting interfaces which allocate buffers at runtime
> (so it is possible to pass any number of strings of any length) as well as the
> ones which have limited buffer for a single input string.

I'm not sure I understand this.

the ARM semihosting interface (see below for the specs) uses a single buffer, usually of 80 bytes for embedded applications, that must be entirely allocated before the call, and in this buffer a single null terminated string is returned, assuming the buffer is large enough, otherwise -1 is returned.

what the semihosting debugger (qemu in our case) must do, is to get this string from somewhere, preferably from the command line, and pass it to the debugged application.


to me, the natural implementation is to use a single string.

can you explain what are the advantages of handling multiple strings that must be later concatenated with spaces separators, to compose the full command line?

in other words, why is it better to define something like 

  "gcm --gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --gtest_repeat=10" 

as 

  -semihosting-config target=native,arg="gcm",arg="--gtest_output=xml:gcm.xml",arg="--gtest_filter=FooTest.*",arg="--gtest_repeat=10"

instead of

  -semihosting-config target=native,cmdline="gcm --gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --gtest_repeat=10"

(the example if from the Google Test infrastructure).

please note that the command line is generally passed to the test from an upper layer, and, in your case, it needs to be parsed, split into substrings, and these substrings be surrounded by ,arg="%s".

for my use case, this solution is not convenient at all.


regards,

Liviu



SYS_GET_CMDLINE (0x15)
Returns the command line used to call the executable, that is, argc and argv.

Entry
-----
On entry, R1 points to a two-word data block to be used for returning the command string and its length:

word 1
	a pointer to a buffer of at least the size specified in word two

word 2
	the length of the buffer in bytes.

Return
------
On exit:

Register R1 points to a two-word data block:
	
word 1
	a pointer to null-terminated string of the command line
word 2
	the length of the string.

The debug agent might impose limits on the maximum length of the string that can be transferred. However, the agent must be able to transfer a command line of at least 80 bytes. 

Register R0 contains an error code:
	• 0 if the call is successful
	• -1 if the call is not successful, for example, because of a communications error.

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Christopher Covington]

Hi Leon,

On 04/01/2015 09:18 AM, Leon Alrae wrote:
> Signed-off-by: Leon Alrae <leon.alrae@imgtec.com>
> ---
> Hi,
> 
> Continuing the discussion related to extending QEMU's command line with new
> argument allowing to pass semi-hosting input arguments to the guest program:
> https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg05960.html
> 
> This simple patch adds "arg" sub-argument which in my opinion is flexible
> enough to satisfy semi-hosting interfaces which allocate buffers at runtime
> (so it is possible to pass any number of strings of any length) as well as the
> ones which have limited buffer for a single input string.
> 
> Highlights:
> 
> * arg can appear multiple times in -semihosting-config group of sub-arguments
>   and is used to create argc/argv.
> 
> * Single string cmdline can be still easily created:
>     static void concat_semihosting_args(char *buffer, int buffer_size,
>                                         const char *separator)
>     {
>         int i;
>         buffer[0] = '\0';
>         for (i = 0; i < semihosting_argc - 1; i++) {
>             pstrcat(buffer, buffer_size, semihosting_argv[i]);
>             pstrcat(buffer, buffer_size, separator);
>         }
>         pstrcat(buffer, buffer_size, semihosting_argv[i]);
>     }
> 
> * argv[0] is set by the first occurence of "arg" rather than taking the same
>   string from "-kernel". This might be required in cases where the path is too
>   long and semi-hosting app's buffer for the cmdline is fixed and small.
> 
> * No altering the parser itself, just traversing the -semihosting-config opts
>   and picking all arg strings.
> 
> * This patch doesn't fix the need of using double-commas if user want to have
>   a comma in the string.
> 
> If we agree on this new option then I would like to follow up with a clean up
> where we've got a semi-hosting specific structure rather than keep generating
> more semihosting_* variables (I skipped it in this patch to avoid generating
> clean up related noise in the diff).

I think the commit message and help text should include some information on
how this is different from -append and what the result is of specifying both
-append foo=a and -semihosting-config arg=foo=b.

Thanks,
Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Matthew Fortune]

Liviu Ionescu <ilg@livius.net> writes:
> > On 01 Apr 2015, at 16:18, Leon Alrae <leon.alrae@imgtec.com> wrote:
> >
> > ... This simple patch adds "arg" sub-argument which in my opinion is
> > flexible enough to satisfy semi-hosting interfaces which allocate
> > buffers at runtime (so it is possible to pass any number of strings of
> > any length) as well as the ones which have limited buffer for a single
> input string.
> 
> I'm not sure I understand this.
> 
> the ARM semihosting interface (see below for the specs) uses a single

Please bear in mind that ARM is not the only architecture with a well
defined semi-hosting interface. MIPS latest semihosting interface (UHI)
can support passing of arguments with spaces for such cases where that
has value. There is also support for obtaining the argument vectors at
runtime and allocating space from a stack for storage which removes
the size limitation. UHI does of course still support the simple case of
an argument vector being pre-allocated with simple argc/argv being passed
in.

> buffer, usually of 80 bytes for embedded applications, that must be
> entirely allocated before the call, and in this buffer a single null
> terminated string is returned, assuming the buffer is large enough,
> otherwise -1 is returned.
> 
> what the semihosting debugger (qemu in our case) must do, is to get this
> string from somewhere, preferably from the command line, and pass it to
> the debugged application.
> 
> 
> to me, the natural implementation is to use a single string.

When the semihosting naturally just supports a single string then that of
course makes sense but I'm sure we can get all use-cases catered for
without adding extra steps for anyone.

> can you explain what are the advantages of handling multiple strings
> that must be later concatenated with spaces separators, to compose the
> full command line?

For the use cases where they are not concatenated together and retain
spaces embedded in an argument. This also avoids carving up the command
line in target code and moves it to the host.

> in other words, why is it better to define something like
> 
>   "gcm --gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --
> gtest_repeat=10"
> 
> as
> 
>   -semihosting-config target=native,arg="gcm",arg="--
> gtest_output=xml:gcm.xml",arg="--gtest_filter=FooTest.*",arg="--
> gtest_repeat=10"
> 
> instead of
> 
>   -semihosting-config target=native,cmdline="gcm --
> gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --gtest_repeat=10"
> 
> (the example if from the Google Test infrastructure).
> 
> please note that the command line is generally passed to the test from
> an upper layer, and, in your case, it needs to be parsed, split into
> substrings, and these substrings be surrounded by ,arg="%s".
> 
> for my use case, this solution is not convenient at all.

I don't see any need to remove the 'cmdline' option but just make the use
of it mutually exclusive to using 'arg'. That seems like it would address
all the issues?

Thanks,
Matthew

> 
> 
> regards,
> 
> Liviu
> 
> 
> 
> SYS_GET_CMDLINE (0x15)
> Returns the command line used to call the executable, that is, argc and
> argv.
> 
> Entry
> -----
> On entry, R1 points to a two-word data block to be used for returning
> the command string and its length:
> 
> word 1
> 	a pointer to a buffer of at least the size specified in word two
> 
> word 2
> 	the length of the buffer in bytes.
> 
> Return
> ------
> On exit:
> 
> Register R1 points to a two-word data block:
> 
> word 1
> 	a pointer to null-terminated string of the command line word 2
> 	the length of the string.
> 
> The debug agent might impose limits on the maximum length of the string
> that can be transferred. However, the agent must be able to transfer a
> command line of at least 80 bytes.
> 
> Register R0 contains an error code:
> 	• 0 if the call is successful
> 	• -1 if the call is not successful, for example, because of a
> communications error.
> 
> 

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 01 Apr 2015, at 18:45, Christopher Covington <cov@codeaurora.org> wrote:
> 
> I think the commit message and help text should include some information on
> how this is different from -append and what the result is of specifying both
> -append foo=a and -semihosting-config arg=foo=b.

if I remember right (did not check the sources recently), -append is used to add args to the kernel, a mechanism that always passes the long kernel absolute path as argv[0].

for semihosting cases the command line has a different purpose, and must not hardwire argv[0] to the kernel path.

in my use case I actually do not use -kernel/-append at all, I load the image via the gdb server, as for any existing bare metal board. passing semihosting options via -semihosting-config is the natural solution.

I think that configuring the kernel and configuring semihosting are two different things, and should have different command line options.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 01/04/2015 16:21, Liviu Ionescu wrote:
> 
>> On 01 Apr 2015, at 16:18, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>
>> ... This simple patch adds "arg" sub-argument which in my opinion is flexible
>> enough to satisfy semi-hosting interfaces which allocate buffers at runtime
>> (so it is possible to pass any number of strings of any length) as well as the
>> ones which have limited buffer for a single input string.
> 
> I'm not sure I understand this.
> 
> the ARM semihosting interface (see below for the specs) uses a single buffer, usually of 80 bytes for embedded applications, that must be entirely allocated before the call, and in this buffer a single null terminated string is returned, assuming the buffer is large enough, otherwise -1 is returned.
> 
> what the semihosting debugger (qemu in our case) must do, is to get this string from somewhere, preferably from the command line, and pass it to the debugged application.
> 
> 
> to me, the natural implementation is to use a single string.
> 
> can you explain what are the advantages of handling multiple strings that must be later concatenated with spaces separators, to compose the full command line?
> 
> in other words, why is it better to define something like 
> 
>   "gcm --gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --gtest_repeat=10" 
> 
> as 
> 
>   -semihosting-config target=native,arg="gcm",arg="--gtest_output=xml:gcm.xml",arg="--gtest_filter=FooTest.*",arg="--gtest_repeat=10"
> 
> instead of
> 
>   -semihosting-config target=native,cmdline="gcm --gtest_output=xml:gcm.xml --gtest_filter=FooTest.* --gtest_repeat=10"

The ARM semihosting supports SYS_GET_CMDLINE syscall which returns a
single string containing all arguments (whitespace separated I presume).
The UHI semihosting supports Argn syscall which returns single argument,
and Argn is called multiple times to build argv (each argument can
contain whitespaces).

I believe having just one flexible command line option in QEMU
satisfying these two cases would be ideal. But if arg doesn't suit your
needs then as Matthew already mentioned we probably should go for having
cmdline and arg mutually exclusive.

Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 02 Apr 2015, at 10:56, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... The UHI semihosting supports Argn syscall which returns single argument,
> and Argn is called multiple times to build argv (each argument can
> contain whitespaces).

ok, I don't want to comment too much on UHI design, since I'm not familiar with; at first sight it seems an improvement, but at second sight, from the user perspective, with your implementation, it looks overkill; all shells I know accept a single command line string, possibly with quotes if substrings need to preserve the separator spaces, and internally the shells themselves split the string into substrings and pass them to the application as (argc, argv).

in my opinion, if UHI semihosting supports single argument Argn syscalls, then it is the responsibility of the debugger (qemu in our case) to parse the command line and to provide the individual substrings in the desired format, and not ask the user to do this task manually.

the ARM semihosting case, being more primitive in this respect, requires to run the code to split the line inside the debugged application.

as an example, the projects generated by my GNU ARM Eclipse templates use

https://sourceforge.net/p/micro-os-plus/iii/ci/master/tree/packages/portable/newlib.pack/src/newlib/_syscalls.c

which, at line 419, has an __initialize_args() function.

this is simplistic, to meet the bare metal limitations, but seems functional.

I guess something similar, but more elaborate, can be added to qemu for UHI semihosting.

> I believe having just one flexible command line option in QEMU
> satisfying these two cases would be ideal. But if arg doesn't suit your
> needs then as Matthew already mentioned we probably should go for having
> cmdline and arg mutually exclusive.

from the user point of view, I still do not see the advantages of multiple arg=, it is easier to use quotes in a complete command line, like in any shell; so my vote still goes to cmdline=.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 02/04/2015 09:29, Liviu Ionescu wrote:
> 
>> On 02 Apr 2015, at 10:56, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>
>> ... The UHI semihosting supports Argn syscall which returns single argument,
>> and Argn is called multiple times to build argv (each argument can
>> contain whitespaces).
> 
> ok, I don't want to comment too much on UHI design, since I'm not familiar with; at first sight it seems an improvement, but at second sight, from the user perspective, with your implementation, it looks overkill; all shells I know accept a single command line string, possibly with quotes if substrings need to preserve the separator spaces, and internally the shells themselves split the string into substrings and pass them to the application as (argc, argv).

How would you pass arguments containing whitespaces via
-semihosting-config cmdline without having to reinvent shell within QEMU?

> 
> in my opinion, if UHI semihosting supports single argument Argn syscalls, then it is the responsibility of the debugger (qemu in our case) to parse the command line and to provide the individual substrings in the desired format, and not ask the user to do this task manually.

Couldn't the user have a wrapper around QEMU doing it for him?

Leon

> 
> the ARM semihosting case, being more primitive in this respect, requires to run the code to split the line inside the debugged application.
> 
> as an example, the projects generated by my GNU ARM Eclipse templates use
> 
> https://sourceforge.net/p/micro-os-plus/iii/ci/master/tree/packages/portable/newlib.pack/src/newlib/_syscalls.c
> 
> which, at line 419, has an __initialize_args() function.
> 
> this is simplistic, to meet the bare metal limitations, but seems functional.
> 
> I guess something similar, but more elaborate, can be added to qemu for UHI semihosting.
> 
>> I believe having just one flexible command line option in QEMU
>> satisfying these two cases would be ideal. But if arg doesn't suit your
>> needs then as Matthew already mentioned we probably should go for having
>> cmdline and arg mutually exclusive.
> 
> from the user point of view, I still do not see the advantages of multiple arg=, it is easier to use quotes in a complete command line, like in any shell; so my vote still goes to cmdline=.
> 
> 
> regards,
> 
> Liviu
> 

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 02/04/2015 10:14, Leon Alrae wrote:
> On 02/04/2015 09:29, Liviu Ionescu wrote:
>>
>>> On 02 Apr 2015, at 10:56, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>>
>>> ... The UHI semihosting supports Argn syscall which returns single argument,
>>> and Argn is called multiple times to build argv (each argument can
>>> contain whitespaces).
>>
>> ok, I don't want to comment too much on UHI design, since I'm not familiar with; at first sight it seems an improvement, but at second sight, from the user perspective, with your implementation, it looks overkill; all shells I know accept a single command line string, possibly with quotes if substrings need to preserve the separator spaces, and internally the shells themselves split the string into substrings and pass them to the application as (argc, argv).
> 
> How would you pass arguments containing whitespaces via
> -semihosting-config cmdline without having to reinvent shell within QEMU?

I missed the fact that you already provided the implementation for this
below... sorry. But still, the main point of arg is to keep it simple
and avoid dealing with implementing shell within QEMU. It is also
flexible, thus if a user finds it annoying to set each arg separately he
can have a wrapper responsible for parsing the command line and
translating into QEMU command line options.

Leon

> 
>>
>> in my opinion, if UHI semihosting supports single argument Argn syscalls, then it is the responsibility of the debugger (qemu in our case) to parse the command line and to provide the individual substrings in the desired format, and not ask the user to do this task manually.
> 
> Couldn't the user have a wrapper around QEMU doing it for him?
> 
> Leon
> 
>>
>> the ARM semihosting case, being more primitive in this respect, requires to run the code to split the line inside the debugged application.
>>
>> as an example, the projects generated by my GNU ARM Eclipse templates use
>>
>> https://sourceforge.net/p/micro-os-plus/iii/ci/master/tree/packages/portable/newlib.pack/src/newlib/_syscalls.c
>>
>> which, at line 419, has an __initialize_args() function.
>>
>> this is simplistic, to meet the bare metal limitations, but seems functional.
>>
>> I guess something similar, but more elaborate, can be added to qemu for UHI semihosting.
>>
>>> I believe having just one flexible command line option in QEMU
>>> satisfying these two cases would be ideal. But if arg doesn't suit your
>>> needs then as Matthew already mentioned we probably should go for having
>>> cmdline and arg mutually exclusive.
>>
>> from the user point of view, I still do not see the advantages of multiple arg=, it is easier to use quotes in a complete command line, like in any shell; so my vote still goes to cmdline=.
>>
>>
>> regards,
>>
>> Liviu
>>
> 

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 02 Apr 2015, at 13:36, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
>> How would you pass arguments containing whitespaces via
>> -semihosting-config cmdline without having to reinvent shell within QEMU?
> 
> I missed the fact that you already provided the implementation for this
> below...


for completeness:

ilg-mbp:gnuarmeclipse-qemu.git ilg$ "/Applications/GNU ARM Eclipse/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse" -verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config enable=on,target=native,cmdline='n "1 a" 2 3'

GNU ARM Eclipse QEMU v2.2.91 (qemu-system-gnuarmeclipse).
Board: 'STM32-H103' (Olimex Header Board for STM32F103RBT6 (Experimental)).
Device: 'STM32F103RB' (cortex-m3, MPU), Flash: 128 KB, RAM: 20 KB.
Command line: 'n "1 a" 2 3' (11 bytes).
GDB Server listening on: 'tcp::1234'...
... connection accepted from 127.0.0.1.

Execute 'mon system_reset'.
main(argc=4, argv=["n", "1 a", "2", "3"]);
Hello ARM World!

(note: while testing this I encountered and fixed a small bug in my args parser)

> ... But still, the main point of arg is to keep it simple
> ... can have a wrapper responsible for parsing the command line and
> translating into QEMU command line options.

sure, in Unix any such trick is feasible, but what do you do with the unfortunate Windows users?


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Matthew Fortune]

Liviu Ionescu <ilg@livius.net> writes:
> > On 02 Apr 2015, at 13:36, Leon Alrae <leon.alrae@imgtec.com> wrote:
> >
> >> How would you pass arguments containing whitespaces via
> >> -semihosting-config cmdline without having to reinvent shell within
> QEMU?
> >
> > I missed the fact that you already provided the implementation for
> > this below...
> 
> 
> for completeness:
> 
> ilg-mbp:gnuarmeclipse-qemu.git ilg$ "/Applications/GNU ARM
> Eclipse/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse" -
> verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config
> enable=on,target=native,cmdline='n "1 a" 2 3'

I see here that you have switched quotes because you know that you are
providing a double quoted argument within the string. The root of all
argument passing issues tends to boil down to how to quote and or escape
characters appropriately. Because you know the arguments you can quote
correctly but for an unknown user-provided set of arguments (where they
do not know how their arguments will flow down to an application) then
the appropriate quoting and escaping becomes harder.

The problem characters are obviously single and double quotes. Having
different quoting rules for any of the layers between a user and
the emulated program is always going to cause some confusion so when
quoting is necessary I have always found that following the standard
argument passing rules of the native application is the least
problematic.

qemu-system... -semihosting-config "arg=foo bar" -semihosting-config "arg=second"

This should give an argv of ["foo bar", "second"]

I'd propose that the cmdline option is retained for those cases that
do not need to cope with spaces (and also can avoid commas in the
cmdline or have them automatically doubled) and introduce arg= as a
more flexible case such that both commas and spaces are naturally
handled (the arg= just needs chopping off the start).

This could be implemented by saying that as soon as the parser sees
arg= then the rest of the string is one argument. The comma issue for
cmdline could be avoided like this too by mandating that cmdline is
the last part of the config options.

Just my thoughts :-)

Matthew

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 02 Apr 2015, at 17:27, Matthew Fortune <Matthew.Fortune@imgtec.com> wrote:
> 
> Liviu Ionescu <ilg@livius.net> writes:
>> for completeness:
>> 
>> ilg-mbp:gnuarmeclipse-qemu.git ilg$ "/Applications/GNU ARM
>> Eclipse/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse" -
>> verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config
>> enable=on,target=native,cmdline='n "1 a" 2 3'
> 
> I see here that you have switched quotes because you know that you are
> providing a double quoted argument within the string.

my code works both ways symmetrically:

ilg-mbp:~ ilg$ "/Applications/GNU ARM Eclipse"/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse -verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config enable=on,target=native,cmdline="name 1 '2 a' 3"

GNU ARM Eclipse QEMU v2.2.91 (qemu-system-gnuarmeclipse).
Board: 'STM32-H103' (Olimex Header Board for STM32F103RBT6 (Experimental)).
Device: 'STM32F103RB' (cortex-m3, MPU), Flash: 128 KB, RAM: 20 KB.
Command line: 'name 1 '2 a' 3' (14 bytes).
GDB Server listening on: 'tcp::1234'...
... connection accepted from 127.0.0.1.

Execute 'mon system_reset'.
main(argc=4, argv=["name", "1", "2 a", "3"]);
Hello ARM World!

in other words, after being delivered by SYS_GET_CMDLINE to the application, the startup parser uses either single or double quotes to split the string into arguments, similarly to a real life shell.

> The root of all
> argument passing issues tends to boil down to how to quote and or escape
> characters appropriately. Because you know the arguments you can quote
> correctly but for an unknown user-provided set of arguments (where they
> do not know how their arguments will flow down to an application) then
> the appropriate quoting and escaping becomes harder.
> 
> The problem characters are obviously single and double quotes. Having
> different quoting rules for any of the layers between a user and
> the emulated program is always going to cause some confusion so when
> quoting is necessary I have always found that following the standard
> argument passing rules of the native application is the least
> problematic.

I guess you are generally right, but in this case any quoting is perfectly fine.

> qemu-system... -semihosting-config "arg=foo bar" -semihosting-config "arg=second"
> 
> This should give an argv of ["foo bar", "second"]

yes, but for the casual user the above syntax is already not natural and even more confusing: should I use "arg=foo bar" or arg="foo bar"? should I repeat -semihosting-config for each argument, or I can group all together (-semihosting-config arg="foo bar",arg="second")?


if for your use cases this syntax is acceptable, ok, go for it.


my first use of qemu is as a background GDB server started by an Eclipse plug-in. the plug-in will have a single edit field, where the user can enter the semihosting command line args. for args that include spaces, the user should use either single or double quotes, as in any other field.

my preferred implemention for this would be the original -semihosting-cmdline "some string".

both proposed implementations (-semihosting-config cmdline="some-string") and even worse for the multiple (-semihosting-config arg=some-string), are way more complicated to use in my Eclipse plug-in, requiring special precautions for passing single/double quotes, commas and possibly equals.

with the -semihosting-cmdline some-string, the only thing I have to do is to pass [-semihosting-cmdline] in one string array element, and the unmodified [some-string], regardless the quoting used, as the next string array element, and call exec().

given the above reasons, there are good chances that in my qemu fork (GNU ARM Eclipse QEMU) I'll have to use the separate -semihosting-cmdline option; if you'll accept my patches in the main trunk, ok, if not... no problem; 

as a conclusion, if you like arg=, go for it, but just be aware that this solution does not cover all use cases properly.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

[-- Attachment #1: Type: text/plain, Size: 2720 bytes --]

hi,

as promised, I proceeded with the implementation of the GNU ARM Eclipse QEMU plug-in and so I could run more tests with semihosting command line option.

the current first plug-in configuration page looks like this:


for the above configuration, the string array used to start QEMU (one arg per line) is:

/Users/ilg/Work/qemu/build/osx/qemu/gnuarmeclipse-softmmu/qemu-system-gnuarmeclipse 
-verbose 
-machine 
STM32-H103 
-gdb 
tcp::1234 
-semihosting-config 
enable=on,target=native 
-semihosting-cmdline 
cm 1 "2 a" '3 b' 4=c 5,d 6

the QEMU console shows:

GNU ARM Eclipse QEMU v2.2.92 (qemu-system-gnuarmeclipse).
Board: 'STM32-H103' (Olimex Header Board for STM32F103RBT6 (Experimental)).
Device: 'STM32F103RB' (cortex-m3, MPU), Flash: 128 KB, RAM: 20 KB.
Command line: 'cm 1 "2 a" '3 b' 4=c 5,d 6' (26 bytes).
GDB Server listening on: 'tcp::1234'...
... connection accepted from 127.0.0.1.

Execute 'mon system_reset'.

main(argc=7, argv=["cm", "1", "2 a", "3 b", "4=c", "5,d", "6"]);
Hello ARM World!

---

attempts to pack the above command line (I agree that it is a bit extreme), as "-semihosting-config enable=on,target=native,cmdline=???" failed, since I could not find a reliable way to quote the string, so that the qemu parser to accept it.

I don't claim that this cannot be done, probably someone with a better understanding of the qemu parser will find a way, but it might be not very simple, not to say intuitive.


I also thought about passing the above string using the "-semihosting-config arg=???" method, and, with some effort, it might be possible to split the original string into substrings and generate one option, but it requires some effort, definitely more than when using "-semihosting-cmdline", when the string is simply passed as-is. in my java code, as in any programming language, this can be done, but in a simple script this is more difficult.

---

as a conclusion, my final vote goes to a separate "-semihosting-cmdline ???" option. for ARM, the string can be passed directly to SYS_GET_CMDLINE; for UHI hosts, the string can be parsed and substrings provided in an iterative way.

the only 'cons' is that this is a top-level command, and Peter suggested to limit their number, but, in my opinion, a certain degree of flexibility is always required, and in this case the gain of not having a top-level command is not worth the complexity introduced by the alternatives.

my code is available from https://sourceforge.net/p/gnuarmeclipse/qemu/ci/gnuarmeclipse-dev/tree/ (mind the gnuarmeclipse-dev branch).

until we find a better idea, I'll probably stick with this solution.

Peter?


regards,

Liviu







[-- Attachment #2.1: Type: text/html, Size: 4098 bytes --]

[-- Attachment #2.2: Screen Shot 2015-04-03 at 17.56.20.png --]
[-- Type: image/png, Size: 179558 bytes --]

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 01 Apr 2015, at 16:18, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... I would like to follow up with a clean up
> where we've got a semi-hosting specific structure rather than keep generating
> more semihosting_* variables.

I checked the differences between my fork and the master branch (SourceTree for Mac is a great git tool!), and I can summarise my changes:

- in qemu-options.hx I added -semihosting_cmdline

	DEF("semihosting-cmdline", HAS_ARG, QEMU_OPTION_semihosting_cmdline,
    	"-semihosting-cmdline [string]   semihosting command line\n",
	QEMU_ARCH_ARM | QEMU_ARCH_M68K | QEMU_ARCH_XTENSA | QEMU_ARCH_LM32)

- in vl.c I defined I defined qemu_semihosting_cmdline_opts

static QemuOptsList qemu_semihosting_cmdline_opts = {
    .name = "semihosting-cmdline",
    .implied_opt_name = "cmdline",
    .merge_lists = true,
    .head = QTAILQ_HEAD_INITIALIZER(qemu_semihosting_cmdline_opts.head),
    .desc = {
         { /* end of list */ }
    },
};

- in vl.c I identify the "semihosting-cmdline", get the next option argument and store it in the cmdline option field:

            case QEMU_OPTION_semihosting_cmdline:
                opts = qemu_opts_parse(qemu_find_opts("semihosting-cmdline"),
                                           optarg, 0);
                if (opts != NULL) {
                    Error *local_err = NULL;
                    qemu_opt_set(opts, "cmdline", optarg, &local_err);
                }

- in target-arm/arm-semi.c, the initial code computed the semihosting command line by concatenating the kernel path and the kernel cmdline. the patched version first checks if -semihosting-cmdline is not null, and uses it as-is, otherwise it falls back to the kernel path and kernel cmdline.

the intention was to remain compatible with existing configurations, and I guess it was met.

access to semihosting cmdline is simple:

	opts = qemu_opts_find(qemu_find_opts("semihosting-cmdline"), NULL);
	cmdline = qemu_opt_get(opts, "cmdline");

https://sourceforge.net/p/gnuarmeclipse/qemu/ci/gnuarmeclipse-dev/tree/target-arm/arm-semi.c

- in hw/arm/cortexm.c, the equivalent of armv7m.c, the env->boot_info pointer must be initialised, otherwise the arm-semi.c code fails.

    /* Fill-in a minimalistic boot info, required for semihosting.  */
    cortexm_binfo.kernel_cmdline = kernel_cmdline;
    cortexm_binfo.kernel_filename = machine->kernel_cmdline;
    

    env->boot_info = &cortexm_binfo;


you can use these code snippets, or I can pack them in a patch, if the semihosting-cmdline solution is accepted in the master branch.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 02/04/2015 17:47, Liviu Ionescu wrote:
> 
>> On 02 Apr 2015, at 17:27, Matthew Fortune <Matthew.Fortune@imgtec.com> wrote:
>>
>> Liviu Ionescu <ilg@livius.net> writes:
>>> for completeness:
>>>
>>> ilg-mbp:gnuarmeclipse-qemu.git ilg$ "/Applications/GNU ARM
>>> Eclipse/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse" -
>>> verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config
>>> enable=on,target=native,cmdline='n "1 a" 2 3'
>>
>> I see here that you have switched quotes because you know that you are
>> providing a double quoted argument within the string.
> 
> my code works both ways symmetrically:
> 
> ilg-mbp:~ ilg$ "/Applications/GNU ARM Eclipse"/QEMU/2.2.91-201504021111-dev/bin/qemu-system-gnuarmeclipse -verbose -machine STM32-H103 -gdb tcp::1234 -semihosting-config enable=on,target=native,cmdline="name 1 '2 a' 3"
> 
> GNU ARM Eclipse QEMU v2.2.91 (qemu-system-gnuarmeclipse).
> Board: 'STM32-H103' (Olimex Header Board for STM32F103RBT6 (Experimental)).
> Device: 'STM32F103RB' (cortex-m3, MPU), Flash: 128 KB, RAM: 20 KB.
> Command line: 'name 1 '2 a' 3' (14 bytes).
> GDB Server listening on: 'tcp::1234'...
> ... connection accepted from 127.0.0.1.
> 
> Execute 'mon system_reset'.
> main(argc=4, argv=["name", "1", "2 a", "3"]);
> Hello ARM World!
> 
> in other words, after being delivered by SYS_GET_CMDLINE to the application, the startup parser uses either single or double quotes to split the string into arguments, similarly to a real life shell.
> 
>> The root of all
>> argument passing issues tends to boil down to how to quote and or escape
>> characters appropriately. Because you know the arguments you can quote
>> correctly but for an unknown user-provided set of arguments (where they
>> do not know how their arguments will flow down to an application) then
>> the appropriate quoting and escaping becomes harder.
>>
>> The problem characters are obviously single and double quotes. Having
>> different quoting rules for any of the layers between a user and
>> the emulated program is always going to cause some confusion so when
>> quoting is necessary I have always found that following the standard
>> argument passing rules of the native application is the least
>> problematic.
> 
> I guess you are generally right, but in this case any quoting is perfectly fine.
> 
>> qemu-system... -semihosting-config "arg=foo bar" -semihosting-config "arg=second"
>>
>> This should give an argv of ["foo bar", "second"]
> 
> yes, but for the casual user the above syntax is already not natural and even more confusing: should I use "arg=foo bar" or arg="foo bar"? should I repeat -semihosting-config for each argument, or I can group all together (-semihosting-config arg="foo bar",arg="second")?
> 
> 
> if for your use cases this syntax is acceptable, ok, go for it.

Hmm... the -semihosting-config group would be indeed unusual comparing
to the other groups, but on the other hand this would solve also the
double-comma problem. For simple cases the casual user can just stick to
the cmdline.

> 
> 
> my first use of qemu is as a background GDB server started by an Eclipse plug-in. the plug-in will have a single edit field, where the user can enter the semihosting command line args. for args that include spaces, the user should use either single or double quotes, as in any other field.
> 
> my preferred implemention for this would be the original -semihosting-cmdline "some string".
> 
> both proposed implementations (-semihosting-config cmdline="some-string") and even worse for the multiple (-semihosting-config arg=some-string), are way more complicated to use in my Eclipse plug-in, requiring special precautions for passing single/double quotes, commas and possibly equals.
> 
> with the -semihosting-cmdline some-string, the only thing I have to do is to pass [-semihosting-cmdline] in one string array element, and the unmodified [some-string], regardless the quoting used, as the next string array element, and call exec().
> 
> given the above reasons, there are good chances that in my qemu fork (GNU ARM Eclipse QEMU) I'll have to use the separate -semihosting-cmdline option; if you'll accept my patches in the main trunk, ok, if not... no problem; 
> 
> as a conclusion, if you like arg=, go for it, but just be aware that this solution does not cover all use cases properly.

I wasn't focussing on a specific use case, I just thought that having
only one flexible command line option used for passing input arguments
regardless of semi-hosting interface would be ideal. I do understand
however that in your particular case cmdline is more convenient, thus I
personally don’t mind having both options: the user-friendly cmdline and
more flexible arg.

Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 08 Apr 2015, at 19:20, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... I do understand
> however that in your particular case cmdline is more convenient, thus I
> personally don’t mind having both options: the user-friendly cmdline and
> more flexible arg.

I was a bit optimistic with the first implementation of "--semihosting-cmdline string", since this approach complicates things when calling the emulator from a script.

a normal script might do something like

~~~
# identify own args and use shift to 'eat' them
exec qemu ... --semihosting-cmdline $@
~~~

the first thought was to transform the $@ array into a single string with quotes, like "$@", but this does not work properly if the arguments already include spaces.

so, after long considerations, I had to revert to the very first idea, to use a variable number args option, which, obviously, cannot be placed anywhere else but at the end.

the new implementation is available from:

  https://sourceforge.net/p/gnuarmeclipse/qemu/ci/gnuarmeclipse-dev/tree/vl.c

the option is identified in the first parsing loop, args are stored and argc is diminished accordingly, for the second pass to work without any changes.

one more thing to note is the concatenate_semihosting_cmdline() function, which is not as simple as Leon suggested in the original post, since it has to add quotes to arguments containing spaces.

I updated the Eclipse plug-in to use this mechanism and there were no problems. passing the args from a script is also pretty easy, so to me this looks like the right solution.


any comments?


Livius

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

Peter/Leon,

I'm planning for my future releases, and, to avoid incompatible versions, I would like to help with the semihosting configuration issue, to speed up things. is there any chance to close this issue in the near future?

we currently have two solutions:

- multiple: --semihosting-config arg="..." 
- a single option, placed at the end, and followed by any number of arguments: --semihosting-cmdline $@ \n

both can generate the array of args required by UHI and both can pass args with spaces.

the second one is more scripting friendly, and is somehow in the spirit of other unix programs that need to pass such arguments.

please let me know how can I be of any help.


regards,

Liviu



> On 14 Apr 2015, at 20:42, Liviu Ionescu <ilg@livius.net> wrote:
> 
> 
>> On 08 Apr 2015, at 19:20, Leon Alrae <leon.alrae@imgtec.com> wrote:
>> 
>> ... I do understand
>> however that in your particular case cmdline is more convenient, thus I
>> personally don’t mind having both options: the user-friendly cmdline and
>> more flexible arg.
> 
> I was a bit optimistic with the first implementation of "--semihosting-cmdline string", since this approach complicates things when calling the emulator from a script.
> 
> a normal script might do something like
> 
> ~~~
> # identify own args and use shift to 'eat' them
> exec qemu ... --semihosting-cmdline $@
> ~~~
> 
> the first thought was to transform the $@ array into a single string with quotes, like "$@", but this does not work properly if the arguments already include spaces.
> 
> so, after long considerations, I had to revert to the very first idea, to use a variable number args option, which, obviously, cannot be placed anywhere else but at the end.
> 
> the new implementation is available from:
> 
>  https://sourceforge.net/p/gnuarmeclipse/qemu/ci/gnuarmeclipse-dev/tree/vl.c
> 
> the option is identified in the first parsing loop, args are stored and argc is diminished accordingly, for the second pass to work without any changes.
> 
> one more thing to note is the concatenate_semihosting_cmdline() function, which is not as simple as Leon suggested in the original post, since it has to add quotes to arguments containing spaces.
> 
> I updated the Eclipse plug-in to use this mechanism and there were no problems. passing the args from a script is also pretty easy, so to me this looks like the right solution.
> 
> 
> any comments?
> 
> 
> Livius
> 
> 
> 
> 
> 

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 15/04/2015 10:09, Liviu Ionescu wrote:
> Peter/Leon,
> 
> I'm planning for my future releases, and, to avoid incompatible versions, I would like to help with the semihosting configuration issue, to speed up things. is there any chance to close this issue in the near future?

I'm hoping so :) I'm holding up UHI patches respin until we resolve this.

> 
> we currently have two solutions:
> 
> - multiple: --semihosting-config arg="..." 
> - a single option, placed at the end, and followed by any number of arguments: --semihosting-cmdline $@ \n
> 
> both can generate the array of args required by UHI and both can pass args with spaces.
> 
> the second one is more scripting friendly, and is somehow in the spirit of other unix programs that need to pass such arguments.

I don't have a strong preference here, but I think I would be leaning
towards the second one as it just looks better. On the other hand it
reserves the last position in the QEMU command line, which I'm not sure
if can become a problem at some point?

Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Matthew Fortune]

Leon Alrae <Leon.Alrae@imgtec.com> writes:
> On 15/04/2015 10:09, Liviu Ionescu wrote:
> > Peter/Leon,
> >
> > I'm planning for my future releases, and, to avoid incompatible
> versions, I would like to help with the semihosting configuration issue,
> to speed up things. is there any chance to close this issue in the near
> future?
> 
> I'm hoping so :) I'm holding up UHI patches respin until we resolve
> this.
> 
> >
> > we currently have two solutions:
> >
> > - multiple: --semihosting-config arg="..."
> > - a single option, placed at the end, and followed by any number of
> > arguments: --semihosting-cmdline $@ \n
> >
> > both can generate the array of args required by UHI and both can pass
> args with spaces.
> >
> > the second one is more scripting friendly, and is somehow in the
> spirit of other unix programs that need to pass such arguments.
> 
> I don't have a strong preference here, but I think I would be leaning
> towards the second one as it just looks better. On the other hand it
> reserves the last position in the QEMU command line, which I'm not sure
> if can become a problem at some point?

For what it's worth, I like the idea of using the end of the command line
for this as it makes a qemu wrapper really simple. Isn't this also how the
user-mode simulator works for qemu, consistency between these wouldn't
be a bad thing.

Thanks,
Matthew

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 15 Apr 2015, at 14:53, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> On 15/04/2015 10:09, Liviu Ionescu wrote:
>> Peter/Leon,
>> 
>> I'm planning for my future releases, and, to avoid incompatible versions, I would like to help with the semihosting configuration issue, to speed up things. is there any chance to close this issue in the near future?
> 
> I'm hoping so :) I'm holding up UHI patches respin until we resolve this.

how do you suggest we should proceed?

>> we currently have two solutions:
>> 
>> - multiple: --semihosting-config arg="..." 
>> - a single option, placed at the end, and followed by any number of arguments: --semihosting-cmdline $@ \n
>> 
>> both can generate the array of args required by UHI and both can pass args with spaces.
>> 
>> the second one is more scripting friendly, and is somehow in the spirit of other unix programs that need to pass such arguments.
> 
> I don't have a strong preference here, but I think I would be leaning
> towards the second one as it just looks better. On the other hand it
> reserves the last position in the QEMU command line, which I'm not sure
> if can become a problem at some point?

do you have anything specific in mind? I don't know qemu very well, but apart from setting semihosting and kernel args, I doubt there are other use cases to require variable number of arguments. 


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 15/04/2015 13:06, Liviu Ionescu wrote:
> 
>> On 15 Apr 2015, at 14:53, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>
>> On 15/04/2015 10:09, Liviu Ionescu wrote:
>>> Peter/Leon,
>>>
>>> I'm planning for my future releases, and, to avoid incompatible versions, I would like to help with the semihosting configuration issue, to speed up things. is there any chance to close this issue in the near future?
>>
>> I'm hoping so :) I'm holding up UHI patches respin until we resolve this.
> 
> how do you suggest we should proceed?

If there aren't any objections for the proposed solution, then let's
just cook up the patch. It would be nice to get ACK/NACK from Peter as
initially it was suggested to avoid adding top level arguments.
But I think there were reasonable enough arguments for having top level
--semihosting-cmdline placed at the end of command line.

> 
>>> we currently have two solutions:
>>>
>>> - multiple: --semihosting-config arg="..." 
>>> - a single option, placed at the end, and followed by any number of arguments: --semihosting-cmdline $@ \n
>>>
>>> both can generate the array of args required by UHI and both can pass args with spaces.
>>>
>>> the second one is more scripting friendly, and is somehow in the spirit of other unix programs that need to pass such arguments.
>>
>> I don't have a strong preference here, but I think I would be leaning
>> towards the second one as it just looks better. On the other hand it
>> reserves the last position in the QEMU command line, which I'm not sure
>> if can become a problem at some point?
> 
> do you have anything specific in mind? I don't know qemu very well, but apart from setting semihosting and kernel args, I doubt there are other use cases to require variable number of arguments. 

No, I don't.

Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 15 Apr 2015, at 15:49, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... If there aren't any objections for the proposed solution, then let's
> just cook up the patch.

the changes related to moving --semihosting-cmdline to the end of the line are in:

  git://git.code.sf.net/p/gnuarmeclipse/qemu gnuarmeclipse-qemu

the gnuarmeclipse-dev branch, commit 8010d2 

although I did not group the semihosting related variables in a structure, that code may still be a good starting point (especially since it is already functional in my fork).

I suggest you compare the gnuarmeclipse-dev branch with master and get the semihosting related patches.

if you need any help, please don't hesitate to ask. 

alternatively I can create the patch, but the changes must also touch non-arm code, and I have no experience there.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 15/04/2015 17:08, Liviu Ionescu wrote:
> 
>> On 15 Apr 2015, at 15:49, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>
>> ... If there aren't any objections for the proposed solution, then let's
>> just cook up the patch.
> 
> the changes related to moving --semihosting-cmdline to the end of the line are in:
> 
>   git://git.code.sf.net/p/gnuarmeclipse/qemu gnuarmeclipse-qemu
> 
> the gnuarmeclipse-dev branch, commit 8010d2 
> 
> although I did not group the semihosting related variables in a structure, that code may still be a good starting point (especially since it is already functional in my fork).
> 
> I suggest you compare the gnuarmeclipse-dev branch with master and get the semihosting related patches.
> 
> if you need any help, please don't hesitate to ask. 
> 
> alternatively I can create the patch, but the changes must also touch non-arm code, and I have no experience there.

Since the new option and your ARM semi-hosting improvements already work
for you I believe it would be much easier if you sent the patches first
rather than us figuring out the changes from your branch. Then we could
go from there reviewing / refining / testing etc.

Thanks,
Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Peter Maydell]

On 15 April 2015 at 10:09, Liviu Ionescu <ilg@livius.net> wrote:
> we currently have two solutions:
>
> - multiple: --semihosting-config arg="..."
> - a single option, placed at the end, and followed by any number
>   of arguments: --semihosting-cmdline $@ \n
>
> both can generate the array of args required by UHI and both can pass
> args with spaces.

I strongly dislike the second of these options -- it is nothing like
the way QEMU's system-emulator mode handles other command line
arguments. (The user-mode executable is not a good model to try
to copy, because (a) its sole purpose is executing other binaries
and passing them command line options (b) it doesn't have the
QemuOptions infrastructure.)

I think the first syntax is fine. If you happen to be on ARM where
there's only a single 'semihosting arguments' string in the ABI
then you can just say --semihosting-config arg="foo bar baz".
This has the comma-escaping ugliness, of course, but if you're
providing a GUI for users anyway it's trivial for the GUI to
comma-escape things.

-- PMM

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 16 Apr 2015, at 12:27, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> ... I believe it would be much easier if you sent the patches first

I just did this.

> On 16 Apr 2015, at 17:22, Peter Maydell <peter.maydell@linaro.org> wrote:

> it is nothing like
> the way QEMU's system-emulator mode handles other command line
> arguments.

generally passing an entire command line has little to do with passing singular command line arguments.

there are many unix programs that use this mechanism (for example the well known xargs) and most shells have special support for this with $@.

> This has the comma-escaping ugliness, of course, but if you're
> providing a GUI for users anyway it's trivial for the GUI to
> comma-escape things.

the Eclipse plug-in might do this, but try to do the same in multi-platform shell scripts used to run unit tests and you'll see how complicated things become.

this solution was the only one which met my requirements, and I currently do not see any disadvantages of using it in the system-emulator mode.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 16/04/2015 15:22, Peter Maydell wrote:
> On 15 April 2015 at 10:09, Liviu Ionescu <ilg@livius.net> wrote:
>> we currently have two solutions:
>>
>> - multiple: --semihosting-config arg="..."
>> - a single option, placed at the end, and followed by any number
>>   of arguments: --semihosting-cmdline $@ \n
>>
>> both can generate the array of args required by UHI and both can pass
>> args with spaces.
> 
> I strongly dislike the second of these options -- it is nothing like
> the way QEMU's system-emulator mode handles other command line
> arguments. (The user-mode executable is not a good model to try
> to copy, because (a) its sole purpose is executing other binaries
> and passing them command line options (b) it doesn't have the
> QemuOptions infrastructure.)
> 
> I think the first syntax is fine. If you happen to be on ARM where
> there's only a single 'semihosting arguments' string in the ABI
> then you can just say --semihosting-config arg="foo bar baz".
> This has the comma-escaping ugliness, of course, but if you're
> providing a GUI for users anyway it's trivial for the GUI to
> comma-escape things.

The patch which started this thread is the way to go then.

I'll resend the patch with dropped "RFC" as I would really like to go
forward and finally close this issue.

Thanks,
Leon

(PS: --semihosting-cmdline $@ is probably the most convenient from GUI /
wrapper point of view, but I don't have an opinion whether this is
enough to justify adding a top level QEMU command line option with such
a unique syntax)

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 21 Apr 2015, at 16:34, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> (PS: --semihosting-cmdline $@ is probably the most convenient from GUI /
> wrapper point of view, but I don't have an opinion whether this is
> enough to justify adding a top level QEMU command line option with such
> a unique syntax)

nope, the GUI wrapper is fine, I can do in Java any string processing; the problem is with scripts.

if you insist on the arg= solution, could you be so kind and provide a bash script wrapper that passes all arguments ($@) using your syntax?


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Peter Maydell]

On 21 April 2015 at 16:14, Liviu Ionescu <ilg@livius.net> wrote:
> if you insist on the arg= solution, could you be so kind and
> provide a bash script wrapper that passes all arguments ($@)
> using your syntax?


If you're willing to accept bashisms, then:

argstr=""
for arg in "$@"; do
    argstr=${argstr:+$argstr,}arg=${arg//,/,,}
done

qemu-system-arm --semihosting-options "$argstr"


If you want pure POSIX shell then it is slightly
uglier as we have to invoke sed:

argstr=""
for arg in "$@"; do
     o=$(printf '%s.\n' "$arg" | sed s/,/,,/g)
     argstr=${argstr:+$argstr,}arg=${o%.}
done

qemu-system-arm --semihosting-options "$argstr"

(Note the trick with the '.' to ensure correct handling
of trailing newlines in arguments; credit to mdw for
that one.)

-- PMM

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 21 Apr 2015, at 18:55, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On 21 April 2015 at 16:14, Liviu Ionescu <ilg@livius.net> wrote:
>> if you insist on the arg= solution, could you be so kind and
>> provide a bash script wrapper that passes all arguments ($@)
>> using your syntax?
> 
> 
> ... bashisms ... pure POSIX shell

do these scripts properly handle args containing spaces or embedded apostrophes/quotes?


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Peter Maydell]

On 21 April 2015 at 17:21, Liviu Ionescu <ilg@livius.net> wrote:
>
>> On 21 Apr 2015, at 18:55, Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On 21 April 2015 at 16:14, Liviu Ionescu <ilg@livius.net> wrote:
>>> if you insist on the arg= solution, could you be so kind and
>>> provide a bash script wrapper that passes all arguments ($@)
>>> using your syntax?
>>
>>
>> ... bashisms ... pure POSIX shell
>
> do these scripts properly handle args containing spaces or embedded apostrophes/quotes?

Yes. Also arguments containing newlines. I tested using a trivial
C program which prints out what it received:
#include <stdio.h>

int main(int argc, char **argv) {
    int i;

    for (i = 0; i < argc; i++) {
         printf("arg %d: >>>%s<<<\n", i, argv[i]);
    }
    return 0;
}

-- PMM

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On 04/21/2015 09:55 AM, Peter Maydell wrote:
> If you want pure POSIX shell then it is slightly
> uglier as we have to invoke sed:
> 
> argstr=""
> for arg in "$@"; do
>      o=$(printf '%s.\n' "$arg" | sed s/,/,,/g)
>      argstr=${argstr:+$argstr,}arg=${o%.}
> done

It's possible to use IFS= and case to avoid the need to fork, while
still writing portable shell, but the solution then takes LOTS more
lines of code, so I'm not even going to bother to try writing it here.

However, one thing this does NOT handle is an argument that is a literal
','.  That is, if I pass 'a', ',' and 'b' as my arguments, this script
results in $argstr being a,,,,b, which the qemu parser would read as a
single argument of 'a,,b' rather than the three intended arguments.

In order to handle that, you'd need some sort of solution that requires
an unambiguous separation between arguments.  For example,

--semihosting-options opt=a,,,,b
vs.
--semihosting-options opt=a,opt=,,,opt=b

would be unambiguous representations of 1 vs. 3 options.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 21 Apr 2015, at 19:23, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
>> do these scripts properly handle args containing spaces or embedded apostrophes/quotes?
> 
> Yes. Also arguments containing newlines.

super!

could you also suggest a solution for a .BAT script when running on windows?

in my opinion both these scripts (posix & windows) should be provided together with QEMU, and mentioned in the documentation.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Peter Maydell]

On 21 April 2015 at 17:48, Eric Blake <eblake@redhat.com> wrote:
> On 04/21/2015 09:55 AM, Peter Maydell wrote:
>> If you want pure POSIX shell then it is slightly
>> uglier as we have to invoke sed:
>>
>> argstr=""
>> for arg in "$@"; do
>>      o=$(printf '%s.\n' "$arg" | sed s/,/,,/g)
>>      argstr=${argstr:+$argstr,}arg=${o%.}
>> done
>
> It's possible to use IFS= and case to avoid the need to fork, while
> still writing portable shell, but the solution then takes LOTS more
> lines of code, so I'm not even going to bother to try writing it here.
>
> However, one thing this does NOT handle is an argument that is a literal
> ','.  That is, if I pass 'a', ',' and 'b' as my arguments, this script
> results in $argstr being a,,,,b, which the qemu parser would read as a
> single argument of 'a,,b' rather than the three intended arguments.

Hmm? You get "arg=a,arg=,,,arg=b". Does the parser not read that
the way I would expect?

> In order to handle that, you'd need some sort of solution that requires
> an unambiguous separation between arguments.  For example,
>
> --semihosting-options opt=a,,,,b
> vs.
> --semihosting-options opt=a,opt=,,,opt=b

The latter is what the script works with (except 'arg' rather
than 'opt').

Incidentally if you have a better idea for achieving the
desired goal than this messing around with comma-escaping
I would really like to know. I don't like either of the
current two proposals very much :-(

-- PMM

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On 04/21/2015 10:54 AM, Peter Maydell wrote:
> On 21 April 2015 at 17:48, Eric Blake <eblake@redhat.com> wrote:
>> On 04/21/2015 09:55 AM, Peter Maydell wrote:
>>> If you want pure POSIX shell then it is slightly
>>> uglier as we have to invoke sed:
>>>
>>> argstr=""
>>> for arg in "$@"; do
>>>      o=$(printf '%s.\n' "$arg" | sed s/,/,,/g)
>>>      argstr=${argstr:+$argstr,}arg=${o%.}
>>> done
>>
>> It's possible to use IFS= and case to avoid the need to fork, while
>> still writing portable shell, but the solution then takes LOTS more
>> lines of code, so I'm not even going to bother to try writing it here.
>>
>> However, one thing this does NOT handle is an argument that is a literal
>> ','.  That is, if I pass 'a', ',' and 'b' as my arguments, this script
>> results in $argstr being a,,,,b, which the qemu parser would read as a
>> single argument of 'a,,b' rather than the three intended arguments.
> 
> Hmm? You get "arg=a,arg=,,,arg=b". Does the parser not read that
> the way I would expect?

D'oh. I missed the 'arg=' insertion.

> 
>> In order to handle that, you'd need some sort of solution that requires
>> an unambiguous separation between arguments.  For example,
>>
>> --semihosting-options opt=a,,,,b
>> vs.
>> --semihosting-options opt=a,opt=,,,opt=b
> 
> The latter is what the script works with (except 'arg' rather
> than 'opt').

Indeed, we have an unambiguous representation already, and I just
overlooked it.

> 
> Incidentally if you have a better idea for achieving the
> desired goal than this messing around with comma-escaping
> I would really like to know. I don't like either of the
> current two proposals very much :-(

Libvirt is already used to doing comma escaping; so while I may not like
the resulting syntax, I _do_ like the consistency factor (that is, it's
much easier to reuse the same mechanism everywhere than it is to invent
yet another mechanism).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 21 Apr 2015, at 19:54, Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> Incidentally if you have a better idea for achieving the
> desired goal than this messing around with comma-escaping
> I would really like to know. I don't like either of the
> current two proposals very much :-(

please keep in mind that the main use of passing semihosting arguments is running unit tests.

I run lots of unit tests natively (although my main interest is for embedded, properly written tests are portable and have no problem running on OS X/Linux/Windos); I would like to run the same tests cross compiled for ARM and executed on QEMU.

the native tests are invoked with something like:

.../osx/test1 arg1 ... argn
.../linux/test1 arg1 ... argn

ideally the ARM tests would be invoked similarly:

qemu-wraper .../arm/test1 arg1 ... argn


similarly when running the native tests on windows (compiled with MinGW-w64), I need a wrapper to run the ARM tests with QEMU from a .BAT file (I keep the windows setup minimal, I do not want to install msys2 only to run a shell script)

as you can see, for this use case the "--semihosting-cmdline $@" is the natural solution, but if you prefer the comma separated arg= syntax and we can provide the proper wrappers, then it should be just a nuisance, not a show stopper.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Peter Maydell]

On 21 April 2015 at 17:51, Liviu Ionescu <ilg@livius.net> wrote:
> could you also suggest a solution for a .BAT script when running on windows?

I don't do windows, so you're on your own there, I'm afraid,
though a little googling suggests windows bat syntax does
have sufficient power (eg string-replacement functionality)
that it is doable.

-- PMM

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 21 Apr 2015, at 16:34, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> I'll resend the patch with dropped "RFC" as I would really like to go
> forward and finally close this issue.

perhaps you should first take a closer look at the patch I submitted, since it covers not only parsing the arguments, but also grouping all semihosting related variables in a structure and using them where needed.

also please note that the current arm code crashes with null pointers, and this problem was also fixed in my patch.

I think it would be more efficient to try to apply my patch and then change only the parsing part to suit your needs.


regards,

Liviu

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Leon Alrae]

On 21/04/2015 19:08, Liviu Ionescu wrote:
> 
>> On 21 Apr 2015, at 16:34, Leon Alrae <leon.alrae@imgtec.com> wrote:
>>
>> I'll resend the patch with dropped "RFC" as I would really like to go
>> forward and finally close this issue.
> 
> perhaps you should first take a closer look at the patch I submitted, since it covers not only parsing the arguments, but also grouping all semihosting related variables in a structure and using them where needed.

Thanks for that patch. The issue I can see is that these changes could
have been broken into 3 separate patches: (1) add -semihosting-cmdline
option (2) convert to semihosting struct (3) improve the ARM
semihosting. I think during the discussion eventually we resigned from
adding top level -semihosting-cmdline option (1), but the other two
parts (2) and (3) probably are still valuable.

Please let me know if you want to extract (2) change and send it as a
separate patch (note that there were some suggestions regarding the
clean up on the other thread so probably syncing would be a good idea).
If not that's fine, since this is relatively small amount of changes I
can do that all in one shot while adding "arg=" suboption.

Thanks,
Leon

Re: [Qemu-devel] [RFC PATCH] vl.c: add -semihosting-config "arg" sub-argument

[Liviu Ionescu]

> On 23 Apr 2015, at 14:55, Leon Alrae <leon.alrae@imgtec.com> wrote:
> 
> have been broken into 3 separate patches:

please feel free to reorganise the patches as you think it is more appropriate.

to avoid conflicts in my branch I would obviously appreciate keeping the structure as is, and also keeping the cmdline as a concatenated string in the structure, but you find a better solution, no problem, I'll update my code.


regards,

Liviu