[Qemu-devel] linux-user: enabling binfmt P flag

[Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

Hi,

After weekend, I think the solution to using the P flag is to
go back to Joakim's original patch:

http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html

With this, we get:

If you continue to use qemu-x-static in your binfmt_misc registration,
nothing changes - both old and new qemu work using the old binfmt
registration.

If you rename the binary qemu-x-binfmt, you need to update the
binfmt_misc register to have P flag and new binary  - you get correct
argv with new qemu. Any old qemu you still have around, will stop
working. But with "file not found" error rather than obscurely eating
one of the arguments and running regardless.

This leaves us with one case - people who are used to running
qemu-x-static ./binary to test single binaries. Distro's will need
leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
check doesn't trigger, and qemu works as before.

The key point: this way nobody's working setup will break, unless they
update binfmt registration. As long as the change is done by users
them self (I need correct argv0 -> I will update binfmt), there is very
little surprise for anyone. 

There will be some fallout once *distributions* change the binfmt - users
will notice their existing qemu chroots stop working with a "file not
found" error for any binary they try to run.

If we find even this breakage too much, I'm not sure this can be fixed.

Riku

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Alexander Graf]

On 25.08.14 11:09, Riku Voipio wrote:
> Hi,
> 
> After weekend, I think the solution to using the P flag is to
> go back to Joakim's original patch:
> 
> http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> 
> With this, we get:
> 
> If you continue to use qemu-x-static in your binfmt_misc registration,
> nothing changes - both old and new qemu work using the old binfmt
> registration.
> 
> If you rename the binary qemu-x-binfmt, you need to update the
> binfmt_misc register to have P flag and new binary  - you get correct
> argv with new qemu. Any old qemu you still have around, will stop
> working. But with "file not found" error rather than obscurely eating
> one of the arguments and running regardless.
> 
> This leaves us with one case - people who are used to running
> qemu-x-static ./binary to test single binaries. Distro's will need
> leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> check doesn't trigger, and qemu works as before.
> 
> The key point: this way nobody's working setup will break, unless they
> update binfmt registration. As long as the change is done by users
> them self (I need correct argv0 -> I will update binfmt), there is very
> little surprise for anyone. 
> 
> There will be some fallout once *distributions* change the binfmt - users
> will notice their existing qemu chroots stop working with a "file not
> found" error for any binary they try to run.
> 
> If we find even this breakage too much, I'm not sure this can be fixed.

I would very much prefer if we could stick with only a single binary.
And yes, switching semantics when you use binfmt wrappers will hurt for
a short while, but after that everyone will have their setups changed
and we're safe for the future.


Alex

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Alexander Graf <agraf@suse.de> wrote on 2014/08/25 11:14:58:
> 
> On 25.08.14 11:09, Riku Voipio wrote:
> > Hi,
> > 
> > After weekend, I think the solution to using the P flag is to
> > go back to Joakim's original patch:
> > 
> > http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> > 
> > With this, we get:
> > 
> > If you continue to use qemu-x-static in your binfmt_misc registration,
> > nothing changes - both old and new qemu work using the old binfmt
> > registration.
> > 
> > If you rename the binary qemu-x-binfmt, you need to update the
> > binfmt_misc register to have P flag and new binary  - you get correct
> > argv with new qemu. Any old qemu you still have around, will stop
> > working. But with "file not found" error rather than obscurely eating
> > one of the arguments and running regardless.
> > 
> > This leaves us with one case - people who are used to running
> > qemu-x-static ./binary to test single binaries. Distro's will need
> > leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> > check doesn't trigger, and qemu works as before.
> > 
> > The key point: this way nobody's working setup will break, unless they
> > update binfmt registration. As long as the change is done by users
> > them self (I need correct argv0 -> I will update binfmt), there is 
very
> > little surprise for anyone. 
> > 
> > There will be some fallout once *distributions* change the binfmt - 
users
> > will notice their existing qemu chroots stop working with a "file not
> > found" error for any binary they try to run.
> > 
> > If we find even this breakage too much, I'm not sure this can be 
fixed.
> 
> I would very much prefer if we could stick with only a single binary.
> And yes, switching semantics when you use binfmt wrappers will hurt for
> a short while, but after that everyone will have their setups changed
> and we're safe for the future.

Yes, but as this is going to break something whatever we do I feel it is 
better
move towards always requiring P flag as this how binfmt should have been 
impl.
from the beginning and have users switch over now.
Then ask the kernel folks for a way to explicitly see what flags are used
from within linux-user so you can adapt automatically.

 Jocke 

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

On Mon, Aug 25, 2014 at 11:14:58AM +0200, Alexander Graf wrote:
> 
> 
> On 25.08.14 11:09, Riku Voipio wrote:
> > Hi,
> > 
> > After weekend, I think the solution to using the P flag is to
> > go back to Joakim's original patch:
> > 
> > http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> > 
> > With this, we get:
> > 
> > If you continue to use qemu-x-static in your binfmt_misc registration,
> > nothing changes - both old and new qemu work using the old binfmt
> > registration.
> > 
> > If you rename the binary qemu-x-binfmt, you need to update the
> > binfmt_misc register to have P flag and new binary  - you get correct
> > argv with new qemu. Any old qemu you still have around, will stop
> > working. But with "file not found" error rather than obscurely eating
> > one of the arguments and running regardless.
> > 
> > This leaves us with one case - people who are used to running
> > qemu-x-static ./binary to test single binaries. Distro's will need
> > leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> > check doesn't trigger, and qemu works as before.
> > 
> > The key point: this way nobody's working setup will break, unless they
> > update binfmt registration. As long as the change is done by users
> > them self (I need correct argv0 -> I will update binfmt), there is very
> > little surprise for anyone. 
> > 
> > There will be some fallout once *distributions* change the binfmt - users
> > will notice their existing qemu chroots stop working with a "file not
> > found" error for any binary they try to run.
> > 
> > If we find even this breakage too much, I'm not sure this can be fixed.
 
> I would very much prefer if we could stick with only a single binary.
> And yes, switching semantics when you use binfmt wrappers will hurt for
> a short while, but after that everyone will have their setups changed
> and we're safe for the future.
 
I don't really the unpredictable nature of the breakage. Take 
$ rm a b c

With P flag:    /bin/rm rm a b c
Without P flag: /bin/rm a b c

If we use old qemu with P flag: qemu will run /bin/rm with argv: "/bin/rm rm a b c"
 -> tries to delete "rm"
If we use new qemu without P flag, qemu will run /bin/rm with argv: "a b c" 
 -> fails to delete "a"

This is the black magic errors that drive users nuts when they try to debug what
is happening... "File not found" when the qemu binary is not in the
right place is confusing enough.

Riku

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Alexander Graf]

On 25.08.14 14:42, Riku Voipio wrote:
> On Mon, Aug 25, 2014 at 11:14:58AM +0200, Alexander Graf wrote:
>>
>>
>> On 25.08.14 11:09, Riku Voipio wrote:
>>> Hi,
>>>
>>> After weekend, I think the solution to using the P flag is to
>>> go back to Joakim's original patch:
>>>
>>> http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
>>>
>>> With this, we get:
>>>
>>> If you continue to use qemu-x-static in your binfmt_misc registration,
>>> nothing changes - both old and new qemu work using the old binfmt
>>> registration.
>>>
>>> If you rename the binary qemu-x-binfmt, you need to update the
>>> binfmt_misc register to have P flag and new binary  - you get correct
>>> argv with new qemu. Any old qemu you still have around, will stop
>>> working. But with "file not found" error rather than obscurely eating
>>> one of the arguments and running regardless.
>>>
>>> This leaves us with one case - people who are used to running
>>> qemu-x-static ./binary to test single binaries. Distro's will need
>>> leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
>>> check doesn't trigger, and qemu works as before.
>>>
>>> The key point: this way nobody's working setup will break, unless they
>>> update binfmt registration. As long as the change is done by users
>>> them self (I need correct argv0 -> I will update binfmt), there is very
>>> little surprise for anyone. 
>>>
>>> There will be some fallout once *distributions* change the binfmt - users
>>> will notice their existing qemu chroots stop working with a "file not
>>> found" error for any binary they try to run.
>>>
>>> If we find even this breakage too much, I'm not sure this can be fixed.
>  
>> I would very much prefer if we could stick with only a single binary.
>> And yes, switching semantics when you use binfmt wrappers will hurt for
>> a short while, but after that everyone will have their setups changed
>> and we're safe for the future.
>  
> I don't really the unpredictable nature of the breakage. Take 
> $ rm a b c
> 
> With P flag:    /bin/rm rm a b c
> Without P flag: /bin/rm a b c
> 
> If we use old qemu with P flag: qemu will run /bin/rm with argv: "/bin/rm rm a b c"
>  -> tries to delete "rm"
> If we use new qemu without P flag, qemu will run /bin/rm with argv: "a b c" 
>  -> fails to delete "a"
> 
> This is the black magic errors that drive users nuts when they try to debug what
> is happening... "File not found" when the qemu binary is not in the
> right place is confusing enough.

Yes, but is anyone actually using the "P" flag? We've never advertised
anywhere that QEMU supports it.

Maybe we should just make the next version be 3.0 and declare it a major
ABI breakage ;).


Alex

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

On Mon, Aug 25, 2014 at 02:46:21PM +0200, Alexander Graf wrote:
> 
> 
> On 25.08.14 14:42, Riku Voipio wrote:
> > On Mon, Aug 25, 2014 at 11:14:58AM +0200, Alexander Graf wrote:
> >>
> >>
> >> On 25.08.14 11:09, Riku Voipio wrote:
> >>> Hi,
> >>>
> >>> After weekend, I think the solution to using the P flag is to
> >>> go back to Joakim's original patch:
> >>>
> >>> http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> >>>
> >>> With this, we get:
> >>>
> >>> If you continue to use qemu-x-static in your binfmt_misc registration,
> >>> nothing changes - both old and new qemu work using the old binfmt
> >>> registration.
> >>>
> >>> If you rename the binary qemu-x-binfmt, you need to update the
> >>> binfmt_misc register to have P flag and new binary  - you get correct
> >>> argv with new qemu. Any old qemu you still have around, will stop
> >>> working. But with "file not found" error rather than obscurely eating
> >>> one of the arguments and running regardless.
> >>>
> >>> This leaves us with one case - people who are used to running
> >>> qemu-x-static ./binary to test single binaries. Distro's will need
> >>> leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> >>> check doesn't trigger, and qemu works as before.
> >>>
> >>> The key point: this way nobody's working setup will break, unless they
> >>> update binfmt registration. As long as the change is done by users
> >>> them self (I need correct argv0 -> I will update binfmt), there is very
> >>> little surprise for anyone. 
> >>>
> >>> There will be some fallout once *distributions* change the binfmt - users
> >>> will notice their existing qemu chroots stop working with a "file not
> >>> found" error for any binary they try to run.
> >>>
> >>> If we find even this breakage too much, I'm not sure this can be fixed.
> >  
> >> I would very much prefer if we could stick with only a single binary.
> >> And yes, switching semantics when you use binfmt wrappers will hurt for
> >> a short while, but after that everyone will have their setups changed
> >> and we're safe for the future.
> >  
> > I don't really the unpredictable nature of the breakage. Take 
> > $ rm a b c
> > 
> > With P flag:    /bin/rm rm a b c
> > Without P flag: /bin/rm a b c
> > 
> > If we use old qemu with P flag: qemu will run /bin/rm with argv: "/bin/rm rm a b c"
> >  -> tries to delete "rm"
> > If we use new qemu without P flag, qemu will run /bin/rm with argv: "a b c" 
> >  -> fails to delete "a"
> > 
> > This is the black magic errors that drive users nuts when they try to debug what
> > is happening... "File not found" when the qemu binary is not in the
> > right place is confusing enough.
 
> Yes, but is anyone actually using the "P" flag? We've never advertised
> anywhere that QEMU supports it.

No, we don't use the P flag now. The changeset has not been merged in.
With "new Qemu" I meant Joakims "if O assume P" patch. 

> Maybe we should just make the next version be 3.0 and declare it a major
> ABI breakage ;).
> 
> 
> Alex

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 3094 bytes --]


> Le 25 août 2014 à 14:46, Alexander Graf <agraf@suse.de> a écrit :
>
>
>
>
> On 25.08.14 14:42, Riku Voipio wrote:
> > On Mon, Aug 25, 2014 at 11:14:58AM +0200, Alexander Graf wrote:
> >>
> >>
> >> On 25.08.14 11:09, Riku Voipio wrote:
> >>> Hi,
> >>>
> >>> After weekend, I think the solution to using the P flag is to
> >>> go back to Joakim's original patch:
> >>>
> >>> http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> >>>
> >>> With this, we get:
> >>>
> >>> If you continue to use qemu-x-static in your binfmt_misc registration,
> >>> nothing changes - both old and new qemu work using the old binfmt
> >>> registration.
> >>>
> >>> If you rename the binary qemu-x-binfmt, you need to update the
> >>> binfmt_misc register to have P flag and new binary - you get correct
> >>> argv with new qemu. Any old qemu you still have around, will stop
> >>> working. But with "file not found" error rather than obscurely eating
> >>> one of the arguments and running regardless.
> >>>
> >>> This leaves us with one case - people who are used to running
> >>> qemu-x-static ./binary to test single binaries. Distro's will need
> >>> leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> >>> check doesn't trigger, and qemu works as before.
> >>>
> >>> The key point: this way nobody's working setup will break, unless they
> >>> update binfmt registration. As long as the change is done by users
> >>> them self (I need correct argv0 -> I will update binfmt), there is very
> >>> little surprise for anyone.
> >>>
> >>> There will be some fallout once *distributions* change the binfmt - users
> >>> will notice their existing qemu chroots stop working with a "file not
> >>> found" error for any binary they try to run.
> >>>
> >>> If we find even this breakage too much, I'm not sure this can be fixed.
> >
> >> I would very much prefer if we could stick with only a single binary.
> >> And yes, switching semantics when you use binfmt wrappers will hurt for
> >> a short while, but after that everyone will have their setups changed
> >> and we're safe for the future.
> >
> > I don't really the unpredictable nature of the breakage. Take
> > $ rm a b c
> >
> > With P flag: /bin/rm rm a b c
> > Without P flag: /bin/rm a b c
> >
> > If we use old qemu with P flag: qemu will run /bin/rm with argv: "/bin/rm rm
> > a b c"
> > -> tries to delete "rm"
> > If we use new qemu without P flag, qemu will run /bin/rm with argv: "a b c"
> > -> fails to delete "a"
> >
> > This is the black magic errors that drive users nuts when they try to debug
> > what
> > is happening... "File not found" when the qemu binary is not in the
> > right place is confusing enough.
>
> Yes, but is anyone actually using the "P" flag? We've never advertised
> anywhere that QEMU supports it.
>
> Maybe we should just make the next version be 3.0 and declare it a major
> ABI breakage ;).

You can also add the feature and let's the configure manages if it must be
enabled or not.

Regards,
Laurent

[-- Attachment #2: Type: text/html, Size: 5377 bytes --]

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 14:42:57:
> 
> On Mon, Aug 25, 2014 at 11:14:58AM +0200, Alexander Graf wrote:
> > 
> > 
> > On 25.08.14 11:09, Riku Voipio wrote:
> > > Hi,
> > > 
> > > After weekend, I think the solution to using the P flag is to
> > > go back to Joakim's original patch:
> > > 
> > > http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
> > > 
> > > With this, we get:
> > > 
> > > If you continue to use qemu-x-static in your binfmt_misc 
registration,
> > > nothing changes - both old and new qemu work using the old binfmt
> > > registration.
> > > 
> > > If you rename the binary qemu-x-binfmt, you need to update the
> > > binfmt_misc register to have P flag and new binary  - you get 
correct
> > > argv with new qemu. Any old qemu you still have around, will stop
> > > working. But with "file not found" error rather than obscurely 
eating
> > > one of the arguments and running regardless.
> > > 
> > > This leaves us with one case - people who are used to running
> > > qemu-x-static ./binary to test single binaries. Distro's will need
> > > leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" 
string
> > > check doesn't trigger, and qemu works as before.
> > > 
> > > The key point: this way nobody's working setup will break, unless 
they
> > > update binfmt registration. As long as the change is done by users
> > > them self (I need correct argv0 -> I will update binfmt), there is 
very
> > > little surprise for anyone. 
> > > 
> > > There will be some fallout once *distributions* change the binfmt - 
users
> > > will notice their existing qemu chroots stop working with a "file 
not
> > > found" error for any binary they try to run.
> > > 
> > > If we find even this breakage too much, I'm not sure this can be 
fixed.
> 
> > I would very much prefer if we could stick with only a single binary.
> > And yes, switching semantics when you use binfmt wrappers will hurt 
for
> > a short while, but after that everyone will have their setups changed
> > and we're safe for the future.
> 
> I don't really the unpredictable nature of the breakage. Take 
> $ rm a b c
> 
> With P flag:    /bin/rm rm a b c
> Without P flag: /bin/rm a b c
> 
> If we use old qemu with P flag: qemu will run /bin/rm with argv: 
"/bin/rm rm a b c"
>  -> tries to delete "rm"
> If we use new qemu without P flag, qemu will run /bin/rm with argv: "a b 
c" 
>  -> fails to delete "a"
> 
> This is the black magic errors that drive users nuts when they try to 
debug what
> is happening... "File not found" when the qemu binary is not in the
> right place is confusing enough.

Then consider when you run a LXC without P flag. bash expects argv0=-bash 
to be a 
login shell and source "profile" files. This can result in even worse 
"black magic errors"

Either way stuff may break :(
Oh well, perhaps the binfmt wrapper is the best although then you
will be stuck with the -binfmt magic handling in QEMU for a long time.

 Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

Hi,

On Mon, Aug 25, 2014 at 03:39:19PM +0200, Joakim Tjernlund wrote:
> Then consider when you run a LXC without P flag. 

Please remember that your usecase of running Qemu in LXC is a new feature,
never before supported. Adding new features is always nice. However, it must
not happen with expense of regressing already working features
(traditional chroot).

Riku

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 15:55:55:
> 
> Hi,
> 
> On Mon, Aug 25, 2014 at 03:39:19PM +0200, Joakim Tjernlund wrote:
> > Then consider when you run a LXC without P flag. 
> 
> Please remember that your usecase of running Qemu in LXC is a new 
feature,
> never before supported. Adding new features is always nice. However, it 
must
> not happen with expense of regressing already working features
> (traditional chroot).

OK, but traditional chroot is unaffected with my patch. Only binfmt
users which uses the binfmt flag O will have to change to PO. How
many is that? 

 Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

On Mon, Aug 25, 2014 at 04:30:40PM +0200, Joakim Tjernlund wrote:
> Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 15:55:55:
> > 
> > Hi,
> > 
> > On Mon, Aug 25, 2014 at 03:39:19PM +0200, Joakim Tjernlund wrote:
> > > Then consider when you run a LXC without P flag. 
> > 
> > Please remember that your usecase of running Qemu in LXC is a new 
> feature,
> > never before supported. Adding new features is always nice. However, it 
> must
> > not happen with expense of regressing already working features
> > (traditional chroot).
 
> OK, but traditional chroot is unaffected with my patch. Only binfmt
> users which uses the binfmt flag O will have to change to PO. How
> many is that? 

Traditional chroot is with binfmt_misc. Debian and ubuntu packages of
Qemu enable OC flags, so at least all debian/ubuntu users who use qemu
linux-user in chroots are affected. 

Using OC seems quite typical:

http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-emulation/qemu/files/qemu-binfmt.initd-r1?view=markup
http://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-64-host-using-qemu-userspace-emulation/#content

Others, like Suse, scratchbox use the P flag and wrapper binary.

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 16:49:17:
> 
> On Mon, Aug 25, 2014 at 04:30:40PM +0200, Joakim Tjernlund wrote:
> > Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 15:55:55:
> > > 
> > > Hi,
> > > 
> > > On Mon, Aug 25, 2014 at 03:39:19PM +0200, Joakim Tjernlund wrote:
> > > > Then consider when you run a LXC without P flag. 
> > > 
> > > Please remember that your usecase of running Qemu in LXC is a new 
> > feature,
> > > never before supported. Adding new features is always nice. However, 
it 
> > must
> > > not happen with expense of regressing already working features
> > > (traditional chroot).
> 
> > OK, but traditional chroot is unaffected with my patch. Only binfmt
> > users which uses the binfmt flag O will have to change to PO. How
> > many is that? 
> 
> Traditional chroot is with binfmt_misc. Debian and ubuntu packages of

Sorry, I figured you meant plain chroot 

> Qemu enable OC flags, so at least all debian/ubuntu users who use qemu
> linux-user in chroots are affected. 
> 
> Using OC seems quite typical:
> 
> 
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-emulation/qemu/files/qemu-binfmt.initd-r1?view=markup

> 
http://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-64-host-using-qemu-userspace-emulation/#content

> 
> Others, like Suse, scratchbox use the P flag and wrapper binary.

Yes, but I meant those not using the dist. default as dists will update
their binfmt scripts when upgrading QEMU.

Basically this will only be a potential problem for DIYs QEMU users.

 Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Any resolution for how to handle P flag yet?

 Jocke

Joakim Tjernlund/Transmode wrote on 2014/08/25 17:02:42:
> 
> Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 16:49:17:
> > 
> > On Mon, Aug 25, 2014 at 04:30:40PM +0200, Joakim Tjernlund wrote:
> > > Riku Voipio <riku.voipio@iki.fi> wrote on 2014/08/25 15:55:55:
> > > > 
> > > > Hi,
> > > > 
> > > > On Mon, Aug 25, 2014 at 03:39:19PM +0200, Joakim Tjernlund wrote:
> > > > > Then consider when you run a LXC without P flag. 
> > > > 
> > > > Please remember that your usecase of running Qemu in LXC is a new 
> > > feature,
> > > > never before supported. Adding new features is always nice. 
However, it 
> > > must
> > > > not happen with expense of regressing already working features
> > > > (traditional chroot).
> > 
> > > OK, but traditional chroot is unaffected with my patch. Only binfmt
> > > users which uses the binfmt flag O will have to change to PO. How
> > > many is that? 
> > 
> > Traditional chroot is with binfmt_misc. Debian and ubuntu packages of
> 
> Sorry, I figured you meant plain chroot 
> 
> > Qemu enable OC flags, so at least all debian/ubuntu users who use qemu
> > linux-user in chroots are affected. 
> > 
> > Using OC seems quite typical:
> > 
> > 
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-emulation/qemu/files/qemu-binfmt.initd-r1?view=markup

> > 
http://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-64-host-using-qemu-userspace-emulation/#content

> > 
> > Others, like Suse, scratchbox use the P flag and wrapper binary.

> Yes, but I meant those not using the dist. default as dists will update
> their binfmt scripts when upgrading QEMU.
> 
> Basically this will only be a potential problem for DIYs QEMU users.
> 
>  Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Peter Maydell]

[cc'ing MJT for more distro opinion since I think fundamentally
the choice we ought to make upstream is "what's not going to
screw over distros"... Paolo, is there a RedHat QEMU maintainer
who would have an opinion here?]

On 25 August 2014 10:09, Riku Voipio <riku.voipio@iki.fi> wrote:
> After weekend, I think the solution to using the P flag is to
> go back to Joakim's original patch:
>
> http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg02269.html
>
> With this, we get:
>
> If you continue to use qemu-x-static in your binfmt_misc registration,
> nothing changes - both old and new qemu work using the old binfmt
> registration.
>
> If you rename the binary qemu-x-binfmt, you need to update the
> binfmt_misc register to have P flag and new binary  - you get correct
> argv with new qemu. Any old qemu you still have around, will stop
> working. But with "file not found" error rather than obscurely eating
> one of the arguments and running regardless.
>
> This leaves us with one case - people who are used to running
> qemu-x-static ./binary to test single binaries. Distro's will need
> leave a symlink from qemu-x-binfmt qemu-x-static. The "-binfmt" string
> check doesn't trigger, and qemu works as before.
>
> The key point: this way nobody's working setup will break, unless they
> update binfmt registration. As long as the change is done by users
> them self (I need correct argv0 -> I will update binfmt), there is very
> little surprise for anyone.
>
> There will be some fallout once *distributions* change the binfmt - users
> will notice their existing qemu chroots stop working with a "file not
> found" error for any binary they try to run.
>
> If we find even this breakage too much, I'm not sure this can be fixed.

My take on this:
 * I agree we can't break existing working setups with binfmt
registered with "O"
 * It would be nice to have a forward path to binfmt registered
with "OP"
 * The best we can do is provide a QEMU which can work in
both "O" and "OP" configurations, and clearly advertise to
distros and individual users that (a) we recommend the "OP"
config and binary name (b) updating the binfmt config will
break any old QEMU binaries they have lying around in
chroots (c) they should probably provide both binary names
for back-compatibility

ie essentially leave it up to distros whether they want to
move forward to the bright new "correct argv0" future at
the cost of some transition pain, or if they'll just sit with
their current config.

What I really would like is a way for the kernel to tell the
interpreter binary what particular mangling it's chosen to do
of the command line arguments. Then we could have one
binary that coped in both situations and when run via the
command line, and it would just come down to "recommend
OP and tell people/distros to make sure to update the
QEMU binaries in any chroots they have when they make
that change". Unfortunately the kernel doesn't
do that and it would need a kernel change to fix it :-(

-- PMM

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Peter Maydell <peter.maydell@linaro.org> wrote on 2014/08/29 20:01:12:

.....
> 
> What I really would like is a way for the kernel to tell the
> interpreter binary what particular mangling it's chosen to do
> of the command line arguments. Then we could have one
> binary that coped in both situations and when run via the
> command line, and it would just come down to "recommend
> OP and tell people/distros to make sure to update the
> QEMU binaries in any chroots they have when they make
> that change". Unfortunately the kernel doesn't
> do that and it would need a kernel change to fix it :-(

I think QEMU should request such a change, you have a good reason for it.
There will be more flags in the future which will make it impossible to
do the right thing automatically. I would like to see a flag which strips
away the qemu-ARCH from argv, much like the other interpreters do.

Perhaps similar to AT_EXECFD? An AT_BINFMTFLAGS which holds the flags(or 
an empty string when no flags)

 Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Paolo Bonzini]

Il 29/08/2014 20:01, Peter Maydell ha scritto:
> [cc'ing MJT for more distro opinion since I think fundamentally
> the choice we ought to make upstream is "what's not going to
> screw over distros"... Paolo, is there a RedHat QEMU maintainer
> who would have an opinion here?]

There's Cole Robinson.

BTW, Fedora doesn't use the binfmt scripts from QEMU, but does reuse the
binfmt lines.  We'd just add Ps and we'd be fine.

However, the problem is not really for distros.  Packagers just read the
release notes and adjust whatever needs to be adjusted.  The problem is
for people who compile from source and are bit by conflicting binfmt
formats from their distro.

The solution could be to extend binfmt_misc so that it sets two
environment variables BINFMT_MISC_PID and BINFMT_MISC_ORIG_ARGV0.  The
former is set to the pid of the binfmt "interpreter" program, the latter
to the argv[0] value.  Then QEMU can check if BINFMT_MISC_PID matches
getpid() and, if so, trust the BINFMT_MISC_ORIG_ARGV0 value.

Paolo

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Peter Maydell]

On 1 September 2014 09:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 29/08/2014 20:01, Peter Maydell ha scritto:
>> [cc'ing MJT for more distro opinion since I think fundamentally
>> the choice we ought to make upstream is "what's not going to
>> screw over distros"... Paolo, is there a RedHat QEMU maintainer
>> who would have an opinion here?]
>
> There's Cole Robinson.
>
> BTW, Fedora doesn't use the binfmt scripts from QEMU

That's ok, nobody with any sense doesn't.

>, but does reuse the
> binfmt lines.  We'd just add Ps and we'd be fine.

But this would break all your existing users' existing
chroot setups. That's the question I'm after an answer to:
what do you (as a distro) think would be acceptable as
transitional breakage, if anything?

> However, the problem is not really for distros.  Packagers just read the
> release notes and adjust whatever needs to be adjusted.  The problem is
> for people who compile from source and are bit by conflicting binfmt
> formats from their distro.

This is one reason I like the "one binary name for O and
one for P" approach.

> The solution could be to extend binfmt_misc so that it sets two
> environment variables BINFMT_MISC_PID and BINFMT_MISC_ORIG_ARGV0.  The
> former is set to the pid of the binfmt "interpreter" program, the latter
> to the argv[0] value.  Then QEMU can check if BINFMT_MISC_PID matches
> getpid() and, if so, trust the BINFMT_MISC_ORIG_ARGV0 value.

Certainly if we're in a position to get the kernel to be more
informative about how it invoked us that would be the ideal.

thanks
-- PMM

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Paolo Bonzini]

Il 01/09/2014 11:12, Peter Maydell ha scritto:
>> We'd just add Ps and we'd be fine.
> 
> But this would break all your existing users' existing
> chroot setups. That's the question I'm after an answer to:
> what do you (as a distro) think would be acceptable as
> transitional breakage, if anything?

Yes, but it's not like Fedora'd have choice.  Distros have to follow
what upstream does, even if it breaks something else (or they could
revert the "P" patch and get an entirely specular set of bug reports).

>> > The solution could be to extend binfmt_misc so that it sets two
>> > environment variables BINFMT_MISC_PID and BINFMT_MISC_ORIG_ARGV0.  The
>> > former is set to the pid of the binfmt "interpreter" program, the latter
>> > to the argv[0] value.  Then QEMU can check if BINFMT_MISC_PID matches
>> > getpid() and, if so, trust the BINFMT_MISC_ORIG_ARGV0 value.
> Certainly if we're in a position to get the kernel to be more
> informative about how it invoked us that would be the ideal.

I think it's simply that "P" was ill-designed (possibly because
implementing the above is not trivial).

Paolo

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Peter Maydell]

On 1 September 2014 10:28, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 01/09/2014 11:12, Peter Maydell ha scritto:
>>> We'd just add Ps and we'd be fine.
>>
>> But this would break all your existing users' existing
>> chroot setups. That's the question I'm after an answer to:
>> what do you (as a distro) think would be acceptable as
>> transitional breakage, if anything?
>
> Yes, but it's not like Fedora'd have choice.  Distros have to follow
> what upstream does, even if it breaks something else (or they could
> revert the "P" patch and get an entirely specular set of bug reports).

Yes, that's why I'm asking what you'd prefer before we
change upstream QEMU rather than just picking something
randomly and letting you deal with the consequences :-)

-- PMM

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Riku Voipio]

On Mon, Sep 01, 2014 at 10:12:18AM +0100, Peter Maydell wrote:
> On 1 September 2014 09:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 29/08/2014 20:01, Peter Maydell ha scritto:
> >> [cc'ing MJT for more distro opinion since I think fundamentally
> >> the choice we ought to make upstream is "what's not going to
> >> screw over distros"... Paolo, is there a RedHat QEMU maintainer
> >> who would have an opinion here?]
> >
> > There's Cole Robinson.
> >
> > BTW, Fedora doesn't use the binfmt scripts from QEMU
> 
> That's ok, nobody with any sense doesn't.
> 
> >, but does reuse the
> > binfmt lines.  We'd just add Ps and we'd be fine.
> 
> But this would break all your existing users' existing
> chroot setups. That's the question I'm after an answer to:
> what do you (as a distro) think would be acceptable as
> transitional breakage, if anything?
> 
> > However, the problem is not really for distros.  Packagers just read the
> > release notes and adjust whatever needs to be adjusted.  The problem is
> > for people who compile from source and are bit by conflicting binfmt
> > formats from their distro.

Or people with chroots from older/different distros, already
having a qemu-static inside.

> This is one reason I like the "one binary name for O and
> one for P" approach.

Maybe the new binary name could be something more generic than qemu-x-binfmt.
Say qemu-x-user. Then distributions and users can drop the old binary
name over time, and we are back to one binary again eventually.

> > The solution could be to extend binfmt_misc so that it sets two
> > environment variables BINFMT_MISC_PID and BINFMT_MISC_ORIG_ARGV0.  The
> > former is set to the pid of the binfmt "interpreter" program, the latter
> > to the argv[0] value.  Then QEMU can check if BINFMT_MISC_PID matches
> > getpid() and, if so, trust the BINFMT_MISC_ORIG_ARGV0 value.
 
> Certainly if we're in a position to get the kernel to be more
> informative about how it invoked us that would be the ideal.

There is AT_FLAGS, that seems unused atm (only ever set to 0).

http://lxr.free-electrons.com/source/fs/binfmt_elf.c#L240

As indeed I afree with Paolo that (in hindsight) it was misdesign for the
kernel to tell the application how it invoked us..

Riku

Riku

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Joakim Tjernlund]

Riku Voipio <riku.voipio@iki.fi> wrote on 2014/09/01 11:51:15:
> 
> On Mon, Sep 01, 2014 at 10:12:18AM +0100, Peter Maydell wrote:
> > On 1 September 2014 09:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > > Il 29/08/2014 20:01, Peter Maydell ha scritto:
> > >> [cc'ing MJT for more distro opinion since I think fundamentally
> > >> the choice we ought to make upstream is "what's not going to
> > >> screw over distros"... Paolo, is there a RedHat QEMU maintainer
> > >> who would have an opinion here?]
> > >
> > > There's Cole Robinson.
> > >
> > > BTW, Fedora doesn't use the binfmt scripts from QEMU
> > 
> > That's ok, nobody with any sense doesn't.
> > 
> > >, but does reuse the
> > > binfmt lines.  We'd just add Ps and we'd be fine.
> > 
> > But this would break all your existing users' existing
> > chroot setups. That's the question I'm after an answer to:
> > what do you (as a distro) think would be acceptable as
> > transitional breakage, if anything?
> > 
> > > However, the problem is not really for distros.  Packagers just read 
the
> > > release notes and adjust whatever needs to be adjusted.  The problem 
is
> > > for people who compile from source and are bit by conflicting binfmt
> > > formats from their distro.
> 
> Or people with chroots from older/different distros, already
> having a qemu-static inside.
> 
> > This is one reason I like the "one binary name for O and
> > one for P" approach.
> 
> Maybe the new binary name could be something more generic than 
qemu-x-binfmt.
> Say qemu-x-user. Then distributions and users can drop the old binary
> name over time, and we are back to one binary again eventually.
> 
> > > The solution could be to extend binfmt_misc so that it sets two
> > > environment variables BINFMT_MISC_PID and BINFMT_MISC_ORIG_ARGV0. 
The
> > > former is set to the pid of the binfmt "interpreter" program, the 
latter
> > > to the argv[0] value.  Then QEMU can check if BINFMT_MISC_PID 
matches
> > > getpid() and, if so, trust the BINFMT_MISC_ORIG_ARGV0 value.
> 
> > Certainly if we're in a position to get the kernel to be more
> > informative about how it invoked us that would be the ideal.
> 
> There is AT_FLAGS, that seems unused atm (only ever set to 0).
> 
> http://lxr.free-electrons.com/source/fs/binfmt_elf.c#L240
> 
> As indeed I afree with Paolo that (in hindsight) it was misdesign for 
the
> kernel to tell the application how it invoked us..

Did this go anywhere ? Is there a solution in sight?

   Jocke

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Peter Maydell]

On 17 September 2014 08:34, Joakim Tjernlund
<joakim.tjernlund@transmode.se> wrote:
> Did this go anywhere ? Is there a solution in sight?

I was hoping for more distro input, but absent that:

Does anybody care to try to float the idea of an API
extension to the binfmt stuff to pass us the argv0
out-of-band (eg via an elf auxv vector entry if some
new binfmt flag is passed)? That would I think be the
best approach, though obviously it's a kernel change.

Failing that I'm not sure we can change upstream
QEMU's behaviour because it will just break too
much to do that. This sucks but the kernel as it
stands just gives us no manoeuvering room :-(

thanks
-- PMM

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Paolo Bonzini]

Il 17/09/2014 18:12, Peter Maydell ha scritto:
> On 17 September 2014 08:34, Joakim Tjernlund
> <joakim.tjernlund@transmode.se> wrote:
>> Did this go anywhere ? Is there a solution in sight?
> 
> I was hoping for more distro input, but absent that:
> 
> Does anybody care to try to float the idea of an API
> extension to the binfmt stuff to pass us the argv0
> out-of-band (eg via an elf auxv vector entry if some
> new binfmt flag is passed)? That would I think be the
> best approach, though obviously it's a kernel change.
> 
> Failing that I'm not sure we can change upstream
> QEMU's behaviour because it will just break too
> much to do that. This sucks but the kernel as it
> stands just gives us no manoeuvering room :-(

Another idea is to add a configure option that defaults to yes.

Paolo

Re: [Qemu-devel] linux-user: enabling binfmt P flag

[Peter Maydell]

On 17 September 2014 12:25, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 17/09/2014 18:12, Peter Maydell ha scritto:
>> Does anybody care to try to float the idea of an API
>> extension to the binfmt stuff to pass us the argv0
>> out-of-band (eg via an elf auxv vector entry if some
>> new binfmt flag is passed)? That would I think be the
>> best approach, though obviously it's a kernel change.
>>
>> Failing that I'm not sure we can change upstream
>> QEMU's behaviour because it will just break too
>> much to do that. This sucks but the kernel as it
>> stands just gives us no manoeuvering room :-(
>
> Another idea is to add a configure option that defaults to yes.

I think that still ends up breaking a bunch of people
at some point in the future.

I spoke to a kernel guy this morning and he seemed
vaguely positive about the idea of an API extension,
so I'll see if I can find time to write and send out
a patch for that.

-- PMM