[bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
---
 lib/bb/fetch2/git.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 59a2ee8f..a8fcc2bf 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -156,6 +156,9 @@ class Git(FetchMethod):
         if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
             raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
 
+        if ud.proto in ('http', 'https'):
+            ud.path = ud.path.replace(" ", "%20")
+
         ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
 
         ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
-- 
2.17.1

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Ping

Regards
Ankur

On Thu, 4 Jul 2019, 4:45 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com> wrote:

> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
> ---
>  lib/bb/fetch2/git.py | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 59a2ee8f..a8fcc2bf 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -156,6 +156,9 @@ class Git(FetchMethod):
>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https',
> 'rsync'):
>              raise bb.fetch2.ParameterError("Invalid protocol type",
> ud.url)
>
> +        if ud.proto in ('http', 'https'):
> +            ud.path = ud.path.replace(" ", "%20")
> +
>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
>
>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> --
> 2.17.1
>
>

[-- Attachment #2: Type: text/html, Size: 1509 bytes --]

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[akuster808]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]



On 7/7/19 12:55 PM, Ankur Tyagi wrote:
> Ping
>
> Regards
> Ankur
>

This needs to land in master first before it can be backported.

I don't see this in the patchwork either., not sure why

- armin

> On Thu, 4 Jul 2019, 4:45 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com
> <mailto:ankur.tyagi85@gmail.com>> wrote:
>
>     Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com
>     <mailto:ankur.tyagi85@gmail.com>>
>     ---
>      lib/bb/fetch2/git.py | 3 +++
>      1 file changed, 3 insertions(+)
>
>     diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>     index 59a2ee8f..a8fcc2bf 100644
>     --- a/lib/bb/fetch2/git.py
>     +++ b/lib/bb/fetch2/git.py
>     @@ -156,6 +156,9 @@ class Git(FetchMethod):
>              if not ud.proto in ('git', 'file', 'ssh', 'http',
>     'https', 'rsync'):
>                  raise bb.fetch2.ParameterError("Invalid protocol
>     type", ud.url)
>
>     +        if ud.proto in ('http', 'https'):
>     +            ud.path = ud.path.replace(" ", "%20")
>     +
>              ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
>
>              ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
>     -- 
>     2.17.1
>
>


[-- Attachment #2: Type: text/html, Size: 2699 bytes --]

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

Thanks Armin, I will wait for some more time then

On Mon, Jul 8, 2019 at 3:06 PM akuster808 <akuster808@gmail.com> wrote:
>
>
>
> On 7/7/19 12:55 PM, Ankur Tyagi wrote:
>
> Ping
>
> Regards
> Ankur
>
>
> This needs to land in master first before it can be backported.
>
> I don't see this in the patchwork either., not sure why
>
> - armin
>
> On Thu, 4 Jul 2019, 4:45 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com> wrote:
>>
>> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
>> ---
>>  lib/bb/fetch2/git.py | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>> index 59a2ee8f..a8fcc2bf 100644
>> --- a/lib/bb/fetch2/git.py
>> +++ b/lib/bb/fetch2/git.py
>> @@ -156,6 +156,9 @@ class Git(FetchMethod):
>>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
>>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
>>
>> +        if ud.proto in ('http', 'https'):
>> +            ud.path = ud.path.replace(" ", "%20")
>> +
>>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
>>
>>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
>> --
>> 2.17.1
>>
>
>

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Mark Hatle]

How is a space being added?  I'm a bit confused by this.  Is it due to the gitsm
fetching?  or has someone figured out a way in SRC_URI to included spaces?

If it's only in the gitsm fetching, then maybe we need to do the conversion
there instead.

--Mark

On 7/3/19 11:46 PM, Ankur Tyagi wrote:
> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
> ---
>  lib/bb/fetch2/git.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 59a2ee8f..a8fcc2bf 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -156,6 +156,9 @@ class Git(FetchMethod):
>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
>  
> +        if ud.proto in ('http', 'https'):
> +            ud.path = ud.path.replace(" ", "%20")
> +
>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
>  
>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> 

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

We are using TFS in our company and within TFS we have different
project names that can have spaces.
With new version of TFS, git repo can also be created inside it and
the path of those repo also includes the project name that can have
space.
Hence the SRC_URI will have a space and that's when I started seeing errors.

Hope it answers your question

regards
Ankur

On Tue, Jul 9, 2019 at 6:28 AM Mark Hatle <mark.hatle@windriver.com> wrote:
>
> How is a space being added?  I'm a bit confused by this.  Is it due to the gitsm
> fetching?  or has someone figured out a way in SRC_URI to included spaces?
>
> If it's only in the gitsm fetching, then maybe we need to do the conversion
> there instead.
>
> --Mark
>
> On 7/3/19 11:46 PM, Ankur Tyagi wrote:
> > Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
> > ---
> >  lib/bb/fetch2/git.py | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > index 59a2ee8f..a8fcc2bf 100644
> > --- a/lib/bb/fetch2/git.py
> > +++ b/lib/bb/fetch2/git.py
> > @@ -156,6 +156,9 @@ class Git(FetchMethod):
> >          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
> >              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
> >
> > +        if ud.proto in ('http', 'https'):
> > +            ud.path = ud.path.replace(" ", "%20")
> > +
> >          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
> >
> >          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> >
>

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Mark Hatle]

On 7/8/19 4:12 PM, Ankur Tyagi wrote:
> We are using TFS in our company and within TFS we have different
> project names that can have spaces.
> With new version of TFS, git repo can also be created inside it and
> the path of those repo also includes the project name that can have
> space.

Yes, spaces are valid.  I don't dispute that.. but however are you entering them?

> Hence the SRC_URI will have a space and that's when I started seeing errors.


SRC_URI = "https://foo.com/bar foobar bar"

That will result in -three- entries:

https://foo.com/bar
foobar
bar

The later two are invalid, because SRC_URI is split on whitespace (spaces and tabs.)

So what I don't understand is how are spaces being entered into the SRC_URI
(other then with %20) unless they are coming in via gitsm?

--Mark

> Hope it answers your question
> 
> regards
> Ankur
> 
> On Tue, Jul 9, 2019 at 6:28 AM Mark Hatle <mark.hatle@windriver.com> wrote:
>>
>> How is a space being added?  I'm a bit confused by this.  Is it due to the gitsm
>> fetching?  or has someone figured out a way in SRC_URI to included spaces?
>>
>> If it's only in the gitsm fetching, then maybe we need to do the conversion
>> there instead.
>>
>> --Mark
>>
>> On 7/3/19 11:46 PM, Ankur Tyagi wrote:
>>> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
>>> ---
>>>  lib/bb/fetch2/git.py | 3 +++
>>>  1 file changed, 3 insertions(+)
>>>
>>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
>>> index 59a2ee8f..a8fcc2bf 100644
>>> --- a/lib/bb/fetch2/git.py
>>> +++ b/lib/bb/fetch2/git.py
>>> @@ -156,6 +156,9 @@ class Git(FetchMethod):
>>>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
>>>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
>>>
>>> +        if ud.proto in ('http', 'https'):
>>> +            ud.path = ud.path.replace(" ", "%20")
>>> +
>>>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
>>>
>>>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
>>>
>>

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

On Tue, Jul 9, 2019 at 11:01 AM Mark Hatle <mark.hatle@windriver.com> wrote:
>
> On 7/8/19 4:12 PM, Ankur Tyagi wrote:
> > We are using TFS in our company and within TFS we have different
> > project names that can have spaces.
> > With new version of TFS, git repo can also be created inside it and
> > the path of those repo also includes the project name that can have
> > space.
>
> Yes, spaces are valid.  I don't dispute that.. but however are you entering them?

I am entering the url in SRC_URI with %20 in place of space

> > Hence the SRC_URI will have a space and that's when I started seeing errors.
>
>
> SRC_URI = "https://foo.com/bar foobar bar"

In my case, it is SRC_URI = "https://foo.com/bar%20foobar%20bar"

> That will result in -three- entries:
>
> https://foo.com/bar
> foobar
> bar
>
> The later two are invalid, because SRC_URI is split on whitespace (spaces and tabs.)
>
> So what I don't understand is how are spaces being entered into the SRC_URI
> (other then with %20) unless they are coming in via gitsm?

Then during the fetch task, git clone command looks like this
"git clone https://foo.com/bar foobar bar" which results in errors

That's why I am trying to insert "%20" back

Regards
Ankur
>
> --Mark
>
> > Hope it answers your question
> >
> > regards
> > Ankur
> >
> > On Tue, Jul 9, 2019 at 6:28 AM Mark Hatle <mark.hatle@windriver.com> wrote:
> >>
> >> How is a space being added?  I'm a bit confused by this.  Is it due to the gitsm
> >> fetching?  or has someone figured out a way in SRC_URI to included spaces?
> >>
> >> If it's only in the gitsm fetching, then maybe we need to do the conversion
> >> there instead.
> >>
> >> --Mark
> >>
> >> On 7/3/19 11:46 PM, Ankur Tyagi wrote:
> >>> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
> >>> ---
> >>>  lib/bb/fetch2/git.py | 3 +++
> >>>  1 file changed, 3 insertions(+)
> >>>
> >>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> >>> index 59a2ee8f..a8fcc2bf 100644
> >>> --- a/lib/bb/fetch2/git.py
> >>> +++ b/lib/bb/fetch2/git.py
> >>> @@ -156,6 +156,9 @@ class Git(FetchMethod):
> >>>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
> >>>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
> >>>
> >>> +        if ud.proto in ('http', 'https'):
> >>> +            ud.path = ud.path.replace(" ", "%20")
> >>> +
> >>>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
> >>>
> >>>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> >>>
> >>
>

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

Hi,

Is there something else that needs to be done in this patch? Or
perhaps there is a better to handle this?
I am willing to learn and try it out if original patch is not good.

Thanks
Ankur

On Tue, Jul 9, 2019 at 11:50 AM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> On Tue, Jul 9, 2019 at 11:01 AM Mark Hatle <mark.hatle@windriver.com> wrote:
> >
> > On 7/8/19 4:12 PM, Ankur Tyagi wrote:
> > > We are using TFS in our company and within TFS we have different
> > > project names that can have spaces.
> > > With new version of TFS, git repo can also be created inside it and
> > > the path of those repo also includes the project name that can have
> > > space.
> >
> > Yes, spaces are valid.  I don't dispute that.. but however are you entering them?
>
> I am entering the url in SRC_URI with %20 in place of space
>
> > > Hence the SRC_URI will have a space and that's when I started seeing errors.
> >
> >
> > SRC_URI = "https://foo.com/bar foobar bar"
>
> In my case, it is SRC_URI = "https://foo.com/bar%20foobar%20bar"
>
> > That will result in -three- entries:
> >
> > https://foo.com/bar
> > foobar
> > bar
> >
> > The later two are invalid, because SRC_URI is split on whitespace (spaces and tabs.)
> >
> > So what I don't understand is how are spaces being entered into the SRC_URI
> > (other then with %20) unless they are coming in via gitsm?
>
> Then during the fetch task, git clone command looks like this
> "git clone https://foo.com/bar foobar bar" which results in errors
>
> That's why I am trying to insert "%20" back
>
> Regards
> Ankur
> >
> > --Mark
> >
> > > Hope it answers your question
> > >
> > > regards
> > > Ankur
> > >
> > > On Tue, Jul 9, 2019 at 6:28 AM Mark Hatle <mark.hatle@windriver.com> wrote:
> > >>
> > >> How is a space being added?  I'm a bit confused by this.  Is it due to the gitsm
> > >> fetching?  or has someone figured out a way in SRC_URI to included spaces?
> > >>
> > >> If it's only in the gitsm fetching, then maybe we need to do the conversion
> > >> there instead.
> > >>
> > >> --Mark
> > >>
> > >> On 7/3/19 11:46 PM, Ankur Tyagi wrote:
> > >>> Signed-off-by: Ankur Tyagi <ankur.tyagi85@gmail.com>
> > >>> ---
> > >>>  lib/bb/fetch2/git.py | 3 +++
> > >>>  1 file changed, 3 insertions(+)
> > >>>
> > >>> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > >>> index 59a2ee8f..a8fcc2bf 100644
> > >>> --- a/lib/bb/fetch2/git.py
> > >>> +++ b/lib/bb/fetch2/git.py
> > >>> @@ -156,6 +156,9 @@ class Git(FetchMethod):
> > >>>          if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
> > >>>              raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
> > >>>
> > >>> +        if ud.proto in ('http', 'https'):
> > >>> +            ud.path = ud.path.replace(" ", "%20")
> > >>> +
> > >>>          ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
> > >>>
> > >>>          ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
> > >>>
> > >>
> >

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Richard Purdie]

On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> Hi,
> 
> Is there something else that needs to be done in this patch? Or
> perhaps there is a better to handle this?
> I am willing to learn and try it out if original patch is not good.

I think Mark is right, we need to figure out where the %20 characters
are being stripped out and preserve them rather than injecting them
back in again.

The question is therefore where they're being stripped out in the code?

Cheers,

Richard

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

Hi,

Sorry for the late reply but I have figured out where %20 is being stripped out.

File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
path as urllib.parse.unquote(path) which strips out %20
So if I return path as it is, then %20 is preserved and works as expected.

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f6b5529b..97fd59e1 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -389,7 +389,7 @@ def decodeurl(url):
                 s1, s2 = s.split('=')
                 p[s1] = s2

-    return type, host, urllib.parse.unquote(path), user, pswd, p
+    return type, host, path, user, pswd, p


Is above change acceptable?

Regards
Ankur

On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> > Hi,
> >
> > Is there something else that needs to be done in this patch? Or
> > perhaps there is a better to handle this?
> > I am willing to learn and try it out if original patch is not good.
>
> I think Mark is right, we need to figure out where the %20 characters
> are being stripped out and preserve them rather than injecting them
> back in again.
>
> The question is therefore where they're being stripped out in the code?
>
> Cheers,
>
> Richard
>

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

Hi,

Previous idea is stupid, please discard that.
Just checked quote/unquote were added as part of commit
b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce

So explaining the situation again --> when fecthing from repo url
https://tfs.local/repos/foo%bar

During FetchData initialization, decodeurl(d.expand(url)) returns path
with %20 stripped

[log][__init__][decodeurl] Path returned /repos/foo bar

And when git method urldata_init() is called, it sets gitsrcname with
space instead of %20

[log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar

And when git method _get_repo_url is called, it always returns Path
with space instead of %20

[log][git][_get_repo_url] Path returned /repos/foo bar

So I am bit confused now on how to handle it properly as unquote is
removing %20 and it cannot be added back in later stage.

Regards
Ankur


On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> Hi,
>
> Sorry for the late reply but I have figured out where %20 is being stripped out.
>
> File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
> path as urllib.parse.unquote(path) which strips out %20
> So if I return path as it is, then %20 is preserved and works as expected.
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f6b5529b..97fd59e1 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -389,7 +389,7 @@ def decodeurl(url):
>                  s1, s2 = s.split('=')
>                  p[s1] = s2
>
> -    return type, host, urllib.parse.unquote(path), user, pswd, p
> +    return type, host, path, user, pswd, p
>
>
> Is above change acceptable?
>
> Regards
> Ankur
>
> On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> > > Hi,
> > >
> > > Is there something else that needs to be done in this patch? Or
> > > perhaps there is a better to handle this?
> > > I am willing to learn and try it out if original patch is not good.
> >
> > I think Mark is right, we need to figure out where the %20 characters
> > are being stripped out and preserve them rather than injecting them
> > back in again.
> >
> > The question is therefore where they're being stripped out in the code?
> >
> > Cheers,
> >
> > Richard
> >

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

How about something like this:

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f6b5529b..fa1f9e27 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -380,6 +380,9 @@ def decodeurl(url):
         user = ''
         pswd = ''

+    if type not in ['git']:
+        path = urllib.parse.unquote(path)
+
     p = collections.OrderedDict()
     if parm:
         for s in parm.split(';'):
@@ -389,7 +392,7 @@ def decodeurl(url):
                 s1, s2 = s.split('=')
                 p[s1] = s2

-    return type, host, urllib.parse.unquote(path), user, pswd, p
+    return type, host, path, user, pswd, p

 def encodeurl(decoded):
     """Encodes a URL from tokens (scheme, network location, path,

regards
Ankur

On Fri, Jul 26, 2019 at 3:13 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> Hi,
>
> Previous idea is stupid, please discard that.
> Just checked quote/unquote were added as part of commit
> b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce
>
> So explaining the situation again --> when fecthing from repo url
> https://tfs.local/repos/foo%bar
>
> During FetchData initialization, decodeurl(d.expand(url)) returns path
> with %20 stripped
>
> [log][__init__][decodeurl] Path returned /repos/foo bar
>
> And when git method urldata_init() is called, it sets gitsrcname with
> space instead of %20
>
> [log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar
>
> And when git method _get_repo_url is called, it always returns Path
> with space instead of %20
>
> [log][git][_get_repo_url] Path returned /repos/foo bar
>
> So I am bit confused now on how to handle it properly as unquote is
> removing %20 and it cannot be added back in later stage.
>
> Regards
> Ankur
>
>
> On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
> >
> > Hi,
> >
> > Sorry for the late reply but I have figured out where %20 is being stripped out.
> >
> > File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
> > path as urllib.parse.unquote(path) which strips out %20
> > So if I return path as it is, then %20 is preserved and works as expected.
> >
> > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> > index f6b5529b..97fd59e1 100644
> > --- a/lib/bb/fetch2/__init__.py
> > +++ b/lib/bb/fetch2/__init__.py
> > @@ -389,7 +389,7 @@ def decodeurl(url):
> >                  s1, s2 = s.split('=')
> >                  p[s1] = s2
> >
> > -    return type, host, urllib.parse.unquote(path), user, pswd, p
> > +    return type, host, path, user, pswd, p
> >
> >
> > Is above change acceptable?
> >
> > Regards
> > Ankur
> >
> > On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > >
> > > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> > > > Hi,
> > > >
> > > > Is there something else that needs to be done in this patch? Or
> > > > perhaps there is a better to handle this?
> > > > I am willing to learn and try it out if original patch is not good.
> > >
> > > I think Mark is right, we need to figure out where the %20 characters
> > > are being stripped out and preserve them rather than injecting them
> > > back in again.
> > >
> > > The question is therefore where they're being stripped out in the code?
> > >
> > > Cheers,
> > >
> > > Richard
> > >

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

I haven't heard from anyone and unless if everyone is busy, please let
me know and I'll stop pinging.

regards
Ankur

On Sun, Jul 28, 2019 at 9:15 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
>
> How about something like this:
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f6b5529b..fa1f9e27 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -380,6 +380,9 @@ def decodeurl(url):
>          user = ''
>          pswd = ''
>
> +    if type not in ['git']:
> +        path = urllib.parse.unquote(path)
> +
>      p = collections.OrderedDict()
>      if parm:
>          for s in parm.split(';'):
> @@ -389,7 +392,7 @@ def decodeurl(url):
>                  s1, s2 = s.split('=')
>                  p[s1] = s2
>
> -    return type, host, urllib.parse.unquote(path), user, pswd, p
> +    return type, host, path, user, pswd, p
>
>  def encodeurl(decoded):
>      """Encodes a URL from tokens (scheme, network location, path,
>
> regards
> Ankur
>
> On Fri, Jul 26, 2019 at 3:13 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
> >
> > Hi,
> >
> > Previous idea is stupid, please discard that.
> > Just checked quote/unquote were added as part of commit
> > b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce
> >
> > So explaining the situation again --> when fecthing from repo url
> > https://tfs.local/repos/foo%bar
> >
> > During FetchData initialization, decodeurl(d.expand(url)) returns path
> > with %20 stripped
> >
> > [log][__init__][decodeurl] Path returned /repos/foo bar
> >
> > And when git method urldata_init() is called, it sets gitsrcname with
> > space instead of %20
> >
> > [log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar
> >
> > And when git method _get_repo_url is called, it always returns Path
> > with space instead of %20
> >
> > [log][git][_get_repo_url] Path returned /repos/foo bar
> >
> > So I am bit confused now on how to handle it properly as unquote is
> > removing %20 and it cannot be added back in later stage.
> >
> > Regards
> > Ankur
> >
> >
> > On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi <ankur.tyagi85@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > Sorry for the late reply but I have figured out where %20 is being stripped out.
> > >
> > > File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
> > > path as urllib.parse.unquote(path) which strips out %20
> > > So if I return path as it is, then %20 is preserved and works as expected.
> > >
> > > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> > > index f6b5529b..97fd59e1 100644
> > > --- a/lib/bb/fetch2/__init__.py
> > > +++ b/lib/bb/fetch2/__init__.py
> > > @@ -389,7 +389,7 @@ def decodeurl(url):
> > >                  s1, s2 = s.split('=')
> > >                  p[s1] = s2
> > >
> > > -    return type, host, urllib.parse.unquote(path), user, pswd, p
> > > +    return type, host, path, user, pswd, p
> > >
> > >
> > > Is above change acceptable?
> > >
> > > Regards
> > > Ankur
> > >
> > > On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:
> > > >
> > > > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> > > > > Hi,
> > > > >
> > > > > Is there something else that needs to be done in this patch? Or
> > > > > perhaps there is a better to handle this?
> > > > > I am willing to learn and try it out if original patch is not good.
> > > >
> > > > I think Mark is right, we need to figure out where the %20 characters
> > > > are being stripped out and preserve them rather than injecting them
> > > > back in again.
> > > >
> > > > The question is therefore where they're being stripped out in the code?
> > > >
> > > > Cheers,
> > > >
> > > > Richard
> > > >

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 3878 bytes --]

Hi Richard,

Any input on how it should be handled? I have identified the area like you
asked and some ways to fix it as well

Thanks
Ankur

On Sun, 28 Jul 2019, 9:15 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com> wrote:

> How about something like this:
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index f6b5529b..fa1f9e27 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -380,6 +380,9 @@ def decodeurl(url):
>          user = ''
>          pswd = ''
>
> +    if type not in ['git']:
> +        path = urllib.parse.unquote(path)
> +
>      p = collections.OrderedDict()
>      if parm:
>          for s in parm.split(';'):
> @@ -389,7 +392,7 @@ def decodeurl(url):
>                  s1, s2 = s.split('=')
>                  p[s1] = s2
>
> -    return type, host, urllib.parse.unquote(path), user, pswd, p
> +    return type, host, path, user, pswd, p
>
>  def encodeurl(decoded):
>      """Encodes a URL from tokens (scheme, network location, path,
>
> regards
> Ankur
>
> On Fri, Jul 26, 2019 at 3:13 PM Ankur Tyagi <ankur.tyagi85@gmail.com>
> wrote:
> >
> > Hi,
> >
> > Previous idea is stupid, please discard that.
> > Just checked quote/unquote were added as part of commit
> > b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce
> >
> > So explaining the situation again --> when fecthing from repo url
> > https://tfs.local/repos/foo%bar
> >
> > During FetchData initialization, decodeurl(d.expand(url)) returns path
> > with %20 stripped
> >
> > [log][__init__][decodeurl] Path returned /repos/foo bar
> >
> > And when git method urldata_init() is called, it sets gitsrcname with
> > space instead of %20
> >
> > [log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar
> >
> > And when git method _get_repo_url is called, it always returns Path
> > with space instead of %20
> >
> > [log][git][_get_repo_url] Path returned /repos/foo bar
> >
> > So I am bit confused now on how to handle it properly as unquote is
> > removing %20 and it cannot be added back in later stage.
> >
> > Regards
> > Ankur
> >
> >
> > On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi <ankur.tyagi85@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > Sorry for the late reply but I have figured out where %20 is being
> stripped out.
> > >
> > > File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
> > > path as urllib.parse.unquote(path) which strips out %20
> > > So if I return path as it is, then %20 is preserved and works as
> expected.
> > >
> > > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> > > index f6b5529b..97fd59e1 100644
> > > --- a/lib/bb/fetch2/__init__.py
> > > +++ b/lib/bb/fetch2/__init__.py
> > > @@ -389,7 +389,7 @@ def decodeurl(url):
> > >                  s1, s2 = s.split('=')
> > >                  p[s1] = s2
> > >
> > > -    return type, host, urllib.parse.unquote(path), user, pswd, p
> > > +    return type, host, path, user, pswd, p
> > >
> > >
> > > Is above change acceptable?
> > >
> > > Regards
> > > Ankur
> > >
> > > On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:
> > > >
> > > > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> > > > > Hi,
> > > > >
> > > > > Is there something else that needs to be done in this patch? Or
> > > > > perhaps there is a better to handle this?
> > > > > I am willing to learn and try it out if original patch is not good.
> > > >
> > > > I think Mark is right, we need to figure out where the %20 characters
> > > > are being stripped out and preserve them rather than injecting them
> > > > back in again.
> > > >
> > > > The question is therefore where they're being stripped out in the
> code?
> > > >
> > > > Cheers,
> > > >
> > > > Richard
> > > >
>

[-- Attachment #2: Type: text/html, Size: 5420 bytes --]

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Khem Raj]

On 8/3/19 2:34 PM, Ankur Tyagi wrote:
> Hi Richard,
> 
> Any input on how it should be handled? I have identified the area like 
> you asked and some ways to fix it as well
> 
> Thanks
> Ankur
> 
> On Sun, 28 Jul 2019, 9:15 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com 
> <mailto:ankur.tyagi85@gmail.com>> wrote:
> 
>     How about something like this:
> 
>     diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
>     index f6b5529b..fa1f9e27 100644
>     --- a/lib/bb/fetch2/__init__.py
>     +++ b/lib/bb/fetch2/__init__.py
>     @@ -380,6 +380,9 @@ def decodeurl(url):
>               user = ''
>               pswd = ''
> 
>     +    if type not in ['git']:
>     +        path = urllib.parse.unquote(path)
>     +
>           p = collections.OrderedDict()
>           if parm:
>               for s in parm.split(';'):
>     @@ -389,7 +392,7 @@ def decodeurl(url):
>                       s1, s2 = s.split('=')
>                       p[s1] = s2
> 
>     -    return type, host, urllib.parse.unquote(path), user, pswd, p
>     +    return type, host, path, user, pswd, p

It would be good to add some self test for this case as well and patch 
needs to be formatted as per 
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

> 
>       def encodeurl(decoded):
>           """Encodes a URL from tokens (scheme, network location, path,
> 
>     regards
>     Ankur
> 
>     On Fri, Jul 26, 2019 at 3:13 PM Ankur Tyagi <ankur.tyagi85@gmail.com
>     <mailto:ankur.tyagi85@gmail.com>> wrote:
>      >
>      > Hi,
>      >
>      > Previous idea is stupid, please discard that.
>      > Just checked quote/unquote were added as part of commit
>      > b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce
>      >
>      > So explaining the situation again --> when fecthing from repo url
>      > https://tfs.local/repos/foo%bar
>      >
>      > During FetchData initialization, decodeurl(d.expand(url)) returns
>     path
>      > with %20 stripped
>      >
>      > [log][__init__][decodeurl] Path returned /repos/foo bar
>      >
>      > And when git method urldata_init() is called, it sets gitsrcname with
>      > space instead of %20
>      >
>      > [log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar
>      >
>      > And when git method _get_repo_url is called, it always returns Path
>      > with space instead of %20
>      >
>      > [log][git][_get_repo_url] Path returned /repos/foo bar
>      >
>      > So I am bit confused now on how to handle it properly as unquote is
>      > removing %20 and it cannot be added back in later stage.
>      >
>      > Regards
>      > Ankur
>      >
>      >
>      > On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi
>     <ankur.tyagi85@gmail.com <mailto:ankur.tyagi85@gmail.com>> wrote:
>      > >
>      > > Hi,
>      > >
>      > > Sorry for the late reply but I have figured out where %20 is
>     being stripped out.
>      > >
>      > > File lib/bb/fetch2/__init__.py, method decodeurl(url) is returning
>      > > path as urllib.parse.unquote(path) which strips out %20
>      > > So if I return path as it is, then %20 is preserved and works
>     as expected.
>      > >
>      > > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
>      > > index f6b5529b..97fd59e1 100644
>      > > --- a/lib/bb/fetch2/__init__.py
>      > > +++ b/lib/bb/fetch2/__init__.py
>      > > @@ -389,7 +389,7 @@ def decodeurl(url):
>      > >                  s1, s2 = s.split('=')
>      > >                  p[s1] = s2
>      > >
>      > > -    return type, host, urllib.parse.unquote(path), user, pswd, p
>      > > +    return type, host, path, user, pswd, p
>      > >
>      > >
>      > > Is above change acceptable?
>      > >
>      > > Regards
>      > > Ankur
>      > >
>      > > On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
>      > > <richard.purdie@linuxfoundation.org
>     <mailto:richard.purdie@linuxfoundation.org>> wrote:
>      > > >
>      > > > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
>      > > > > Hi,
>      > > > >
>      > > > > Is there something else that needs to be done in this patch? Or
>      > > > > perhaps there is a better to handle this?
>      > > > > I am willing to learn and try it out if original patch is
>     not good.
>      > > >
>      > > > I think Mark is right, we need to figure out where the %20
>     characters
>      > > > are being stripped out and preserve them rather than
>     injecting them
>      > > > back in again.
>      > > >
>      > > > The question is therefore where they're being stripped out in
>     the code?
>      > > >
>      > > > Cheers,
>      > > >
>      > > > Richard
>      > > >
> 
> 

Re: [bitbake-devel, 1.42, 1.40, v2] git.py: Handle space in git repo name for http/https protocol

[Ankur Tyagi]

[-- Attachment #1: Type: text/plain, Size: 5347 bytes --]

On Sun, 4 Aug 2019, 11:45 a.m. Khem Raj, <raj.khem@gmail.com> wrote:

> On 8/3/19 2:34 PM, Ankur Tyagi wrote:
> > Hi Richard,
> >
> > Any input on how it should be handled? I have identified the area like
> > you asked and some ways to fix it as well
> >
> > Thanks
> > Ankur
> >
> > On Sun, 28 Jul 2019, 9:15 p.m. Ankur Tyagi, <ankur.tyagi85@gmail.com
> > <mailto:ankur.tyagi85@gmail.com>> wrote:
> >
> >     How about something like this:
> >
> >     diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> >     index f6b5529b..fa1f9e27 100644
> >     --- a/lib/bb/fetch2/__init__.py
> >     +++ b/lib/bb/fetch2/__init__.py
> >     @@ -380,6 +380,9 @@ def decodeurl(url):
> >               user = ''
> >               pswd = ''
> >
> >     +    if type not in ['git']:
> >     +        path = urllib.parse.unquote(path)
> >     +
> >           p = collections.OrderedDict()
> >           if parm:
> >               for s in parm.split(';'):
> >     @@ -389,7 +392,7 @@ def decodeurl(url):
> >                       s1, s2 = s.split('=')
> >                       p[s1] = s2
> >
> >     -    return type, host, urllib.parse.unquote(path), user, pswd, p
> >     +    return type, host, path, user, pswd, p
>
> It would be good to add some self test for this case as well and patch
> needs to be formatted as per
> https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
>
> I wanted to get nod on the fix so that I can send v3 of patch. I like your
> idea for self test , any reference on how to do it would be great. Thanks
>



> >
> >       def encodeurl(decoded):
> >           """Encodes a URL from tokens (scheme, network location, path,
> >
> >     regards
> >     Ankur
> >
> >     On Fri, Jul 26, 2019 at 3:13 PM Ankur Tyagi <ankur.tyagi85@gmail.com
> >     <mailto:ankur.tyagi85@gmail.com>> wrote:
> >      >
> >      > Hi,
> >      >
> >      > Previous idea is stupid, please discard that.
> >      > Just checked quote/unquote were added as part of commit
> >      > b1dbc24ebcc4e5100c32568c2c41fd982fb4bcce
> >      >
> >      > So explaining the situation again --> when fecthing from repo url
> >      > https://tfs.local/repos/foo%bar
> >      >
> >      > During FetchData initialization, decodeurl(d.expand(url)) returns
> >     path
> >      > with %20 stripped
> >      >
> >      > [log][__init__][decodeurl] Path returned /repos/foo bar
> >      >
> >      > And when git method urldata_init() is called, it sets gitsrcname
> with
> >      > space instead of %20
> >      >
> >      > [log][git][urldata_init] gitsrcname is tfs.local.repos.foo bar
> >      >
> >      > And when git method _get_repo_url is called, it always returns
> Path
> >      > with space instead of %20
> >      >
> >      > [log][git][_get_repo_url] Path returned /repos/foo bar
> >      >
> >      > So I am bit confused now on how to handle it properly as unquote
> is
> >      > removing %20 and it cannot be added back in later stage.
> >      >
> >      > Regards
> >      > Ankur
> >      >
> >      >
> >      > On Thu, Jul 25, 2019 at 7:23 PM Ankur Tyagi
> >     <ankur.tyagi85@gmail.com <mailto:ankur.tyagi85@gmail.com>> wrote:
> >      > >
> >      > > Hi,
> >      > >
> >      > > Sorry for the late reply but I have figured out where %20 is
> >     being stripped out.
> >      > >
> >      > > File lib/bb/fetch2/__init__.py, method decodeurl(url) is
> returning
> >      > > path as urllib.parse.unquote(path) which strips out %20
> >      > > So if I return path as it is, then %20 is preserved and works
> >     as expected.
> >      > >
> >      > > diff --git a/lib/bb/fetch2/__init__.py
> b/lib/bb/fetch2/__init__.py
> >      > > index f6b5529b..97fd59e1 100644
> >      > > --- a/lib/bb/fetch2/__init__.py
> >      > > +++ b/lib/bb/fetch2/__init__.py
> >      > > @@ -389,7 +389,7 @@ def decodeurl(url):
> >      > >                  s1, s2 = s.split('=')
> >      > >                  p[s1] = s2
> >      > >
> >      > > -    return type, host, urllib.parse.unquote(path), user, pswd,
> p
> >      > > +    return type, host, path, user, pswd, p
> >      > >
> >      > >
> >      > > Is above change acceptable?
> >      > >
> >      > > Regards
> >      > > Ankur
> >      > >
> >      > > On Wed, Jul 10, 2019 at 4:33 AM Richard Purdie
> >      > > <richard.purdie@linuxfoundation.org
> >     <mailto:richard.purdie@linuxfoundation.org>> wrote:
> >      > > >
> >      > > > On Wed, 2019-07-10 at 21:26 +1200, Ankur Tyagi wrote:
> >      > > > > Hi,
> >      > > > >
> >      > > > > Is there something else that needs to be done in this
> patch? Or
> >      > > > > perhaps there is a better to handle this?
> >      > > > > I am willing to learn and try it out if original patch is
> >     not good.
> >      > > >
> >      > > > I think Mark is right, we need to figure out where the %20
> >     characters
> >      > > > are being stripped out and preserve them rather than
> >     injecting them
> >      > > > back in again.
> >      > > >
> >      > > > The question is therefore where they're being stripped out in
> >     the code?
> >      > > >
> >      > > > Cheers,
> >      > > >
> >      > > > Richard
> >      > > >
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 8507 bytes --]