[PATCH 1/2] docs: kernel-doc comments are ASCII

[PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

From: Randy Dunlap <rdunlap@infradead.org>

kernel-doc parsing uses as ASCII codec, so let people know that
kernel-doc comments should be in ASCII characters only.

WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)

Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
---
 Documentation/doc-guide/kernel-doc.rst |    3 +++
 1 file changed, 3 insertions(+)

--- lnx-413-rc7.orig/Documentation/doc-guide/kernel-doc.rst
+++ lnx-413-rc7/Documentation/doc-guide/kernel-doc.rst
@@ -108,6 +108,9 @@ The function and type kernel-doc comment
 function or type being described. The overview kernel-doc comments may be freely
 placed at the top indentation level.
 
+.. attention:: kernel-doc comments should be written **only** in ASCII
+	       characters since they are processed as ASCII input.
+
 Example kernel-doc function comment::
 
   /**

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jonathan Corbet]

On Mon, 28 Aug 2017 16:10:09 -0700
Randy Dunlap <rdunlap@infradead.org> wrote:

> kernel-doc parsing uses as ASCII codec, so let people know that
> kernel-doc comments should be in ASCII characters only.
> 
> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)

So I don't get this error.  What kind of system are you running the docs
build on?  I would really rather that the docs system could handle modern
text if possible, so it would be better to figure out what's going on
here...

Thanks,

jon

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/30/17 14:23, Jonathan Corbet wrote:
> On Mon, 28 Aug 2017 16:10:09 -0700
> Randy Dunlap <rdunlap@infradead.org> wrote:
> 
>> kernel-doc parsing uses as ASCII codec, so let people know that
>> kernel-doc comments should be in ASCII characters only.
>>
>> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)
> 
> So I don't get this error.  What kind of system are you running the docs
> build on?  I would really rather that the docs system could handle modern
> text if possible, so it would be better to figure out what's going on
> here...

I'm OK with that. Source files in general don't need to be ASCII (0-127).

I did this patch based on this (private) comment:

> Yes, using ASCII should fix the problem.

what kind of system?  HP laptop.

Linux midway.site 4.4.79-18.26-default #1 SMP Thu Aug 10 20:30:05 UTC 2017 (fa5a935) x86_64 x86_64 x86_64 GNU/Linux

> sphinx-build --version
Sphinx (sphinx-build) 1.3.1


-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Mauro Carvalho Chehab]

Em Wed, 30 Aug 2017 15:02:59 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 08/30/17 14:23, Jonathan Corbet wrote:
> > On Mon, 28 Aug 2017 16:10:09 -0700
> > Randy Dunlap <rdunlap@infradead.org> wrote:
> >   
> >> kernel-doc parsing uses as ASCII codec, so let people know that
> >> kernel-doc comments should be in ASCII characters only.
> >>
> >> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)  
> > 
> > So I don't get this error.  What kind of system are you running the docs
> > build on?  I would really rather that the docs system could handle modern
> > text if possible, so it would be better to figure out what's going on
> > here...  
> 
> I'm OK with that. Source files in general don't need to be ASCII (0-127).
> 
> I did this patch based on this (private) comment:
> 
> > Yes, using ASCII should fix the problem.  
> 
> what kind of system?  HP laptop.
> 
> Linux midway.site 4.4.79-18.26-default #1 SMP Thu Aug 10 20:30:05 UTC 2017 (fa5a935) x86_64 x86_64 x86_64 GNU/Linux
> 
> > sphinx-build --version  
> Sphinx (sphinx-build) 1.3.1

I suspect that the problem is not related to the version, but to
what you might have set on LANG.

Maybe if we add something like:
	LANG=C.utf-8

to the Documentation/Makefile or adding:

	.. -*- coding: utf-8; mode: rst -*-

as the first line on the *.rst file that include the kernel-doc 
directive would solve the issue.

Regards,
Mauro

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jonathan Corbet]

On Wed, 30 Aug 2017 19:15:53 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:

> I suspect that the problem is not related to the version, but to
> what you might have set on LANG.
> 
> Maybe if we add something like:
> 	LANG=C.utf-8
> 
> to the Documentation/Makefile 

That's worth a try; Randy, can you give it a quick go?

> or adding:
> 
> 	.. -*- coding: utf-8; mode: rst -*-
> 
> as the first line on the *.rst file that include the kernel-doc 
> directive would solve the issue.

I guess I don't see how that would help, instead.  Emacs reads that line,
but it's not involved in the problem.

I wish I could reproduce this, then we could see what in that massive
try..except block in kerneldoc.py is throwing the exception.  Putting in
an explicit decode call might be enough to make the problem go away.

Thanks,

jon

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/30/17 15:31, Jonathan Corbet wrote:
> On Wed, 30 Aug 2017 19:15:53 -0300
> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> 
>> I suspect that the problem is not related to the version, but to
>> what you might have set on LANG.
>>
>> Maybe if we add something like:
>> 	LANG=C.utf-8
>>
>> to the Documentation/Makefile 
> 
> That's worth a try; Randy, can you give it a quick go?

Yes, that fixes it for me.  Thanks.

>> or adding:
>>
>> 	.. -*- coding: utf-8; mode: rst -*-
>>
>> as the first line on the *.rst file that include the kernel-doc 
>> directive would solve the issue.
> 
> I guess I don't see how that would help, instead.  Emacs reads that line,
> but it's not involved in the problem.
> 
> I wish I could reproduce this, then we could see what in that massive
> try..except block in kerneldoc.py is throwing the exception.  Putting in
> an explicit decode call might be enough to make the problem go away.



-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/30/17 16:01, Randy Dunlap wrote:
> On 08/30/17 15:31, Jonathan Corbet wrote:
>> On Wed, 30 Aug 2017 19:15:53 -0300
>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>
>>> I suspect that the problem is not related to the version, but to
>>> what you might have set on LANG.
>>>
>>> Maybe if we add something like:
>>> 	LANG=C.utf-8
>>>
>>> to the Documentation/Makefile 
>>
>> That's worth a try; Randy, can you give it a quick go?
> 
> Yes, that fixes it for me.  Thanks.

Wait!  I forgot to unpatch demux.h.  I'll test again now....

>>> or adding:
>>>
>>> 	.. -*- coding: utf-8; mode: rst -*-
>>>
>>> as the first line on the *.rst file that include the kernel-doc 
>>> directive would solve the issue.
>>
>> I guess I don't see how that would help, instead.  Emacs reads that line,
>> but it's not involved in the problem.
>>
>> I wish I could reproduce this, then we could see what in that massive
>> try..except block in kerneldoc.py is throwing the exception.  Putting in
>> an explicit decode call might be enough to make the problem go away.
> 
> 
> 


-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/30/17 16:04, Randy Dunlap wrote:
> On 08/30/17 16:01, Randy Dunlap wrote:
>> On 08/30/17 15:31, Jonathan Corbet wrote:
>>> On Wed, 30 Aug 2017 19:15:53 -0300
>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>
>>>> I suspect that the problem is not related to the version, but to
>>>> what you might have set on LANG.
>>>>
>>>> Maybe if we add something like:
>>>> 	LANG=C.utf-8
>>>>
>>>> to the Documentation/Makefile 
>>>
>>> That's worth a try; Randy, can you give it a quick go?
>>
>> Yes, that fixes it for me.  Thanks.
> 
> Wait!  I forgot to unpatch demux.h.  I'll test again now....

OK, still works for me.

> 
>>>> or adding:
>>>>
>>>> 	.. -*- coding: utf-8; mode: rst -*-
>>>>
>>>> as the first line on the *.rst file that include the kernel-doc 
>>>> directive would solve the issue.
>>>
>>> I guess I don't see how that would help, instead.  Emacs reads that line,
>>> but it's not involved in the problem.
>>>
>>> I wish I could reproduce this, then we could see what in that massive
>>> try..except block in kerneldoc.py is throwing the exception.  Putting in
>>> an explicit decode call might be enough to make the problem go away.
>>
>>
>>
> 
> 


-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Mauro Carvalho Chehab]

Em Wed, 30 Aug 2017 15:02:59 -0700
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 08/30/17 14:23, Jonathan Corbet wrote:
> > On Mon, 28 Aug 2017 16:10:09 -0700
> > Randy Dunlap <rdunlap@infradead.org> wrote:
> >   
> >> kernel-doc parsing uses as ASCII codec, so let people know that
> >> kernel-doc comments should be in ASCII characters only.
> >>
> >> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)  
> > 
> > So I don't get this error.  What kind of system are you running the docs
> > build on?  I would really rather that the docs system could handle modern
> > text if possible, so it would be better to figure out what's going on
> > here...  
> 
> I'm OK with that. Source files in general don't need to be ASCII (0-127).
> 
> I did this patch based on this (private) comment:
> 
> > Yes, using ASCII should fix the problem.  
> 
> what kind of system?  HP laptop.
> 
> Linux midway.site 4.4.79-18.26-default #1 SMP Thu Aug 10 20:30:05 UTC 2017 (fa5a935) x86_64 x86_64 x86_64 GNU/Linux
> 
> > sphinx-build --version  
> Sphinx (sphinx-build) 1.3.1

I tried hard to reproduce the error here... I even added some Chinese
chars on a kernel-doc markup and changed the language on my system
to LANG=en_US.iso885915.

No luck. 

As Documentation/conf.py has:

	# -*- coding: utf-8 -*-

on its first line, I suspect that the error you're getting is likely
due to the usage of a python version that doesn't recognize this.

It seems that such dialect was introduced on python version 2.3:

	https://docs.python.org/2.3/whatsnew/section-encodings.html

Yet, the documentation there seems to require a line before it,
e. g.:

	#!/usr/bin/env python
	# -*- coding: UTF-8 -*-

I suspect, however, that, if such line is added, on some systems it
may not work, e. g. if both python 2 and 3 are installed, it could
use the python version that doesn't have Sphinx installed.

So, I suspect that the safest way to fix it is with something like the
enclosed patch. Still, it could be useful to know what's happening,
just in case we get other reports.

Randy,

What's your python version?


Thanks,
Mauro


diff --git a/Documentation/Makefile b/Documentation/Makefile
index 85f7856f0092..94eb98031b56 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -55,6 +55,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
       cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media $2 && \
 	PYTHONDONTWRITEBYTECODE=1 \
 	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
+	LANG=C.utf8\
 	$(SPHINXBUILD) \
 	-b $2 \
 	-c $(abspath $(srctree)/$(src)) \

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jani Nikula]

On Thu, 31 Aug 2017, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> As Documentation/conf.py has:
>
> 	# -*- coding: utf-8 -*-
>
> on its first line, I suspect that the error you're getting is likely
> due to the usage of a python version that doesn't recognize this.

AFAIK that has nothing to do with python I/O, and everything to do with
the encoding of that specific python source file.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Mauro Carvalho Chehab]

Em Thu, 31 Aug 2017 16:26:44 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Thu, 31 Aug 2017, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> > As Documentation/conf.py has:
> >
> > 	# -*- coding: utf-8 -*-
> >
> > on its first line, I suspect that the error you're getting is likely
> > due to the usage of a python version that doesn't recognize this.  
> 
> AFAIK that has nothing to do with python I/O, and everything to do with
> the encoding of that specific python source file.

Jani,

It should have something to do with python version and/or to some
locale info at the system, as neither I or Jon can reproduce it.

Can you reproduce it on your system?

Thanks,
Mauro

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jonathan Corbet]

On Thu, 31 Aug 2017 10:56:26 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:

> It should have something to do with python version and/or to some
> locale info at the system, as neither I or Jon can reproduce it.

I can't reproduce it here, but I have certainly seen situations where
Python 2 wants to run with the ascii codec by default.

Note that the exception happens in our Sphinx extension, not in Sphinx
itself.  We've had other non-ascii text in our docs, so I think Sphinx is
doing the right thing.  The problem is with our own code.  If I could
reproduce it, it shouldn't be too hard to track down - take out that
massive "except anything" block and see where it explodes.

Randy, which distribution are you running, and are you using their version
of Sphinx?

Thanks,

jon

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 02:49, Mauro Carvalho Chehab wrote:
> Em Wed, 30 Aug 2017 15:02:59 -0700
> Randy Dunlap <rdunlap@infradead.org> escreveu:
> 
>> On 08/30/17 14:23, Jonathan Corbet wrote:
>>> On Mon, 28 Aug 2017 16:10:09 -0700
>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>   
>>>> kernel-doc parsing uses as ASCII codec, so let people know that
>>>> kernel-doc comments should be in ASCII characters only.
>>>>
>>>> WARNING: kernel-doc '../scripts/kernel-doc -rst -enable-lineno ../drivers/media/dvb-core/demux.h' processing failed with: 'ascii' codec can't decode byte 0xe2 in position 6368: ordinal not in range(128)  
>>>
>>> So I don't get this error.  What kind of system are you running the docs
>>> build on?  I would really rather that the docs system could handle modern
>>> text if possible, so it would be better to figure out what's going on
>>> here...  
>>
>> I'm OK with that. Source files in general don't need to be ASCII (0-127).
>>
>> I did this patch based on this (private) comment:
>>
>>> Yes, using ASCII should fix the problem.  
>>
>> what kind of system?  HP laptop.
>>
>> Linux midway.site 4.4.79-18.26-default #1 SMP Thu Aug 10 20:30:05 UTC 2017 (fa5a935) x86_64 x86_64 x86_64 GNU/Linux
>>
>>> sphinx-build --version  
>> Sphinx (sphinx-build) 1.3.1
> 
> I tried hard to reproduce the error here... I even added some Chinese
> chars on a kernel-doc markup and changed the language on my system
> to LANG=en_US.iso885915.
> 
> No luck. 
> 
> As Documentation/conf.py has:
> 
> 	# -*- coding: utf-8 -*-
> 
> on its first line, I suspect that the error you're getting is likely
> due to the usage of a python version that doesn't recognize this.
> 
> It seems that such dialect was introduced on python version 2.3:
> 
> 	https://docs.python.org/2.3/whatsnew/section-encodings.html
> 
> Yet, the documentation there seems to require a line before it,
> e. g.:
> 
> 	#!/usr/bin/env python
> 	# -*- coding: UTF-8 -*-
> 
> I suspect, however, that, if such line is added, on some systems it
> may not work, e. g. if both python 2 and 3 are installed, it could
> use the python version that doesn't have Sphinx installed.
> 
> So, I suspect that the safest way to fix it is with something like the
> enclosed patch. Still, it could be useful to know what's happening,
> just in case we get other reports.
> 
> Randy,
> 
> What's your python version?

> python --version
Python 2.7.13



-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 07:17, Jonathan Corbet wrote:
> On Thu, 31 Aug 2017 10:56:26 -0300
> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> 
>> It should have something to do with python version and/or to some
>> locale info at the system, as neither I or Jon can reproduce it.
> 
> I can't reproduce it here, but I have certainly seen situations where
> Python 2 wants to run with the ascii codec by default.
> 
> Note that the exception happens in our Sphinx extension, not in Sphinx
> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
> doing the right thing.  The problem is with our own code.  If I could
> reproduce it, it shouldn't be too hard to track down - take out that
> massive "except anything" block and see where it explodes.
> 
> Randy, which distribution are you running, and are you using their version
> of Sphinx?

opensuse LEAP 42.2
Yes, their sphinx 1.3.1.


-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jani Nikula]

On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 08/31/17 07:17, Jonathan Corbet wrote:
>> On Thu, 31 Aug 2017 10:56:26 -0300
>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>> 
>>> It should have something to do with python version and/or to some
>>> locale info at the system, as neither I or Jon can reproduce it.
>> 
>> I can't reproduce it here, but I have certainly seen situations where
>> Python 2 wants to run with the ascii codec by default.
>> 
>> Note that the exception happens in our Sphinx extension, not in Sphinx
>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>> doing the right thing.  The problem is with our own code.  If I could
>> reproduce it, it shouldn't be too hard to track down - take out that
>> massive "except anything" block and see where it explodes.
>> 
>> Randy, which distribution are you running, and are you using their version
>> of Sphinx?
>
> opensuse LEAP 42.2
> Yes, their sphinx 1.3.1.

What's your LANG setting? I think that's what it boils down to, and
trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
ways.

The odd thing is that I can reproduce the issue using a small python
snippet, but not through Sphinx.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 09:05, Jani Nikula wrote:
> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>
>>>> It should have something to do with python version and/or to some
>>>> locale info at the system, as neither I or Jon can reproduce it.
>>>
>>> I can't reproduce it here, but I have certainly seen situations where
>>> Python 2 wants to run with the ascii codec by default.
>>>
>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>> doing the right thing.  The problem is with our own code.  If I could
>>> reproduce it, it shouldn't be too hard to track down - take out that
>>> massive "except anything" block and see where it explodes.
>>>
>>> Randy, which distribution are you running, and are you using their version
>>> of Sphinx?
>>
>> opensuse LEAP 42.2
>> Yes, their sphinx 1.3.1.
> 
> What's your LANG setting? I think that's what it boils down to, and
> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
> ways.

(default)
LANG=C

until I add the patch:
+LANG=C.utf-8

> The odd thing is that I can reproduce the issue using a small python
> snippet, but not through Sphinx.



-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jani Nikula]

On Thu, 31 Aug 2017, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>> 
>>>> It should have something to do with python version and/or to some
>>>> locale info at the system, as neither I or Jon can reproduce it.
>>> 
>>> I can't reproduce it here, but I have certainly seen situations where
>>> Python 2 wants to run with the ascii codec by default.
>>> 
>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>> doing the right thing.  The problem is with our own code.  If I could
>>> reproduce it, it shouldn't be too hard to track down - take out that
>>> massive "except anything" block and see where it explodes.
>>> 
>>> Randy, which distribution are you running, and are you using their version
>>> of Sphinx?
>>
>> opensuse LEAP 42.2
>> Yes, their sphinx 1.3.1.
>
> What's your LANG setting? I think that's what it boils down to, and
> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
> ways.
>
> The odd thing is that I can reproduce the issue using a small python
> snippet, but not through Sphinx.

Your original error message suggests your Sphinx actually uses python
3. Can you check that? The clue is that it's the *decode* that fails.

Does the below patch help? It avoids the implicit ascii decoding due to
universal_newlines=True and your LANG setting, and does explicit utf-8
decoding instead.

Fingers crossed.

BR,
Jani.


diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
index d15e07f36881..39aa9e8697cc 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -27,6 +27,7 @@
 # Please make sure this works on both python2 and python3.
 #
 
+import codecs
 import os
 import subprocess
 import sys
@@ -88,13 +89,10 @@ class KernelDocDirective(Directive):
         try:
             env.app.verbose('calling kernel-doc \'%s\'' % (" ".join(cmd)))
 
-            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             out, err = p.communicate()
 
-            # python2 needs conversion to unicode.
-            # python3 with universal_newlines=True returns strings.
-            if sys.version_info.major < 3:
-                out, err = unicode(out, 'utf-8'), unicode(err, 'utf-8')
+            out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8')
 
             if p.returncode != 0:
                 sys.stderr.write(err)



-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 09:36, Jani Nikula wrote:
> On Thu, 31 Aug 2017, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>>
>>>>> It should have something to do with python version and/or to some
>>>>> locale info at the system, as neither I or Jon can reproduce it.
>>>>
>>>> I can't reproduce it here, but I have certainly seen situations where
>>>> Python 2 wants to run with the ascii codec by default.
>>>>
>>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>>> doing the right thing.  The problem is with our own code.  If I could
>>>> reproduce it, it shouldn't be too hard to track down - take out that
>>>> massive "except anything" block and see where it explodes.
>>>>
>>>> Randy, which distribution are you running, and are you using their version
>>>> of Sphinx?
>>>
>>> opensuse LEAP 42.2
>>> Yes, their sphinx 1.3.1.
>>
>> What's your LANG setting? I think that's what it boils down to, and
>> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
>> ways.
>>
>> The odd thing is that I can reproduce the issue using a small python
>> snippet, but not through Sphinx.
> 
> Your original error message suggests your Sphinx actually uses python
> 3. Can you check that? The clue is that it's the *decode* that fails.

Where do you see that clue?
My /usr/bin/python is linked to python2.7:

> ll /usr/bin/python
lrwxrwxrwx 1 root root 9 Jun 10 19:59 /usr/bin/python -> python2.7*

> Does the below patch help? It avoids the implicit ascii decoding due to
> universal_newlines=True and your LANG setting, and does explicit utf-8
> decoding instead.
> 
> Fingers crossed.

testing now.

-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 10:34, Randy Dunlap wrote:
> On 08/31/17 09:36, Jani Nikula wrote:
>> On Thu, 31 Aug 2017, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>>>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>>>
>>>>>> It should have something to do with python version and/or to some
>>>>>> locale info at the system, as neither I or Jon can reproduce it.
>>>>>
>>>>> I can't reproduce it here, but I have certainly seen situations where
>>>>> Python 2 wants to run with the ascii codec by default.
>>>>>
>>>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>>>> doing the right thing.  The problem is with our own code.  If I could
>>>>> reproduce it, it shouldn't be too hard to track down - take out that
>>>>> massive "except anything" block and see where it explodes.
>>>>>
>>>>> Randy, which distribution are you running, and are you using their version
>>>>> of Sphinx?
>>>>
>>>> opensuse LEAP 42.2
>>>> Yes, their sphinx 1.3.1.
>>>
>>> What's your LANG setting? I think that's what it boils down to, and
>>> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
>>> ways.
>>>
>>> The odd thing is that I can reproduce the issue using a small python
>>> snippet, but not through Sphinx.
>>
>> Your original error message suggests your Sphinx actually uses python
>> 3. Can you check that? The clue is that it's the *decode* that fails.
> 
> Where do you see that clue?
> My /usr/bin/python is linked to python2.7:
> 
>> ll /usr/bin/python
> lrwxrwxrwx 1 root root 9 Jun 10 19:59 /usr/bin/python -> python2.7*
> 
>> Does the below patch help? It avoids the implicit ascii decoding due to
>> universal_newlines=True and your LANG setting, and does explicit utf-8
>> decoding instead.
>>
>> Fingers crossed.
> 
> testing now.

Yes, that works for me.  Thanks.

-- 
~Randy

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Jani Nikula]

On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
> On 08/31/17 09:36, Jani Nikula wrote:
>> On Thu, 31 Aug 2017, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>>>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>>>
>>>>>> It should have something to do with python version and/or to some
>>>>>> locale info at the system, as neither I or Jon can reproduce it.
>>>>>
>>>>> I can't reproduce it here, but I have certainly seen situations where
>>>>> Python 2 wants to run with the ascii codec by default.
>>>>>
>>>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>>>> doing the right thing.  The problem is with our own code.  If I could
>>>>> reproduce it, it shouldn't be too hard to track down - take out that
>>>>> massive "except anything" block and see where it explodes.
>>>>>
>>>>> Randy, which distribution are you running, and are you using their version
>>>>> of Sphinx?
>>>>
>>>> opensuse LEAP 42.2
>>>> Yes, their sphinx 1.3.1.
>>>
>>> What's your LANG setting? I think that's what it boils down to, and
>>> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
>>> ways.
>>>
>>> The odd thing is that I can reproduce the issue using a small python
>>> snippet, but not through Sphinx.
>> 
>> Your original error message suggests your Sphinx actually uses python
>> 3. Can you check that? The clue is that it's the *decode* that fails.
>
> Where do you see that clue?

The message, "'ascii' codec can't decode byte 0xe2 in position 6368:
ordinal not in range(128)". In my testing I could only get that *decode*
error message using python 3.

> My /usr/bin/python is linked to python2.7:
>
>> ll /usr/bin/python
> lrwxrwxrwx 1 root root 9 Jun 10 19:59 /usr/bin/python -> python2.7*

Sure, but how about 'head $(which sphinx-build)'?

I could be completely mistaken too. ;)

>> Does the below patch help? It avoids the implicit ascii decoding due to
>> universal_newlines=True and your LANG setting, and does explicit utf-8
>> decoding instead.
>> 
>> Fingers crossed.
>
> testing now.

Thanks.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/2] docs: kernel-doc comments are ASCII

[Randy Dunlap]

On 08/31/17 10:55, Jani Nikula wrote:
> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>> On 08/31/17 09:36, Jani Nikula wrote:
>>> On Thu, 31 Aug 2017, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>>> On Thu, 31 Aug 2017, Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>> On 08/31/17 07:17, Jonathan Corbet wrote:
>>>>>> On Thu, 31 Aug 2017 10:56:26 -0300
>>>>>> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
>>>>>>
>>>>>>> It should have something to do with python version and/or to some
>>>>>>> locale info at the system, as neither I or Jon can reproduce it.
>>>>>>
>>>>>> I can't reproduce it here, but I have certainly seen situations where
>>>>>> Python 2 wants to run with the ascii codec by default.
>>>>>>
>>>>>> Note that the exception happens in our Sphinx extension, not in Sphinx
>>>>>> itself.  We've had other non-ascii text in our docs, so I think Sphinx is
>>>>>> doing the right thing.  The problem is with our own code.  If I could
>>>>>> reproduce it, it shouldn't be too hard to track down - take out that
>>>>>> massive "except anything" block and see where it explodes.
>>>>>>
>>>>>> Randy, which distribution are you running, and are you using their version
>>>>>> of Sphinx?
>>>>>
>>>>> opensuse LEAP 42.2
>>>>> Yes, their sphinx 1.3.1.
>>>>
>>>> What's your LANG setting? I think that's what it boils down to, and
>>>> trying to work around non-UTF-8 LANG in both python 2 and 3 compatible
>>>> ways.
>>>>
>>>> The odd thing is that I can reproduce the issue using a small python
>>>> snippet, but not through Sphinx.
>>>
>>> Your original error message suggests your Sphinx actually uses python
>>> 3. Can you check that? The clue is that it's the *decode* that fails.
>>
>> Where do you see that clue?
> 
> The message, "'ascii' codec can't decode byte 0xe2 in position 6368:
> ordinal not in range(128)". In my testing I could only get that *decode*
> error message using python 3.
> 
>> My /usr/bin/python is linked to python2.7:
>>
>>> ll /usr/bin/python
>> lrwxrwxrwx 1 root root 9 Jun 10 19:59 /usr/bin/python -> python2.7*
> 
> Sure, but how about 'head $(which sphinx-build)'?

You are correct:

#!/usr/bin/python3

> I could be completely mistaken too. ;)


-- 
~Randy