[PATCH v2 00/11] Documentation: Add ABI to the admin guide

[PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Mauro Carvalho Chehab]

That's the third attempt to add support for the Kernel ABI
at the Documentation's admin guide.

The first approach was based on a generic extension that
calls a random script. This one is based on a new Sphinx
extension with adds a symbol specific for parsing ABI
symbols.

It adds a new script (scripts/get_abi.pl) with can either
search for ABI symbols that match a regular expression or
outputs the entire documentation found inside a directory
as a ReST book.

Adding the ABI description to the Linux documentation is
as easy as adding a tab like:

	.. kernel-abi:: $srctree/Documentation/ABI/obsolete

On version 3, I improved the script to better parse the
ABI descriptions. On this version, it will identify if the
description can be handled as a normal text. If so, it will
escape characters that has special meaning on Sphinx.
If it detects that the description syntax is too complex,
it outputs inside a literal block.

Also on version 3, I opted to use 4 separate sections, one
for each type of symbol. It also add an entry with the name
of each ABI file, with has cross-references to the symbols
defined on each file.

The parser was also improved to handle any content at the
files that are before the ABI tags.

The first 3 patches on this series fix some syntax errors
on some ABI description.

The next one makes sure that, when a table is present on
the description, the preceding line will end with a colon
(with is one of the ways the script uses to identify
complex layouts).

-

IMHO, we should add another ABI tag to allow using an
enriched text description, e. g. instead of:

Description:  foo

One could use, instead, something like:

ReST-description: foo

On such case, the script wouldn't need to escape the
description contents, as they can be just sent directly to
the ABI ReST output.

Markus Heiser (1):
  doc-rst: customize RTD theme; literal-block

Mauro Carvalho Chehab (10):
  ABI: fix some syntax issues at the ABI database
  ABI: sysfs-driver-hid: the "What" field doesn't parse fine
  ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry
  ABI: better identificate tables
  scripts: add an script to parse the ABI files
  scripts/get_abi.pl: parse files with text at beginning
  scripts/get_abi.pl: avoid use literal blocks when not needed
  scripts/get_abi.pl: split label naming from xref logic
  scripts/get_abi.pl: add support for searching for ABI symbols
  doc-rst: add ABI documentation to the admin-guide book

 .../ABI/obsolete/sysfs-driver-hid-roccat-pyra      |   2 +-
 Documentation/ABI/testing/pstore                   |   2 +-
 .../testing/sysfs-bus-event_source-devices-format  |   2 +-
 .../ABI/testing/sysfs-bus-i2c-devices-hm6352       |   6 +-
 .../ABI/testing/sysfs-bus-iio-distance-srf08       |   4 +-
 .../ABI/testing/sysfs-bus-iio-proximity-as3935     |   4 +-
 .../ABI/testing/sysfs-bus-pci-devices-cciss        |  22 +-
 .../ABI/testing/sysfs-bus-usb-devices-usbsevseg    |  12 +-
 .../testing/sysfs-class-backlight-driver-lm3533    |   6 +-
 Documentation/ABI/testing/sysfs-class-cxl          |   6 +-
 Documentation/ABI/testing/sysfs-class-devfreq      |   2 +-
 .../ABI/testing/sysfs-class-led-driver-lm3533      |   8 +-
 Documentation/ABI/testing/sysfs-class-leds-gt683r  |   4 +-
 Documentation/ABI/testing/sysfs-class-powercap     |   2 +-
 Documentation/ABI/testing/sysfs-class-uwb_rc       |   6 -
 Documentation/ABI/testing/sysfs-driver-hid         |  12 +-
 .../ABI/testing/sysfs-driver-hid-roccat-kone       |   2 +-
 Documentation/ABI/testing/sysfs-kernel-fscaps      |   2 +-
 Documentation/ABI/testing/sysfs-kernel-vmcoreinfo  |   2 +-
 Documentation/admin-guide/abi-obsolete.rst         |  10 +
 Documentation/admin-guide/abi-removed.rst          |   4 +
 Documentation/admin-guide/abi-stable.rst           |  13 +
 Documentation/admin-guide/abi-testing.rst          |  19 +
 Documentation/admin-guide/abi.rst                  |  11 +
 Documentation/admin-guide/index.rst                |   1 +
 Documentation/conf.py                              |   3 +-
 Documentation/sphinx-static/theme_overrides.css    |   7 +
 Documentation/sphinx/kernel_abi.py                 | 155 ++++++++
 scripts/get_abi.pl                                 | 419 +++++++++++++++++++++
 29 files changed, 691 insertions(+), 57 deletions(-)
 create mode 100644 Documentation/admin-guide/abi-obsolete.rst
 create mode 100644 Documentation/admin-guide/abi-removed.rst
 create mode 100644 Documentation/admin-guide/abi-stable.rst
 create mode 100644 Documentation/admin-guide/abi-testing.rst
 create mode 100644 Documentation/admin-guide/abi.rst
 create mode 100644 Documentation/sphinx/kernel_abi.py
 create mode 100755 scripts/get_abi.pl

-- 
2.9.3

[PATCH v2 01/11] doc-rst: customize RTD theme; literal-block

[Mauro Carvalho Chehab]

From: Markus Heiser <markus.heiser@darmarIT.de>

From: Markus Heiser <markus.heiser@darmarIT.de>

Format the literal-block like other code-block elements, with 12px and a
line-high of 1.5.

Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/sphinx-static/theme_overrides.css | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Documentation/sphinx-static/theme_overrides.css b/Documentation/sphinx-static/theme_overrides.css
index d5764a4de5a2..7033d4c05e42 100644
--- a/Documentation/sphinx-static/theme_overrides.css
+++ b/Documentation/sphinx-static/theme_overrides.css
@@ -69,4 +69,11 @@
     .rst-content tt.literal,.rst-content tt.literal,.rst-content code.literal {
         color: inherit;
     }
+
+    /* literal blocks (e.g. parsed-literal directive) */
+
+    pre.literal-block {
+        font-size: 12px;
+        line-height: 1.5;
+    }
 }
-- 
2.9.3

[PATCH v2 02/11] ABI: fix some syntax issues at the ABI database

[Mauro Carvalho Chehab]

On those three files, the ABI representation described at
README are violated.

- at sysfs-bus-iio-proximity-as3935:
	a ':' character is missing after "What"

- at sysfs-class-devfreq:
	there's a typo at Description

- at sysfs-class-cxl, it is using the ":" character at a
	file preamble, causing it to be misinterpreted as a
	tag.

- On the other files, instead of "What", they use "Where".

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/ABI/testing/pstore                   |  2 +-
 .../testing/sysfs-bus-event_source-devices-format  |  2 +-
 .../ABI/testing/sysfs-bus-i2c-devices-hm6352       |  6 +++---
 .../ABI/testing/sysfs-bus-iio-distance-srf08       |  4 ++--
 .../ABI/testing/sysfs-bus-iio-proximity-as3935     |  4 ++--
 .../ABI/testing/sysfs-bus-pci-devices-cciss        | 22 +++++++++++-----------
 .../ABI/testing/sysfs-bus-usb-devices-usbsevseg    | 12 ++++++------
 Documentation/ABI/testing/sysfs-class-cxl          |  6 +++---
 Documentation/ABI/testing/sysfs-class-devfreq      |  2 +-
 Documentation/ABI/testing/sysfs-class-powercap     |  2 +-
 Documentation/ABI/testing/sysfs-kernel-fscaps      |  2 +-
 Documentation/ABI/testing/sysfs-kernel-vmcoreinfo  |  2 +-
 12 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/Documentation/ABI/testing/pstore b/Documentation/ABI/testing/pstore
index 5fca9f5e10a3..8d6e48f4e8ef 100644
--- a/Documentation/ABI/testing/pstore
+++ b/Documentation/ABI/testing/pstore
@@ -1,4 +1,4 @@
-Where:		/sys/fs/pstore/... (or /dev/pstore/...)
+What:		/sys/fs/pstore/... (or /dev/pstore/...)
 Date:		March 2011
 Kernel Version: 2.6.39
 Contact:	tony.luck@intel.com
diff --git a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
index 77f47ff5ee02..b6f8748e0200 100644
--- a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
+++ b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
@@ -1,4 +1,4 @@
-Where:		/sys/bus/event_source/devices/<dev>/format
+What:		/sys/bus/event_source/devices/<dev>/format
 Date:		January 2012
 Kernel Version: 3.3
 Contact:	Jiri Olsa <jolsa@redhat.com>
diff --git a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352 b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
index feb2e4a87075..29bd447e50a0 100644
--- a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
+++ b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
@@ -1,18 +1,18 @@
-Where:		/sys/bus/i2c/devices/.../heading0_input
+What:		/sys/bus/i2c/devices/.../heading0_input
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
 Description:	Reports the current heading from the compass as a floating
 		point value in degrees.
 
-Where:		/sys/bus/i2c/devices/.../power_state
+What:		/sys/bus/i2c/devices/.../power_state
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
 Description:	Sets the power state of the device. 0 sets the device into
 		sleep mode, 1 wakes it up.
 
-Where:		/sys/bus/i2c/devices/.../calibration
+What:		/sys/bus/i2c/devices/.../calibration
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
diff --git a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08 b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
index 0a1ca1487fa9..a133fd8d081a 100644
--- a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
+++ b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
@@ -1,4 +1,4 @@
-What		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
 Date:		January 2017
 KernelVersion:	4.11
 Contact:	linux-iio@vger.kernel.org
@@ -6,7 +6,7 @@ Description:
 		Show or set the gain boost of the amp, from 0-31 range.
 		default 31
 
-What		/sys/bus/iio/devices/iio:deviceX/sensor_max_range
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_max_range
 Date:		January 2017
 KernelVersion:	4.11
 Contact:	linux-iio@vger.kernel.org
diff --git a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935 b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
index 33e96f740639..61a3c9fed07d 100644
--- a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
+++ b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
@@ -1,4 +1,4 @@
-What		/sys/bus/iio/devices/iio:deviceX/in_proximity_input
+What:		/sys/bus/iio/devices/iio:deviceX/in_proximity_input
 Date:		March 2014
 KernelVersion:	3.15
 Contact:	Matt Ranostay <mranostay@gmail.com>
@@ -6,7 +6,7 @@ Description:
 		Get the current distance in meters of storm (1km steps)
 		1000-40000 = distance in meters
 
-What		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
 Date:		March 2014
 KernelVersion:	3.15
 Contact:	Matt Ranostay <mranostay@gmail.com>
diff --git a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
index 53d99edd1d75..eb449169c30b 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
+++ b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
@@ -1,66 +1,66 @@
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/model
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/model
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 model for logical drive
 		Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 revision for logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 83 serial number for logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 vendor for logical drive
 		Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	A symbolic link to /sys/block/cciss!cXdY
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/rescan
+What:		/sys/bus/pci/devices/<dev>/ccissX/rescan
 Date:		August 2009
 Kernel Version:	2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Kicks of a rescan of the controller to discover logical
 		drive topology changes.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the 8-byte LUN ID used to address logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the RAID level of logical drive Y of
 		controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the usage count (number of opens) of logical drive Y
 		of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/resettable
+What:		/sys/bus/pci/devices/<dev>/ccissX/resettable
 Date:		February 2011
 Kernel Version:	2.6.38
 Contact:	iss_storagedev@hp.com
@@ -71,7 +71,7 @@ Description:	Value of 1 indicates the controller can honor the reset_devices
 		a dump device, as kdump requires resetting the device in order
 		to work reliably.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/transport_mode
+What:		/sys/bus/pci/devices/<dev>/ccissX/transport_mode
 Date:		July 2011
 Kernel Version:	3.0
 Contact:	iss_storagedev@hp.com
diff --git a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
index 70d00dfa443d..f6199b314196 100644
--- a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
+++ b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
@@ -1,12 +1,12 @@
-Where:		/sys/bus/usb/.../powered
+What:		/sys/bus/usb/.../powered
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
 Description:	Controls whether the device's display will powered.
 		A value of 0 is off and a non-zero value is on.
 
-Where:		/sys/bus/usb/.../mode_msb
-Where:		/sys/bus/usb/.../mode_lsb
+What:		/sys/bus/usb/.../mode_msb
+What:		/sys/bus/usb/.../mode_lsb
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
@@ -16,7 +16,7 @@ Description:	Controls the devices display mode.
 		for an 8 character display the values are
 			MSB 0x08; LSB 0xFF.
 
-Where:		/sys/bus/usb/.../textmode
+What:		/sys/bus/usb/.../textmode
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
@@ -25,13 +25,13 @@ Description:	Controls the way the device interprets its text buffer.
 		hex:	each character is between 0-15
 		ascii:	each character is between '0'-'9' and 'A'-'F'.
 
-Where:		/sys/bus/usb/.../text
+What:		/sys/bus/usb/.../text
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
 Description:	The text (or data) for the device to display
 
-Where:		/sys/bus/usb/.../decimals
+What:		/sys/bus/usb/.../decimals
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
index 640f65e79ef1..d0b32452dfe1 100644
--- a/Documentation/ABI/testing/sysfs-class-cxl
+++ b/Documentation/ABI/testing/sysfs-class-cxl
@@ -1,6 +1,6 @@
-Note: Attributes that are shared between devices are stored in the directory
-pointed to by the symlink device/.
-Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
+Please notice that attributes that are shared between devices are stored in
+the directory pointed to by the symlink device/.
+For example, the real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
 /sys/class/cxl/afu0.0s/device/irqs_max, i.e. /sys/class/cxl/afu0.0/irqs_max.
 
 
diff --git a/Documentation/ABI/testing/sysfs-class-devfreq b/Documentation/ABI/testing/sysfs-class-devfreq
index ee39acacf6f8..01196e19afca 100644
--- a/Documentation/ABI/testing/sysfs-class-devfreq
+++ b/Documentation/ABI/testing/sysfs-class-devfreq
@@ -47,7 +47,7 @@ Description:
 What:		/sys/class/devfreq/.../trans_stat
 Date:		October 2012
 Contact:	MyungJoo Ham <myungjoo.ham@samsung.com>
-Descrtiption:
+Description:
 		This ABI shows the statistics of devfreq behavior on a
 		specific device. It shows the time spent in each state and
 		the number of transitions between states.
diff --git a/Documentation/ABI/testing/sysfs-class-powercap b/Documentation/ABI/testing/sysfs-class-powercap
index db3b3ff70d84..f333a0ccc29b 100644
--- a/Documentation/ABI/testing/sysfs-class-powercap
+++ b/Documentation/ABI/testing/sysfs-class-powercap
@@ -147,6 +147,6 @@ What:		/sys/class/powercap/.../<power zone>/enabled
 Date:		September 2013
 KernelVersion:	3.13
 Contact:	linux-pm@vger.kernel.org
-Description
+Description:
 		This allows to enable/disable power capping at power zone level.
 		This applies to current power zone and its children.
diff --git a/Documentation/ABI/testing/sysfs-kernel-fscaps b/Documentation/ABI/testing/sysfs-kernel-fscaps
index 50a3033b5e15..bcff34665192 100644
--- a/Documentation/ABI/testing/sysfs-kernel-fscaps
+++ b/Documentation/ABI/testing/sysfs-kernel-fscaps
@@ -2,7 +2,7 @@ What:		/sys/kernel/fscaps
 Date:		February 2011
 KernelVersion:	2.6.38
 Contact:	Ludwig Nussel <ludwig.nussel@suse.de>
-Description
+Description:
 		Shows whether file system capabilities are honored
 		when executing a binary
 
diff --git a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
index 7bd81168e063..1f1087a5f075 100644
--- a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
+++ b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
@@ -4,7 +4,7 @@ KernelVersion:	2.6.24
 Contact:	Ken'ichi Ohmichi <oomichi@mxs.nes.nec.co.jp>
 		Kexec Mailing List <kexec@lists.infradead.org>
 		Vivek Goyal <vgoyal@redhat.com>
-Description
+Description:
 		Shows physical address and size of vmcoreinfo ELF note.
 		First value contains physical address of note in hex and
 		second value contains the size of note in hex. This ELF
-- 
2.9.3

[PATCH v2 03/11] ABI: sysfs-driver-hid: the "What" field doesn't parse fine

[Mauro Carvalho Chehab]

The What: field on this ABI description use a different
syntax than expected, causing it to not be properly parsed
by script.

Fix it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/ABI/testing/sysfs-driver-hid | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-hid b/Documentation/ABI/testing/sysfs-driver-hid
index 48942cacb0bf..a59533410871 100644
--- a/Documentation/ABI/testing/sysfs-driver-hid
+++ b/Documentation/ABI/testing/sysfs-driver-hid
@@ -1,6 +1,6 @@
-What:		For USB devices	: /sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
-		For BT devices	: /sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
-		Symlink		: /sys/class/hidraw/hidraw<num>/device/report_descriptor
+What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
+What:		/sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
+What:		/sys/class/hidraw/hidraw<num>/device/report_descriptor
 Date:		Jan 2011
 KernelVersion:	2.0.39
 Contact:	Alan Ott <alan@signal11.us>
@@ -9,9 +9,9 @@ Description:	When read, this file returns the device's raw binary HID
 		This file cannot be written.
 Users:		HIDAPI library (http://www.signal11.us/oss/hidapi)
 
-What:		For USB devices	: /sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
-		For BT devices	: /sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
-		Symlink		: /sys/class/hidraw/hidraw<num>/device/country
+What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
+What:		/sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
+What:		/sys/class/hidraw/hidraw<num>/device/country
 Date:		February 2015
 KernelVersion:	3.19
 Contact:	Olivier Gay <ogay@logitech.com>
-- 
2.9.3

[PATCH v2 04/11] ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry

[Mauro Carvalho Chehab]

There are two entries for /sys/class/uwb_rc/uwbN/<EUI-48>/BPST.
The second one has a missing description.

Get rid of it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/ABI/testing/sysfs-class-uwb_rc | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-class-uwb_rc b/Documentation/ABI/testing/sysfs-class-uwb_rc
index 85f4875d16ac..a0578751c1e3 100644
--- a/Documentation/ABI/testing/sysfs-class-uwb_rc
+++ b/Documentation/ABI/testing/sysfs-class-uwb_rc
@@ -125,12 +125,6 @@ Description:
                 The EUI-48 of this device in colon separated hex
                 octets.
 
-What:           /sys/class/uwb_rc/uwbN/<EUI-48>/BPST
-Date:           July 2008
-KernelVersion:  2.6.27
-Contact:        linux-usb@vger.kernel.org
-Description:
-
 What:           /sys/class/uwb_rc/uwbN/<EUI-48>/IEs
 Date:           July 2008
 KernelVersion:  2.6.27
-- 
2.9.3

[PATCH v2 05/11] ABI: better identificate tables

[Mauro Carvalho Chehab]

When parsing via script, it is important to know if the script
should consider a description as a literal block that should
be displayed as-is, or if the description can be considered
as a normal text.

Change descriptions to ensure that the preceding line of a table
ends with a colon. That makes easy to identify the need of a
literal block.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra       | 2 +-
 Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533 | 6 +++---
 Documentation/ABI/testing/sysfs-class-led-driver-lm3533       | 8 ++++----
 Documentation/ABI/testing/sysfs-class-leds-gt683r             | 4 ++--
 Documentation/ABI/testing/sysfs-driver-hid-roccat-kone        | 2 +-
 5 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
index 16020b31ae64..5d41ebadf15e 100644
--- a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
+++ b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
@@ -5,7 +5,7 @@ Description:	It is possible to switch the cpi setting of the mouse with the
 		press of a button.
 		When read, this file returns the raw number of the actual cpi
 		setting reported by the mouse. This number has to be further
-		processed to receive the real dpi value.
+		processed to receive the real dpi value:
 
 		VALUE DPI
 		1     400
diff --git a/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533 b/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
index 77cf7ac949af..c0e0a9ae7b3d 100644
--- a/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
+++ b/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
@@ -4,7 +4,7 @@ KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
 		Get the ALS output channel used as input in
-		ALS-current-control mode (0, 1), where
+		ALS-current-control mode (0, 1), where:
 
 		0 - out_current0 (backlight 0)
 		1 - out_current1 (backlight 1)
@@ -28,7 +28,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the brightness-mapping mode (0, 1), where
+		Set the brightness-mapping mode (0, 1), where:
 
 		0 - exponential mode
 		1 - linear mode
@@ -38,7 +38,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the PWM-input control mask (5 bits), where
+		Set the PWM-input control mask (5 bits), where:
 
 		bit 5 - PWM-input enabled in Zone 4
 		bit 4 - PWM-input enabled in Zone 3
diff --git a/Documentation/ABI/testing/sysfs-class-led-driver-lm3533 b/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
index 620ebb3b9baa..e4c89b261546 100644
--- a/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
+++ b/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
@@ -4,7 +4,7 @@ KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
 		Set the ALS output channel to use as input in
-		ALS-current-control mode (1, 2), where
+		ALS-current-control mode (1, 2), where:
 
 		1 - out_current1
 		2 - out_current2
@@ -22,7 +22,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the pattern generator fall and rise times (0..7), where
+		Set the pattern generator fall and rise times (0..7), where:
 
 		0 - 2048 us
 		1 - 262 ms
@@ -45,7 +45,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the brightness-mapping mode (0, 1), where
+		Set the brightness-mapping mode (0, 1), where:
 
 		0 - exponential mode
 		1 - linear mode
@@ -55,7 +55,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the PWM-input control mask (5 bits), where
+		Set the PWM-input control mask (5 bits), where:
 
 		bit 5 - PWM-input enabled in Zone 4
 		bit 4 - PWM-input enabled in Zone 3
diff --git a/Documentation/ABI/testing/sysfs-class-leds-gt683r b/Documentation/ABI/testing/sysfs-class-leds-gt683r
index e4fae6026e79..6adab27f646e 100644
--- a/Documentation/ABI/testing/sysfs-class-leds-gt683r
+++ b/Documentation/ABI/testing/sysfs-class-leds-gt683r
@@ -5,7 +5,7 @@ Contact:	Janne Kanniainen <janne.kanniainen@gmail.com>
 Description:
 		Set the mode of LEDs. You should notice that changing the mode
 		of one LED will update the mode of its two sibling devices as
-		well.
+		well. Possible values are:
 
 		0 - normal
 		1 - audio
@@ -13,4 +13,4 @@ Description:
 
 		Normal: LEDs are fully on when enabled
 		Audio:  LEDs brightness depends on sound level
-		Breathing: LEDs brightness varies at human breathing rate
\ No newline at end of file
+		Breathing: LEDs brightness varies at human breathing rate
diff --git a/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone b/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
index 3ca3971109bf..8f7982c70d72 100644
--- a/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
+++ b/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
@@ -5,7 +5,7 @@ Description:	It is possible to switch the dpi setting of the mouse with the
 		press of a button.
 		When read, this file returns the raw number of the actual dpi
 		setting reported by the mouse. This number has to be further
-		processed to receive the real dpi value.
+		processed to receive the real dpi value:
 
 		VALUE DPI
 		1     800
-- 
2.9.3

[PATCH v2 06/11] scripts: add an script to parse the ABI files

[Mauro Carvalho Chehab]

Add a script to parse the Documentation/ABI files and produce
an output with all entries inside an ABI (sub)directory.

Right now, it outputs its contents on ReST format. It shouldn't
be hard to make it produce other kind of outputs, since the ABI
file parser is implemented in separate than the output generator.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 scripts/get_abi.pl | 212 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 212 insertions(+)
 create mode 100755 scripts/get_abi.pl

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
new file mode 100755
index 000000000000..f7c9944a833c
--- /dev/null
+++ b/scripts/get_abi.pl
@@ -0,0 +1,212 @@
+#!/usr/bin/perl
+
+use strict;
+use Pod::Usage;
+use Getopt::Long;
+use File::Find;
+use Fcntl ':mode';
+
+my $help;
+my $man;
+my $debug;
+
+GetOptions(
+	"debug|d+" => \$debug,
+	'help|?' => \$help,
+	man => \$man
+) or pod2usage(2);
+
+pod2usage(1) if $help;
+pod2usage(-exitstatus => 0, -verbose => 2) if $man;
+
+pod2usage(2) if (scalar @ARGV != 1);
+
+my ($prefix) = @ARGV;
+
+require Data::Dumper if ($debug);
+
+my %data;
+
+#
+# Displays an error message, printing file name and line
+#
+sub parse_error($$$$) {
+	my ($file, $ln, $msg, $data) = @_;
+
+	print STDERR "file $file#$ln: $msg at\n\t$data";
+}
+
+#
+# Parse an ABI file, storing its contents at %data
+#
+sub parse_abi {
+	my $file = $File::Find::name;
+
+	my $mode = (stat($file))[2];
+	return if ($mode & S_IFDIR);
+	return if ($file =~ m,/README,);
+
+	my $name = $file;
+	$name =~ s,.*/,,;
+
+	my $type = $file;
+	$type =~ s,.*/(.*)/.*,$1,;
+
+	my $what;
+	my $new_what;
+	my $tag;
+	my $ln;
+
+	print STDERR "Opening $file\n" if ($debug > 1);
+	open IN, $file;
+	while(<IN>) {
+		$ln++;
+		if (m/^(\S+):\s*(.*)/i) {
+			my $new_tag = lc($1);
+			my $content = $2;
+
+			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
+				if ($tag eq "description") {
+					$data{$what}->{$tag} .= "\n$content";;
+					$data{$what}->{$tag} =~ s/\n+$//;
+					next;
+				} else {
+					parse_error($file, $ln, "tag '$tag' is invalid", $_);
+				}
+			}
+
+			if ($new_tag =~ m/what/) {
+				if ($tag =~ m/what/) {
+					$what .= ", " . $content;
+				} else {
+					$what = $content;
+					$new_what = 1;
+				}
+				$tag = $new_tag;
+				next;
+			}
+
+			$tag = $new_tag;
+
+			if ($new_what) {
+				$new_what = 0;
+
+				$data{$what}->{type} = $type;
+				$data{$what}->{file} = $name;
+				print STDERR "\twhat: $what\n" if ($debug > 1);
+			}
+
+			if (!$what) {
+				parse_error($file, $ln, "'What:' should come first:", $_);
+				next;
+			}
+			$data{$what}->{$tag} = $content;
+			next;
+		}
+
+		# Silently ignore any headers before the database
+		next if (!$tag);
+
+		if (m/^\s*(.*)/) {
+			$data{$what}->{$tag} .= "\n$1";
+			$data{$what}->{$tag} =~ s/\n+$//;
+			next;
+		}
+
+		# Everything else is error
+		parse_error($file, $ln, "Unexpected line:", $_);
+	}
+	close IN;
+}
+
+# Outputs the output on ReST format
+sub output_rest {
+	foreach my $what (sort keys %data) {
+		my $type = $data{$what}->{type};
+		my $file = $data{$what}->{file};
+
+		my $w = $what;
+		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
+
+		print "$w\n\n";
+		print "- defined on file $file (type: $type)\n\n::\n\n";
+
+		my $desc = $data{$what}->{description};
+		$desc =~ s/^\s+//;
+
+		# Remove title markups from the description, as they won't work
+		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
+
+		# put everything inside a code block
+		$desc =~ s/\n/\n /g;
+
+
+		if (!($desc =~ /^\s*$/)) {
+			print " $desc\n\n";
+		} else {
+			print " DESCRIPTION MISSING\n\n";
+		}
+	}
+}
+
+#
+# Parses all ABI files located at $prefix dir
+#
+find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
+
+print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
+
+#
+# Outputs an ReST file with the ABI contents
+#
+output_rest
+
+
+__END__
+
+=head1 NAME
+
+abi_book.pl - parse the Linux ABI files and produce a ReST book.
+
+=head1 SYNOPSIS
+
+B<abi_book.pl> [--debug] <ABI_DIR>]
+
+=head1 OPTIONS
+
+=over 8
+
+=item B<--debug>
+
+Put the script in verbose mode, useful for debugging. Can be called multiple
+times, to increase verbosity.
+
+=item B<--help>
+
+Prints a brief help message and exits.
+
+=item B<--man>
+
+Prints the manual page and exits.
+
+=back
+
+=head1 DESCRIPTION
+
+Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI)
+and produce a ReST book containing the Linux ABI.
+
+=head1 BUGS
+
+Report bugs to Mauro Carvalho Chehab <mchehab@s-opensource.com>
+
+=head1 COPYRIGHT
+
+Copyright (c) 2016 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
+
+License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
+
+This is free software: you are free to change and redistribute it.
+There is NO WARRANTY, to the extent permitted by law.
+
+=cut
-- 
2.9.3

[PATCH v2 07/11] scripts/get_abi.pl: parse files with text at beginning

[Mauro Carvalho Chehab]

It sounds usefult o parse files with has some text at the
beginning. Add support for it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 scripts/get_abi.pl | 59 +++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 54 insertions(+), 5 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index f7c9944a833c..ac0f057fa3ca 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -55,7 +55,10 @@ sub parse_abi {
 	my $what;
 	my $new_what;
 	my $tag;
+	my $label;
 	my $ln;
+	my $has_file;
+	my $xrefs;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
@@ -67,7 +70,7 @@ sub parse_abi {
 
 			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
 				if ($tag eq "description") {
-					$data{$what}->{$tag} .= "\n$content";;
+					$data{$what}->{$tag} .= "\n$content";
 					$data{$what}->{$tag} =~ s/\n+$//;
 					next;
 				} else {
@@ -83,6 +86,25 @@ sub parse_abi {
 					$new_what = 1;
 				}
 				$tag = $new_tag;
+
+				if ($has_file) {
+					$label = "abi_" . $content . " ";
+					$label =~ tr/A-Z/a-z/;
+
+					# Convert special chars to "_"
+					$label =~s/[\x00-\x2f]+/_/g;
+					$label =~s/[\x3a-\x40]+/_/g;
+					$label =~s/[\x7b-\xff]+/_/g;
+					$label =~ s,_+,_,g;
+					$label =~ s,_$,,;
+
+					$data{$what}->{label} .= $label;
+
+					# Escape special chars from content
+					$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
+
+					$xrefs .= "- :ref:`$content <$label>`\n\n";
+				}
 				next;
 			}
 
@@ -104,8 +126,18 @@ sub parse_abi {
 			next;
 		}
 
-		# Silently ignore any headers before the database
-		next if (!$tag);
+		# Store any contents before the database
+		if (!$tag) {
+			next if (/^\n/);
+
+			my $my_what = "File $name";
+			$data{$my_what}->{what} = "File $name";
+			$data{$my_what}->{type} = "File";
+			$data{$my_what}->{file} = $name;
+			$data{$my_what}->{description} .= $_;
+			$has_file = 1;
+			next;
+		}
 
 		if (m/^\s*(.*)/) {
 			$data{$what}->{$tag} .= "\n$1";
@@ -117,6 +149,11 @@ sub parse_abi {
 		parse_error($file, $ln, "Unexpected line:", $_);
 	}
 	close IN;
+
+	if ($has_file) {
+		my $my_what = "File $name";
+                $data{$my_what}->{xrefs} = $xrefs;
+	}
 }
 
 # Outputs the output on ReST format
@@ -128,8 +165,17 @@ sub output_rest {
 		my $w = $what;
 		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
 
+		if ($data{$what}->{label}) {
+			my @labels = split(/\s/, $data{$what}->{label});
+			foreach my $label (@labels) {
+				printf ".. _%s:\n\n", $label;
+			}
+		}
+
 		print "$w\n\n";
-		print "- defined on file $file (type: $type)\n\n::\n\n";
+
+		print "- defined on file $file (type: $type)\n\n" if ($type ne "File");
+		print "::\n\n";
 
 		my $desc = $data{$what}->{description};
 		$desc =~ s/^\s+//;
@@ -144,8 +190,11 @@ sub output_rest {
 		if (!($desc =~ /^\s*$/)) {
 			print " $desc\n\n";
 		} else {
-			print " DESCRIPTION MISSING\n\n";
+			print " DESCRIPTION MISSING for $what\n\n";
 		}
+
+		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
+
 	}
 }
 
-- 
2.9.3

[PATCH v2 08/11] scripts/get_abi.pl: avoid use literal blocks when not needed

[Mauro Carvalho Chehab]

The usage of literal blocks make the document very complex,
causing the browser to take a long time to load.

On most ABI descriptions, they're a plain text, and don't
require a literal block.

So, add a logic there with identifies when a literal block
is needed.

As, on literal blocks, we need to respect the original
document space, the most complex part of this patch is
to preserve the original spacing where needed.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 scripts/get_abi.pl | 104 +++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 78 insertions(+), 26 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index ac0f057fa3ca..ba8a7466f896 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -59,29 +59,34 @@ sub parse_abi {
 	my $ln;
 	my $has_file;
 	my $xrefs;
+	my $space;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
 	while(<IN>) {
 		$ln++;
-		if (m/^(\S+):\s*(.*)/i) {
+		if (m/^(\S+)(:\s*)(.*)/i) {
 			my $new_tag = lc($1);
-			my $content = $2;
+			my $sep = $2;
+			my $content = $3;
 
 			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
 				if ($tag eq "description") {
-					$data{$what}->{$tag} .= "\n$content";
-					$data{$what}->{$tag} =~ s/\n+$//;
-					next;
+					# New "tag" is actually part of
+					# description. Don't consider it a tag
+					$new_tag = "";
 				} else {
 					parse_error($file, $ln, "tag '$tag' is invalid", $_);
 				}
 			}
 
 			if ($new_tag =~ m/what/) {
+				$space = "";
 				if ($tag =~ m/what/) {
 					$what .= ", " . $content;
 				} else {
+					parse_error($file, $ln, "What '$what' doesn't have a description", "") if ($what && !$data{$what}->{description});
+
 					$what = $content;
 					$new_what = 1;
 				}
@@ -108,25 +113,38 @@ sub parse_abi {
 				next;
 			}
 
-			$tag = $new_tag;
+			if ($new_tag) {
+				$tag = $new_tag;
 
-			if ($new_what) {
-				$new_what = 0;
+				if ($new_what) {
+					$new_what = 0;
 
-				$data{$what}->{type} = $type;
-				$data{$what}->{file} = $name;
-				print STDERR "\twhat: $what\n" if ($debug > 1);
-			}
+					$data{$what}->{type} = $type;
+					$data{$what}->{file} = $name;
+					print STDERR "\twhat: $what\n" if ($debug > 1);
+				}
 
-			if (!$what) {
-				parse_error($file, $ln, "'What:' should come first:", $_);
+				if (!$what) {
+					parse_error($file, $ln, "'What:' should come first:", $_);
+					next;
+				}
+				if ($tag eq "description") {
+					next if ($content =~ m/^\s*$/);
+					if ($content =~ m/^(\s*)(.*)/) {
+						my $new_content = $2;
+						$space = $new_tag . $sep . $1;
+						while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+						$space =~ s/./ /g;
+						$data{$what}->{$tag} .= "$new_content\n";
+					}
+				} else {
+					$data{$what}->{$tag} = $content;
+				}
 				next;
 			}
-			$data{$what}->{$tag} = $content;
-			next;
 		}
 
-		# Store any contents before the database
+		# Store any contents before tags at the database
 		if (!$tag) {
 			next if (/^\n/);
 
@@ -139,6 +157,32 @@ sub parse_abi {
 			next;
 		}
 
+		if ($tag eq "description") {
+			if (!$data{$what}->{description}) {
+				next if (m/^\s*\n/);
+				if (m/^(\s*)(.*)/) {
+					$space = $1;
+					while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+					$data{$what}->{$tag} .= "$2\n";
+				}
+			} else {
+				my $content = $_;
+				if (m/^\s*\n/) {
+					$data{$what}->{$tag} .= $content;
+					next;
+				}
+
+				while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+				$space = "" if (!($content =~ s/^($space)//));
+
+				# Compress spaces with tabs
+				$content =~ s<^ {8}> <\t>;
+				$content =~ s<^ {1,7}\t> <\t>;
+				$content =~ s< {1,7}\t> <\t>;
+				$data{$what}->{$tag} .= $content;
+			}
+			next;
+		}
 		if (m/^\s*(.*)/) {
 			$data{$what}->{$tag} .= "\n$1";
 			$data{$what}->{$tag} =~ s/\n+$//;
@@ -165,6 +209,9 @@ sub output_rest {
 		my $w = $what;
 		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
 
+		my $bar = $w;
+		$bar =~ s/./-/g;
+
 		if ($data{$what}->{label}) {
 			my @labels = split(/\s/, $data{$what}->{label});
 			foreach my $label (@labels) {
@@ -172,10 +219,9 @@ sub output_rest {
 			}
 		}
 
-		print "$w\n\n";
+		print "$w\n$bar\n\n";
 
 		print "- defined on file $file (type: $type)\n\n" if ($type ne "File");
-		print "::\n\n";
 
 		my $desc = $data{$what}->{description};
 		$desc =~ s/^\s+//;
@@ -183,18 +229,24 @@ sub output_rest {
 		# Remove title markups from the description, as they won't work
 		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
 
-		# put everything inside a code block
-		$desc =~ s/\n/\n /g;
-
-
 		if (!($desc =~ /^\s*$/)) {
-			print " $desc\n\n";
+	                if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
+				# put everything inside a code block
+				$desc =~ s/\n/\n /g;
+
+				print "::\n\n";
+				print " $desc\n\n";
+			} else {
+				# Escape any special chars from description
+				$desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
+
+				print "$desc\n\n";
+			}
 		} else {
-			print " DESCRIPTION MISSING for $what\n\n";
+			print "DESCRIPTION MISSING for $what\n\n";
 		}
 
 		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
-
 	}
 }
 
-- 
2.9.3

[PATCH v2 09/11] scripts/get_abi.pl: split label naming from xref logic

[Mauro Carvalho Chehab]

Instead of using a ReST compilant label while parsing,
move the label to ReST output. That makes the parsing logic
more generic, allowing it to provide other types of output.

As a side effect, now all files used to generate the output
will be output. We can later add command line arguments to
filter.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 scripts/get_abi.pl | 94 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 53 insertions(+), 41 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index ba8a7466f896..d437e148b1c0 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -49,17 +49,23 @@ sub parse_abi {
 	my $name = $file;
 	$name =~ s,.*/,,;
 
+	my $nametag = "File $name";
+	$data{$nametag}->{what} = "File $name";
+	$data{$nametag}->{type} = "File";
+	$data{$nametag}->{file} = $name;
+	$data{$nametag}->{is_file} = 1;
+
 	my $type = $file;
 	$type =~ s,.*/(.*)/.*,$1,;
 
 	my $what;
 	my $new_what;
 	my $tag;
-	my $label;
 	my $ln;
-	my $has_file;
 	my $xrefs;
 	my $space;
+	my @labels;
+	my $label;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
@@ -88,28 +94,13 @@ sub parse_abi {
 					parse_error($file, $ln, "What '$what' doesn't have a description", "") if ($what && !$data{$what}->{description});
 
 					$what = $content;
+					$label = $content;
 					$new_what = 1;
 				}
+				push @labels, [($content, $label)];
 				$tag = $new_tag;
 
-				if ($has_file) {
-					$label = "abi_" . $content . " ";
-					$label =~ tr/A-Z/a-z/;
-
-					# Convert special chars to "_"
-					$label =~s/[\x00-\x2f]+/_/g;
-					$label =~s/[\x3a-\x40]+/_/g;
-					$label =~s/[\x7b-\xff]+/_/g;
-					$label =~ s,_+,_,g;
-					$label =~ s,_$,,;
-
-					$data{$what}->{label} .= $label;
-
-					# Escape special chars from content
-					$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
-
-					$xrefs .= "- :ref:`$content <$label>`\n\n";
-				}
+				push @{$data{$nametag}->{xrefs}}, [($content, $label)] if ($data{$nametag}->{what});
 				next;
 			}
 
@@ -117,6 +108,9 @@ sub parse_abi {
 				$tag = $new_tag;
 
 				if ($new_what) {
+					@{$data{$what}->{label}} = @labels if ($data{$nametag}->{what});
+					@labels = ();
+					$label = "";
 					$new_what = 0;
 
 					$data{$what}->{type} = $type;
@@ -145,15 +139,8 @@ sub parse_abi {
 		}
 
 		# Store any contents before tags at the database
-		if (!$tag) {
-			next if (/^\n/);
-
-			my $my_what = "File $name";
-			$data{$my_what}->{what} = "File $name";
-			$data{$my_what}->{type} = "File";
-			$data{$my_what}->{file} = $name;
-			$data{$my_what}->{description} .= $_;
-			$has_file = 1;
+		if (!$tag && $data{$nametag}->{what}) {
+			$data{$nametag}->{description} .= $_;
 			next;
 		}
 
@@ -192,12 +179,8 @@ sub parse_abi {
 		# Everything else is error
 		parse_error($file, $ln, "Unexpected line:", $_);
 	}
+	$data{$nametag}->{description} =~ s/^\n+//;
 	close IN;
-
-	if ($has_file) {
-		my $my_what = "File $name";
-                $data{$my_what}->{xrefs} = $xrefs;
-	}
 }
 
 # Outputs the output on ReST format
@@ -212,11 +195,22 @@ sub output_rest {
 		my $bar = $w;
 		$bar =~ s/./-/g;
 
-		if ($data{$what}->{label}) {
-			my @labels = split(/\s/, $data{$what}->{label});
-			foreach my $label (@labels) {
-				printf ".. _%s:\n\n", $label;
-			}
+		foreach my $p (@{$data{$what}->{label}}) {
+			my ($content, $label) = @{$p};
+			$label = "abi_" . $label . " ";
+			$label =~ tr/A-Z/a-z/;
+
+			# Convert special chars to "_"
+			$label =~s/([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff])/_/g;
+			$label =~ s,_+,_,g;
+			$label =~ s,_$,,;
+
+			$data{$what}->{label} .= $label;
+
+			printf ".. _%s:\n\n", $label;
+
+			# only one label is enough
+			last;
 		}
 
 		print "$w\n$bar\n\n";
@@ -243,10 +237,28 @@ sub output_rest {
 				print "$desc\n\n";
 			}
 		} else {
-			print "DESCRIPTION MISSING for $what\n\n";
+			print "DESCRIPTION MISSING for $what\n\n" if (!$data{$what}->{is_file});
 		}
 
-		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
+		if ($data{$what}->{xrefs}) {
+			printf "Has the following ABI:\n\n";
+
+			foreach my $p(@{$data{$what}->{xrefs}}) {
+				my ($content, $label) = @{$p};
+				$label = "abi_" . $label . " ";
+				$label =~ tr/A-Z/a-z/;
+
+				# Convert special chars to "_"
+				$label =~s/([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff])/_/g;
+				$label =~ s,_+,_,g;
+				$label =~ s,_$,,;
+
+				# Escape special chars from content
+				$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
+
+				print "- :ref:`$content <$label>`\n\n";
+			}
+		}
 	}
 }
 
-- 
2.9.3

[PATCH v2 10/11] scripts/get_abi.pl: add support for searching for ABI symbols

[Mauro Carvalho Chehab]

Change its syntax to allow switching between ReST output mode
and a new search mode, with allows to seek for ABI symbols
using rejex.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 scripts/get_abi.pl | 112 ++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 103 insertions(+), 9 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index d437e148b1c0..eb62385630e6 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -9,9 +9,11 @@ use Fcntl ':mode';
 my $help;
 my $man;
 my $debug;
+my $prefix="Documentation/ABI";
 
 GetOptions(
 	"debug|d+" => \$debug,
+	"dir=s" => \$prefix,
 	'help|?' => \$help,
 	man => \$man
 ) or pod2usage(2);
@@ -19,9 +21,12 @@ GetOptions(
 pod2usage(1) if $help;
 pod2usage(-exitstatus => 0, -verbose => 2) if $man;
 
-pod2usage(2) if (scalar @ARGV != 1);
+pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
 
-my ($prefix) = @ARGV;
+my ($cmd, $arg) = @ARGV;
+
+pod2usage(2) if ($cmd ne "search" && $cmd ne "rest");
+pod2usage(2) if ($cmd eq "search" && !$arg);
 
 require Data::Dumper if ($debug);
 
@@ -53,6 +58,7 @@ sub parse_abi {
 	$data{$nametag}->{what} = "File $name";
 	$data{$nametag}->{type} = "File";
 	$data{$nametag}->{file} = $name;
+	$data{$nametag}->{filepath} = $file;
 	$data{$nametag}->{is_file} = 1;
 
 	my $type = $file;
@@ -115,6 +121,7 @@ sub parse_abi {
 
 					$data{$what}->{type} = $type;
 					$data{$what}->{file} = $name;
+					$data{$what}->{filepath} = $file;
 					print STDERR "\twhat: $what\n" if ($debug > 1);
 				}
 
@@ -183,7 +190,9 @@ sub parse_abi {
 	close IN;
 }
 
-# Outputs the output on ReST format
+#
+# Outputs the book on ReST format
+#
 sub output_rest {
 	foreach my $what (sort keys %data) {
 		my $type = $data{$what}->{type};
@@ -263,6 +272,45 @@ sub output_rest {
 }
 
 #
+# Searches for ABI symbols
+#
+sub search_symbols {
+	foreach my $what (sort keys %data) {
+		next if (!($what =~ m/($arg)/));
+
+		my $type = $data{$what}->{type};
+		next if ($type eq "File");
+
+		my $file = $data{$what}->{filepath};
+
+		my $bar = $what;
+		$bar =~ s/./-/g;
+
+		print "\n$what\n$bar\n\n";
+
+		my $kernelversion = $data{$what}->{kernelversion};
+		my $contact = $data{$what}->{contact};
+		my $users = $data{$what}->{users};
+		my $date = $data{$what}->{date};
+		my $desc = $data{$what}->{description};
+		$kernelversion =~ s/^\s+//;
+		$contact =~ s/^\s+//;
+		$users =~ s/^\s+//;
+		$users =~ s/\n//g;
+		$date =~ s/^\s+//;
+		$desc =~ s/^\s+//;
+
+		printf "Kernel version:\t\t%s\n", $kernelversion if ($kernelversion);
+		printf "Date:\t\t\t%s\n", $date if ($date);
+		printf "Contact:\t\t%s\n", $contact if ($contact);
+		printf "Users:\t\t\t%s\n", $users if ($users);
+		print "Defined on file:\t$file\n\n";
+		print "Description:\n\n$desc";
+	}
+}
+
+
+#
 # Parses all ABI files located at $prefix dir
 #
 find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
@@ -270,9 +318,13 @@ find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
 print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
 
 #
-# Outputs an ReST file with the ABI contents
+# Handles the command
 #
-output_rest
+if ($cmd eq "rest") {
+	output_rest;
+} else {
+	search_symbols;
+}
 
 
 __END__
@@ -283,12 +335,27 @@ abi_book.pl - parse the Linux ABI files and produce a ReST book.
 
 =head1 SYNOPSIS
 
-B<abi_book.pl> [--debug] <ABI_DIR>]
+B<abi_book.pl> [--debug] <COMAND> [<ARGUMENT>]
+
+Where <COMMAND> can be:
+
+=over 8
+
+B<search> [SEARCH_REJEX] - search for [SEARCH_REJEX] inside ABI
+
+B<rest>   - output the ABI in ReST markup language
+
+=back
 
 =head1 OPTIONS
 
 =over 8
 
+=item B<--dir>
+
+Changes the location of the ABI search. By default, it uses
+the Documentation/ABI directory.
+
 =item B<--debug>
 
 Put the script in verbose mode, useful for debugging. Can be called multiple
@@ -306,8 +373,35 @@ Prints the manual page and exits.
 
 =head1 DESCRIPTION
 
-Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI)
-and produce a ReST book containing the Linux ABI.
+Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI),
+allowing to search for ABI symbols or to produce a ReST book containing
+the Linux ABI documentation.
+
+=head1 EXAMPLES
+
+Search for all stable symbols with the word "usb":
+
+=over 8
+
+$ scripts/get_abi.pl search usb --dir Documentation/ABI/stable
+
+=back
+
+Search for all symbols that match the rejex expression "usb.*cap":
+
+=over 8
+
+$ scripts/get_abi.pl search usb.*cap
+
+=back
+
+Output all obsoleted symbols in ReST format
+
+=over 8
+
+$ scripts/get_abi.pl rest --dir Documentation/ABI/obsolete
+
+=back
 
 =head1 BUGS
 
@@ -315,7 +409,7 @@ Report bugs to Mauro Carvalho Chehab <mchehab@s-opensource.com>
 
 =head1 COPYRIGHT
 
-Copyright (c) 2016 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
+Copyright (c) 2016-2017 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
 
 License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
 
-- 
2.9.3

[PATCH v2 11/11] doc-rst: add ABI documentation to the admin-guide book

[Mauro Carvalho Chehab]

As we don't want a generic Sphinx extension to execute commands,
change the one proposed to Markus to call the abi_book.pl
script.

Use a script to parse the Documentation/ABI directory and output
it at the admin-guide.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---
 Documentation/admin-guide/abi-obsolete.rst |  10 ++
 Documentation/admin-guide/abi-removed.rst  |   4 +
 Documentation/admin-guide/abi-stable.rst   |  13 +++
 Documentation/admin-guide/abi-testing.rst  |  19 ++++
 Documentation/admin-guide/abi.rst          |  11 ++
 Documentation/admin-guide/index.rst        |   1 +
 Documentation/conf.py                      |   3 +-
 Documentation/sphinx/kernel_abi.py         | 155 +++++++++++++++++++++++++++++
 8 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/admin-guide/abi-obsolete.rst
 create mode 100644 Documentation/admin-guide/abi-removed.rst
 create mode 100644 Documentation/admin-guide/abi-stable.rst
 create mode 100644 Documentation/admin-guide/abi-testing.rst
 create mode 100644 Documentation/admin-guide/abi.rst
 create mode 100644 Documentation/sphinx/kernel_abi.py

diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
new file mode 100644
index 000000000000..cda9168445a5
--- /dev/null
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -0,0 +1,10 @@
+ABI obsolete symbols
+====================
+
+Documents interfaces that are still remaining in the kernel, but are
+marked to be removed at some later point in time.
+
+The description of the interface will document the reason why it is
+obsolete and when it can be expected to be removed.
+
+.. kernel-abi:: $srctree/Documentation/ABI/obsolete
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
new file mode 100644
index 000000000000..497978fc9632
--- /dev/null
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -0,0 +1,4 @@
+ABI removed symbols
+===================
+
+.. kernel-abi:: $srctree/Documentation/ABI/removed
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
new file mode 100644
index 000000000000..7495d7a35048
--- /dev/null
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -0,0 +1,13 @@
+ABI stable symbols
+==================
+
+Documents the interfaces that the developer has defined to be stable.
+
+Userspace programs are free to use these interfaces with no
+restrictions, and backward compatibility for them will be guaranteed
+for at least 2 years.
+
+Most interfaces (like syscalls) are expected to never change and always
+be available.
+
+.. kernel-abi:: $srctree/Documentation/ABI/stable
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
new file mode 100644
index 000000000000..5c886fc50b9e
--- /dev/null
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -0,0 +1,19 @@
+ABI testing symbols
+===================
+
+Documents interfaces that are felt to be stable,
+as the main development of this interface has been completed.
+
+The interface can be changed to add new features, but the
+current interface will not break by doing this, unless grave
+errors or security problems are found in them.
+
+Userspace programs can start to rely on these interfaces, but they must
+be aware of changes that can occur before these interfaces move to
+be marked stable.
+
+Programs that use these interfaces are strongly encouraged to add their
+name to the description of these interfaces, so that the kernel
+developers can easily notify them if any changes occur.
+
+.. kernel-abi:: $srctree/Documentation/ABI/testing
diff --git a/Documentation/admin-guide/abi.rst b/Documentation/admin-guide/abi.rst
new file mode 100644
index 000000000000..3b9645c77469
--- /dev/null
+++ b/Documentation/admin-guide/abi.rst
@@ -0,0 +1,11 @@
+=====================
+Linux ABI description
+=====================
+
+.. toctree::
+   :maxdepth: 1
+
+   abi-stable
+   abi-testing
+   abi-obsolete
+   abi-removed
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 8c60a8a32a1a..da3b10941639 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -16,6 +16,7 @@ etc.
    README
    kernel-parameters
    devices
+   abi
 
 Here is a set of documents aimed at users who are trying to track down
 problems and bugs in particular.
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 08aef4595059..a9449875b79a 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -34,7 +34,8 @@ needs_sphinx = '1.2'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain',
+	      'kfigure', 'kernel_abi']
 
 # The name of the math extension changed on Sphinx 1.4
 if major == 1 and minor > 3:
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
new file mode 100644
index 000000000000..32ce90775d96
--- /dev/null
+++ b/Documentation/sphinx/kernel_abi.py
@@ -0,0 +1,155 @@
+# -*- coding: utf-8; mode: python -*-
+u"""
+    kernel-abi
+    ~~~~~~~~~~
+
+    Implementation of the ``kernel-abi`` reST-directive.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :copyright:  Copyright (C) 2016  Mauro Carvalho Chehab
+    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
+
+    The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
+    scripts/get_abi.pl script to parse the Kernel ABI files.
+
+    Overview of directive's argument and options.
+
+    .. code-block:: rst
+
+        .. kernel-abi:: <ABI directory location>
+            :debug:
+
+    The argument ``<ABI directory location>`` is required. It contains the
+    location of the ABI files to be parsed.
+
+    ``debug``
+      Inserts a code-block with the *raw* reST. Sometimes it is helpful to see
+      what reST is generated.
+
+"""
+
+import sys
+import os
+from os import path
+import subprocess
+
+from sphinx.ext.autodoc import AutodocReporter
+
+from docutils import nodes
+from docutils.parsers.rst import Directive, directives
+from docutils.statemachine import ViewList
+from docutils.utils.error_reporting import ErrorString
+
+
+__version__  = '1.0'
+
+# We can't assume that six is installed
+PY3 = sys.version_info[0] == 3
+PY2 = sys.version_info[0] == 2
+if PY3:
+    # pylint: disable=C0103, W0622
+    unicode     = str
+    basestring  = str
+
+def setup(app):
+
+    app.add_directive("kernel-abi", KernelCmd)
+    return dict(
+        version = __version__
+        , parallel_read_safe = True
+        , parallel_write_safe = True
+    )
+
+class KernelCmd(Directive):
+
+    u"""KernelABI (``kernel-abi``) directive"""
+
+    required_arguments = 1
+    optional_arguments = 0
+    has_content = False
+    final_argument_whitespace = True
+
+    option_spec = {
+        "debug"     : directives.flag
+    }
+
+    def warn(self, message, **replace):
+        replace["fname"]   = self.state.document.current_source
+        replace["line_no"] = replace.get("line_no", self.lineno)
+        message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
+        self.state.document.settings.env.app.warn(message, prefix="")
+
+    def run(self):
+
+        doc = self.state.document
+        if not doc.settings.file_insertion_enabled:
+            raise self.warning("docutils: file insertion disabled")
+
+        env = doc.settings.env
+        cwd = path.dirname(doc.current_source)
+        cmd = "get_abi.pl rest --dir "
+        cmd += self.arguments[0]
+
+        srctree = path.abspath(os.environ["srctree"])
+
+        fname = cmd
+
+        # extend PATH with $(srctree)/scripts
+        path_env = os.pathsep.join([
+            srctree + os.sep + "scripts",
+            os.environ["PATH"]
+        ])
+        shell_env = os.environ.copy()
+        shell_env["PATH"]    = path_env
+        shell_env["srctree"] = srctree
+
+        lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
+        nodeList = self.nestedParse(lines, fname)
+        return nodeList
+
+    def runCmd(self, cmd, **kwargs):
+        u"""Run command ``cmd`` and return it's stdout as unicode."""
+
+        try:
+            proc = subprocess.Popen(
+                cmd
+                , stdout = subprocess.PIPE
+                , stderr = subprocess.PIPE
+                , universal_newlines = True
+                , **kwargs
+            )
+            out, err = proc.communicate()
+            if err:
+                self.warn(err)
+            if proc.returncode != 0:
+                raise self.severe(
+                    u"command '%s' failed with return code %d"
+                    % (cmd, proc.returncode)
+                )
+        except OSError as exc:
+            raise self.severe(u"problems with '%s' directive: %s."
+                              % (self.name, ErrorString(exc)))
+        return unicode(out)
+
+    def nestedParse(self, lines, fname):
+        content = ViewList()
+        node    = nodes.section()
+
+        if "debug" in self.options:
+            code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
+            for l in lines.split("\n"):
+                code_block += "\n    " + l
+            lines = code_block + "\n\n"
+
+        for c, l in enumerate(lines.split("\n")):
+            content.append(l, fname, c)
+
+        buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
+        self.state.memo.title_styles  = []
+        self.state.memo.section_level = 0
+        self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
+        try:
+            self.state.nested_parse(content, 0, node, match_titles=1)
+        finally:
+            self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
+        return node.children
-- 
2.9.3

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Mauro Carvalho Chehab]

Em Thu, 13 Apr 2017 07:08:43 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> That's the third attempt to add support for the Kernel ABI
> at the Documentation's admin guide.
> 
> The first approach was based on a generic extension that
> calls a random script. This one is based on a new Sphinx
> extension with adds a symbol specific for parsing ABI
> symbols.
> 
> It adds a new script (scripts/get_abi.pl) with can either
> search for ABI symbols that match a regular expression or
> outputs the entire documentation found inside a directory
> as a ReST book.
> 
> Adding the ABI description to the Linux documentation is
> as easy as adding a tab like:
> 
> 	.. kernel-abi:: $srctree/Documentation/ABI/obsolete
> 
> On version 3, I improved the script to better parse the
> ABI descriptions. On this version, it will identify if the
> description can be handled as a normal text. If so, it will
> escape characters that has special meaning on Sphinx.
> If it detects that the description syntax is too complex,
> it outputs inside a literal block.
> 
> Also on version 3, I opted to use 4 separate sections, one
> for each type of symbol. It also add an entry with the name
> of each ABI file, with has cross-references to the symbols
> defined on each file.
> 
> The parser was also improved to handle any content at the
> files that are before the ABI tags.

The HTML formatted result can be seen at:
	http://www.infradead.org/~mchehab/kernel_docs/admin-guide/abi.html

And the PDF formatted result at (pages 151 to 565):
	http://www.infradead.org/~mchehab/kernel_docs_pdf/linux-user.pdf

Thanks,
Mauro

Re: [PATCH v2 02/11] ABI: fix some syntax issues at the ABI database

[Andrew Donnellan]

On 13/04/17 20:08, Mauro Carvalho Chehab wrote:
> diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
> index 640f65e79ef1..d0b32452dfe1 100644
> --- a/Documentation/ABI/testing/sysfs-class-cxl
> +++ b/Documentation/ABI/testing/sysfs-class-cxl
> @@ -1,6 +1,6 @@
> -Note: Attributes that are shared between devices are stored in the directory
> -pointed to by the symlink device/.
> -Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
> +Please notice that attributes that are shared between devices are stored in
> +the directory pointed to by the symlink device/.
> +For example, the real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
>  /sys/class/cxl/afu0.0s/device/irqs_max, i.e. /sys/class/cxl/afu0.0/irqs_max.

If you end up respinning this, I'd prefer "Note that" or "Please note 
that" rather than "Please notice". Otherwise, for cxl:

Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com  IBM Australia Limited

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Jonathan Corbet]

On Thu, 13 Apr 2017 07:08:43 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:

> That's the third attempt to add support for the Kernel ABI
> at the Documentation's admin guide.
> 
> The first approach was based on a generic extension that
> calls a random script. This one is based on a new Sphinx
> extension with adds a symbol specific for parsing ABI
> symbols.
> 
> It adds a new script (scripts/get_abi.pl) with can either
> search for ABI symbols that match a regular expression or
> outputs the entire documentation found inside a directory
> as a ReST book.

So I've finally taken a quick look at this.  I'm not quite ready to take
it for 4.12 at this point, but I guess I'm running out of reasons to
block it in general..:)

One question, though: is there a reason for the split between the sphinx
extension and the Perl script that does the work?  Could it all happen in
the extension directly instead?  It seems like a lot of moving parts.

Thanks,

jon

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Mauro Carvalho Chehab]

Em Thu, 20 Apr 2017 15:40:00 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 13 Apr 2017 07:08:43 -0300
> Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> 
> > That's the third attempt to add support for the Kernel ABI
> > at the Documentation's admin guide.
> > 
> > The first approach was based on a generic extension that
> > calls a random script. This one is based on a new Sphinx
> > extension with adds a symbol specific for parsing ABI
> > symbols.
> > 
> > It adds a new script (scripts/get_abi.pl) with can either
> > search for ABI symbols that match a regular expression or
> > outputs the entire documentation found inside a directory
> > as a ReST book.  
> 
> So I've finally taken a quick look at this.  I'm not quite ready to take
> it for 4.12 at this point, but I guess I'm running out of reasons to
> block it in general..:)

Good!

> 
> One question, though: is there a reason for the split between the sphinx
> extension and the Perl script that does the work?  Could it all happen in
> the extension directly instead?  It seems like a lot of moving parts.

There are a couple of reasons:

- On the last feedback from you and Jani about MAINTAINERS parser,
  you both suggested that, if we would be parsing MAINTAINERS, such
  functionality should be at get_maintainer.pl. So, I tried to follow 
  the same way as get_maintainer: to have a single script that would be
  in charge of handling ABI parsing;

- get_abi.pl doesn't only generate ReST documentation. It also
  query the ABI database. So, with get_abi.pl, do things like:

	$ ./scripts/get_abi.pl search usbip_status

	/sys/devices/platform/usbip-vudc.%d/usbip_status
	------------------------------------------------

	Kernel version:		4.6
	Date:			April 2016
	Contact:		Krzysztof Opasiak <k.opasiak@samsung.com>
	Defined on file:	Documentation/ABI/testing/sysfs-platform-usbip-vudc

	Description:

	Current status of the device.
	Allowed values:
	1 - Device is available and can be exported
	2 - Device is currently exported
	3 - Fatal error occurred during communication
	  with peer

  And even use regex, querying for multiple ABI symbols:

	$ ./scripts/get_abi.pl search .*bus.*devices/.*power.*usb.* 
	
	/sys/bus/usb/devices/.../power/usb2_hardware_lpm
	------------------------------------------------
	
	Date:			September 2011
	Contact:		Andiry Xu <andiry.xu@amd.com>
	Defined on file:	Documentation/ABI/testing/sysfs-bus-usb
	
	Description:
	
	If CONFIG_PM is set and a USB 2.0 lpm-capable device is plugged
	in to a xHCI host which support link PM, it will perform a LPM
	test; if the test is passed and host supports USB2 hardware LPM
	(xHCI 1.0 feature), USB2 hardware LPM will be enabled for the
	device and the USB device directory will contain a file named
	power/usb2_hardware_lpm.  The file holds a string value (enable
	or disable) indicating whether or not USB2 hardware LPM is
	enabled for the device. Developer can write y/Y/1 or n/N/0 to
	the file to enable/disable the feature.
	
	
	/sys/bus/usb/devices/.../power/usb2_lpm_besl
	--------------------------------------------
	
	Date:			May 2013
	Contact:		Mathias Nyman <mathias.nyman@linux.intel.com>
	Defined on file:	Documentation/ABI/testing/sysfs-bus-usb
	
	Description:
	
	USB 2.0 devices that support hardware link power management (LPM)
	L1 sleep state now use a best effort service latency value (BESL) to
	indicate the best effort to resumption of service to the device after the
	initiation of the resume event.
	If the device does not have a preferred besl value then the host can select
	one instead. This usb2_lpm_besl attribute allows to tune the host selected besl
	value in order to tune power saving and service latency.
	
	Supported values are 0 - 15.
	More information on how besl values map to microseconds can be found in
	USB 2.0 ECN Errata for Link Power Management, section 4.10)
	
	/sys/bus/usb/devices/.../power/usb2_lpm_l1_timeout
	--------------------------------------------------
	
	Date:			May 2013
	Contact:		Mathias Nyman <mathias.nyman@linux.intel.com>
	Defined on file:	Documentation/ABI/testing/sysfs-bus-usb
	
	Description:
	
	USB 2.0 devices may support hardware link power management (LPM)
	L1 sleep state. The usb2_lpm_l1_timeout attribute allows
	tuning the timeout for L1 inactivity timer (LPM timer), e.g.
	needed inactivity time before host requests the device to go to L1 sleep.
	Useful for power management tuning.
	Supported values are 0 - 65535 microseconds.
	
	
	/sys/bus/usb/devices/.../power/usb3_hardware_lpm_u1
	---------------------------------------------------
	
	Date:			November 2015
	Contact:		Kevin Strasser <kevin.strasser@linux.intel.com>
	Lu Baolu <baolu.lu@linux.intel.com>
	Defined on file:	Documentation/ABI/testing/sysfs-bus-usb
	
	Description:
	
	If CONFIG_PM is set and a USB 3.0 lpm-capable device is plugged
	in to a xHCI host which supports link PM, it will check if U1
	and U2 exit latencies have been set in the BOS descriptor; if
	the check is passed and the host supports USB3 hardware LPM,
	USB3 hardware LPM will be enabled for the device and the USB
	device directory will contain two files named
	power/usb3_hardware_lpm_u1 and power/usb3_hardware_lpm_u2. These
	files hold a string value (enable or disable) indicating whether
	or not USB3 hardware LPM U1 or U2 is enabled for the device.
	
  (the output in this mode not ReST)

- I'm not a python programmer ;-) I just took Markus "generic" kernel-cmd
  code, hardcoding there a call to the script.

  With (a lot of) time, I would likely be able to find a solution to add
  the entire ABI logic there, but, in this case, we would lose the
  capability of calling the script without Sphinx.

Thanks,
Mauro

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Markus Heiser]

Am 21.04.2017 um 01:21 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:

> - I'm not a python programmer ;-) I just took Markus "generic" kernel-cmd
>  code, hardcoding there a call to the script.
> 
>  With (a lot of) time, I would likely be able to find a solution to add
>  the entire ABI logic there, but, in this case, we would lose the
>  capability of calling the script without Sphinx.

Hi Mauro,

I have no problem with calling the perl script, but your last sentence
is not correct: We can implement a python module, which is used by sphinx
and we can add a command line as well.

--Markus--

Re: [PATCH v2 00/11] Documentation: Add ABI to the admin guide

[Mauro Carvalho Chehab]

Em Fri, 21 Apr 2017 08:37:45 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 21.04.2017 um 01:21 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> > - I'm not a python programmer ;-) I just took Markus "generic" kernel-cmd
> >  code, hardcoding there a call to the script.
> > 
> >  With (a lot of) time, I would likely be able to find a solution to add
> >  the entire ABI logic there, but, in this case, we would lose the
> >  capability of calling the script without Sphinx.  
> 
> Hi Mauro,
> 
> I have no problem with calling the perl script, but your last sentence
> is not correct: We can implement a python module, which is used by sphinx
> and we can add a command line as well.

Markus,

Yeah, I guess technically it would be possible to make a Sphinx plugin 
that would also allow have a command line interface. I don't like this
kind of option, because the code can be come messy.

A better design would be to create a library and make two interfaces
for it, one for the ReST plugin and another one to be called via a
command line, but, in this case, we'll need to maintain 3 interdependent
components (library, command line, Sphinx plugin) instead of two (almost)
independent ones (a script and a Sphinx plugin).

So, IMO, the design I took is a good one, and it has a big advantage:
writing in perl is a way easier for me, and I can benefit from my
knowledge to write a script that performs well. On my desktop, it can
parse the entire ABI, search for a string there and output its result
in 100ms:

$ time ./scripts/get_abi.pl search usbip_status

/sys/devices/platform/usbip-vudc.%d/usbip_status
------------------------------------------------

Kernel version:		4.6
Date:			April 2016
Contact:		Krzysztof Opasiak <k.opasiak@samsung.com>
Defined on file:	Documentation/ABI/testing/sysfs-platform-usbip-vudc

Description:

Current status of the device.
Allowed values:
1 - Device is available and can be exported
2 - Device is currently exported
3 - Fatal error occurred during communication
  with peer


real	0m0.112s
user	0m0.106s
sys	0m0.005s

---

With regards to the decision of using perl instead of python,
see below. One might thing it is rant. It isn't. It is just a
matter of optimizing my development time.

I have lots of bad experiences with patchwork and even with
cgit (running on an old LTS debian machine) due to the lack of 
a consistent backward compatible API on python.

Most of my bad experiences on python scripts is related to how badly
it handles a file input with an unknown charset and unsigned chars > 0x7f. 
If Python can't recognize the a char as a valid ascii character 
(or the charset was not explicity defined - or the script doesn't
 know what's the original encoding charset),  the script crashes 
(ok, one could add a "try" block, but it is very painful to do that
all over the code). Also, the way the charset is specified on a python 
script changed several times during 2.x development (causing incompatible
changes), and again on 3.x. It is even worse if a python script is called 
by some other script, as, in such case, Python (not sure if all versions)
ignores script headers like:
	# -*- coding: utf-8 -*-

with would otherwise tell what's the default encoding.

If you look at patchwork git tree, you'll see that it took a really long
time since when the first patch trying to address those issues until the
last one was merged.

That's the first patch there[1]:

	commit ea39a9952e3fa647ebcb4bf16981ce941ec5236a
	Author: Mauro Carvalho Chehab <mchehab@infradead.org>
	Date:   Tue Nov 18 23:00:32 2008 -0200

	    Fix non-ascii character encodings on xmlrpc interface

That seems to be the last one[2]:
	commit 880fc52d2d4ccdcbf4a7b76f1b4ba6b9e7482dff
	Author: Siddhesh Poyarekar <siddhesh@redhat.com>
	Date:   Mon Jul 14 10:21:32 2014 +0800

	    parsemail: Fallback to common charsets when charset is None or x-unknown

AFAIKT, none of those charset fixes are due to a code regression, but,
instead, fixing parsing on different places of the code. So, it took
at least 6 years to get it right there.

So, my decision of writing it in perl is basically due to the fact
that I can write a reliable script it in a few hours, and won't 
need to be concerned that some weird char inside a file or some
new scripting interpreter version would cause my script to crash.

[1] git://github.com/getpatchwork/patchwork
[2] I guess I hit other patchwork charset bugs after 2014.

Thanks,
Mauro