RE: [RFC][PATCH] adding PCI bus information to SCSI layer

RE: [RFC][PATCH] adding PCI bus information to SCSI layer

[Matt_Domsch]

Thanks everyone for your input.

Doug Gilbert said:
> SANE (and probably some other applications) parses the
> output of 'cat /proc/scsi/scsi' so any change to its
> format may trip SANE up. How about another entry in
> the /proc/scsi directory that has a more parsable format
> (e.g. xml :-) ).

This makes a lot of sense, but I don't want to fan the war over what kind of
information should be presented in /proc. :-)  Perhaps this can wait for
scsimon changes.

Doug Gilbert said: 
> Also ISA adapters are not the only non-PCI adapters,
> there are the growing band of pseudo adapters that
> may or may not have a PCI bus at the bottom of some
> other protocol stack.

Alan said:
> An ioctl might be better. We already have an ioctl for 
> querying the lun
> information for a disk. We could also return the bus 
> information for its
> controller(s) [remember multipathing]

If I understand multipathing properly, a /dev/mdX device is composed of
several block devices, and there are IOCTLs available to list the individual
devices that compose a block device.  Each multipath component shows up as
both /dev/sda and /dev/sdb, but you mount /dev/md0 for your I/Os.  BIOSs
don't know about multipath devices, they just see the same disk twice and
don't know it's really only one disk.  Hopefully BIOS only reads the boot
sector and jumps, so this is fine.

I've proposed a SCSI ioctl that returns PCI bus, slot, function, primary and
subsystem vendor and device IDs.  Equivalent /dev/hdX information is
available in /proc/ide/ide0, but not as an IOCTL.  It could then be a
user-space problem to decompose a mdX into component sdX or hdX devices, and
query them appropriately.

I'm still not sure if/how to handle:
1) non-PCI-bus devices

Here I'm going to need some help.  My original design was based on the
information presented by EDD 3.0 Rev 5a (www.t13.org), which only presents
x86 int13-type devices, and shows them to be either ISA or PCI-based, with
the goal of telling the OS which device BIOS thinks is the boot device.
Booting off, say, a parallel-port ZIP drive isn't really handled in EDD.  


2) Devices like i2o scsi disks
	The i2o controller can be on any bus type (though I expect most are
PCI).  The i2o_controller struct has an i2o_hrt member, which does have bus
information, but the PCI struct doesn't include subsystem vendor and device
fields yet.  We'd need to add an IOCTL to retrieve this information,
possibly one at a generic i2o level.   The EDD 3.0 spec shows that a PCI i2o
device returns us a 64-bit Identity Tag, which I'm guessing is the same as a
32-bit target device ID and possibly the TID of the controller too.

3) other block devices (non-SCSI, i2o, or IDE).  I haven't investigated
these.


Doug suggested looking at extending scsimon.  This is a fine idea, and I've
made proposed changes available at http://domsch.com/linux/scsi/.  (Doug may
want to clean this up).  However, this, like my earlier changes to
/proc/scsi/scsi, doesn't actually show the relationship between /dev/sda and
a particular PCI controller and SCSI channel,bus,lun tuple.


I'd like to see the the PCI device information added to the Scsi_Host
structure, the ioctl, and the various PCI SCSI driver changes I mentioned
last week.  I've removed the changes to /proc/scsi/scsi and rebuilt it
against 2.4.4-pre6, available at
http://domsch.com/linux/scsi/linux-2.4.4-pre6-scsi-pci-info-noproc.patch

If/when this goes in, I'll also request the assistance of all of the SCSI
driver maintainers.  I've changed quite a few drivers, but couldn't easily
see how to change a few others.

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer
Dell Linux Systems Group
Linux OS Development
www.dell.com/linux

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Jeff Garzik]

Matt_Domsch@Dell.com wrote:
> I've proposed a SCSI ioctl that returns PCI bus, slot, function, primary and
> subsystem vendor and device IDs.

PCI ids can be derived from bus/slot/function.

-- 
Jeff Garzik      | The difference between America and England is that
Building 1024    | the English think 100 miles is a long distance and
MandrakeSoft     | the Americans think 100 years is a long time.
                 |      (random fortune)

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Alan Cox]

> 	The i2o controller can be on any bus type (though I expect most are
> PCI).  The i2o_controller struct has an i2o_hrt member, which does have bus

I've seen no non PCI one.

> information, but the PCI struct doesn't include subsystem vendor and device
> fields yet.  We'd need to add an IOCTL to retrieve this information,
> possibly one at a generic i2o level.   The EDD 3.0 spec shows that a PCI i2o

You can pull it from /proc if you know the bus info (which you can get by
querying via i2o_config.o)

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Douglas Gilbert]

Matt_Domsch@Dell.com wrote:
> 
> [snip]
> 
> Doug suggested looking at extending scsimon.  This is a fine idea, and I've
> made proposed changes available at http://domsch.com/linux/scsi/.  (Doug may
> want to clean this up).  However, this, like my earlier changes to
> /proc/scsi/scsi, doesn't actually show the relationship between /dev/sda and
> a particular PCI controller and SCSI channel,bus,lun tuple.

Changes look ok. One suggestion: if a #define SCSI_PCI_INFO
(or some such) is defined in driver/scsi/scsi.h as part of
the patch then code like Matt is suggesting can be safely 
added, protected by "#ifdef SCSI_PCI_INFO ... #endif" blocks. 
I have used this technique in sg to support the scsi 
"reset+reservation" patch which still hasn't made it into 
the mid level (but is available in many distros).

The scsimon driver is just a window through to the information
held in the mid level structures. The information printed by
'cat /proc/scsi/scsi' also comes from the mid level. The scsi 
minor device numbers (e.g. /dev/sda) are allocated by each upper 
level driver  (e.g. sd_attach() in the case of sd) and are held 
in upper level driver data structures. Hence they are not 
visible to the mid-level or to other upper level drivers.

As an example of the latter point, using st and sg on the same 
tape device at the same time will most likely confuse st 
(since it maintains a state machine). However there is no 
simple way for the sg or st drivers to detect (or supply
information flagging) this conflict.

Doug Gilbert

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Jeff Garzik]

Matt_Domsch@Dell.com wrote:
> 
> > PCI ids can be derived from bus/slot/function.
> 
> Even better.  I'll remove the extraneous fields then, and only return those.
> 
> typedef struct scsi_pci {
>         unsigned char   bus_number;
>         unsigned int    devfn;          /* encoded device & function index
> */
> } Scsi_Pci;

Why not pci_dev::slot_name?  Presumeably the only reason you need this
is to export it to userspace...  We already have the ASCII text there,
please consider using that :)

-- 
Jeff Garzik      | The difference between America and England is that
Building 1024    | the English think 100 miles is a long distance and
MandrakeSoft     | the Americans think 100 years is a long time.
                 |      (random fortune)

RE: [RFC][PATCH] adding PCI bus information to SCSI layer

[Matt_Domsch]

> PCI ids can be derived from bus/slot/function.

Even better.  I'll remove the extraneous fields then, and only return those.

typedef struct scsi_pci {
        unsigned char   bus_number;
        unsigned int    devfn;          /* encoded device & function index
*/
} Scsi_Pci;

Thanks,
Matt

-- 
Matt Domsch
Sr. Software Engineer
Dell Linux Systems Group
Linux OS Development
www.dell.com/linux

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Douglas Gilbert]

Alan Cox wrote:
> 
> > Also ISA adapters are not the only non-PCI adapters,
> > there are the growing band of pseudo adapters that
> > may or may not have a PCI bus at the bottom of some
> > other protocol stack.
> 
> An ioctl might be better. We already have an ioctl for querying the lun
> information for a disk. We could also return the bus information for its
> controller(s) [remember multipathing]

Both 'cat /proc/scsi/scsi' and ioctls used on
fds belonging to the existing upper level drivers
(e.g. sd and sr) have a problem as far as getting
HBA environment information: there needs to be at
least one SCSI device (target) connected to the
HBA. With no SCSI devices connected, there is no 
fd to do an ioctl on. [The same problem arises
if a device is there but marked offline, has an
exclusive lock on it, ...]

Perhaps Matt could look at the approach I have taken
with the scsimon experimental upper level driver.
Scsimon was originally designed to get scsi based
information to the /sbin/hotplug mechanism. It also
supplies ioctls to probe HBAs as well as SCSI devices.
More information about it can be found at:
  http://www.torque.net/scsi/scsimon.html

It should not be difficult to add HBA PCI bus information
to scsimon (after the Scsi_Host structure is expanded to
hold that information).

Doug Gilbert

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Jonathan Lundell]

At 4:34 PM -0500 2001-04-13, Matt Domsch wrote:
>What I'd like to do is add the PCI location of the SCSI controller to
>the information printed in /proc/scsi/scsi, as follows:
>
>Attached devices:
>Host: scsi0 Channel: 00 Id: 05 Lun: 00 PCI bus: 1 slot: 6 fn: 0
>  Vendor: NEC      Model: CD-ROM DRIVE:466 Rev: 1.06
>  Type:   CD-ROM                           ANSI SCSI revision: 02
>Host: scsi1 Channel: 00 Id: 00 Lun: 00 PCI bus: 0 slot: 2 fn: 1
>  Vendor: DELL     Model: PERCRAID RAID5   Rev: 0001
>  Type:   Direct-Access                    ANSI SCSI revision: 02
>Host: scsi1 Channel: 00 Id: 01 Lun: 00 PCI bus: 0 slot: 2 fn: 1
>  Vendor: DELL     Model: PERCRAID Stripe  Rev: 0001
>  Type:   Direct-Access                    ANSI SCSI revision: 02

Jeff Garzik proposed a scheme for ethtool that addresses somewhat similar needs for network interfaces. It's nice and general, and in particular works for non-PCI interfaces. Why not something like that for SCSI?

At 7:54 AM -0500 2001-03-21, Jeff Garzik wrote:
>In various scenarios where userland needs to interact with hardware,
>userland often needs to know exactly what driver (and driver version) is
>currently running on a given interface.  Hotplugging and other
>applications could -really- use the ability to find out bus information
>for a given interface.  Firmware
>
>So I propose the attached addition to the ethtool interface.  It adds a
>new structure with several ASCII text fields, which are filled in at the
>driver's discretion.  Userland then interprets these fields for their
>own evil designs.
>
>Currently (AFAIK) for all kernel interfaces, userland has no reliable
>way to associate a hardware device with a kernel interface, or a driver
>with a kernel interface[1].  Since we have no generic register_driver()
>interface, solving this problem means implementing a domain-specific
>solution like I have done here...
>
>--
>Jeff Garzik       | May you have warm words on a cold evening,
>Building 1024     | a full mooon on a dark night,
>MandrakeSoft      | and a smooth road all the way to your door.
>Index: include/linux/ethtool.h
>===================================================================
>RCS file: /cvsroot/gkernel/linux_2_4/include/linux/Attic/ethtool.h,v
>retrieving revision 1.1.1.2
>diff -u -r1.1.1.2 ethtool.h
>--- include/linux/ethtool.h	2000/11/14 22:01:49	1.1.1.2
>+++ include/linux/ethtool.h	2001/03/21 12:42:15
>@@ -24,10 +24,21 @@
> 	u32	reserved[4];
> };
> 
>+/* these strings are set to whatever the driver author decides... */
>+struct ethtool_drvinfo {
>+	char	driver[32];	/* driver short name, "tulip", "eepro100" */
>+	char	version[32];	/* driver version string */
>+	char	fw_version[32];	/* firmware version string, if applicable */
>+	char	bus_info[32];	/* Bus info for this interface.  For PCI
>+				 * devices, use pci_dev->slot_name. */
>+	char	reserved1[32];
>+	char	reserved2[32];
>+};
> 
> /* CMDs currently supported */
>-#define ETHTOOL_GSET		0x00000001 /* Get settings, non-privileged. */
>+#define ETHTOOL_GSET		0x00000001 /* Get settings. */
> #define ETHTOOL_SSET		0x00000002 /* Set settings, privileged. */
>+#define ETHTOOL_GDRVINFO	0x00000003 /* Get driver info. */
> 
> /* compatibility with older code */
> #define SPARC_ETH_GSET		ETHTOOL_GSET

-- 
/Jonathan Lundell.

RE: [RFC][PATCH] adding PCI bus information to SCSI layer

[Matt_Domsch]

> An ioctl might be better. We already have an ioctl for 
> querying the lun
> information for a disk. We could also return the bus 
> information for its
> controller(s) [remember multipathing]

I provide such, and a test program at http://domsch.com/linux/scsi for
trying it out.

Thanks,
Matt

-- 
Matt Domsch
Sr. Software Engineer
Dell Linux Systems Group
Linux OS Development
www.dell.com/linux

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Alan Cox]

> Also ISA adapters are not the only non-PCI adapters,
> there are the growing band of pseudo adapters that
> may or may not have a PCI bus at the bottom of some
> other protocol stack.

An ioctl might be better. We already have an ioctl for querying the lun
information for a disk. We could also return the bus information for its
controller(s) [remember multipathing]

> 

Re: [RFC][PATCH] adding PCI bus information to SCSI layer

[Douglas Gilbert]

Matt Domsch <Matt_Domsch@Dell.com> wrote:
> I'm working on an IA-64 user-space application to add a Linux entry to
> the IA-64 boot manager.  To do so, I've got to uniquely identify a
> disk by it's controller PCI address, SCSI channel,
> ID, and LUN.  Essentially, I need to tie /dev/sda to an EFI device.  An
> equivalent problem (with similar solution) exists with i386 where the
> BIOS boot order is not necessarily the Linux driver load order.
> 
> 
> BIOS Enhanced Disk Drive Services 3.0 provides a way to query BIOS for
> what it thinks is it's device location and order.  IA-64 implements
> EDD 3.0, and some i386 BIOS manufactures are adding this feature
> also.  EDD 3.0 information is available at http://www.t13.org.
> 
> What I'd like to do is add the PCI location of the SCSI controller to
> the information printed in /proc/scsi/scsi, as follows:
> 
> Attached devices:
> Host: scsi0 Channel: 00 Id: 05 Lun: 00 PCI bus: 1 slot: 6 fn: 0
>   Vendor: NEC      Model: CD-ROM DRIVE:466 Rev: 1.06
>   Type:   CD-ROM                           ANSI SCSI revision: 02

[snip]

Matt,
SANE (and probably some other applications) parses the
output of 'cat /proc/scsi/scsi' so any change to its
format may trip SANE up. How about another entry in
the /proc/scsi directory that has a more parsable format
(e.g. xml :-) ).

Also ISA adapters are not the only non-PCI adapters,
there are the growing band of pseudo adapters that
may or may not have a PCI bus at the bottom of some
other protocol stack.

Doug Gilbert

[RFC][PATCH] adding PCI bus information to SCSI layer

[Matt Domsch]

I'm working on an IA-64 user-space application to add a Linux entry to
the IA-64 boot manager.  To do so, I've got to uniquely identify a
disk by it's controller PCI address, SCSI channel,
ID, and LUN.  Essentially, I need to tie /dev/sda to an EFI device.  An
equivalent problem (with similar solution) exists with i386 where the
BIOS boot order is not necessarily the Linux driver load order.


BIOS Enhanced Disk Drive Services 3.0 provides a way to query BIOS for
what it thinks is it's device location and order.  IA-64 implements
EDD 3.0, and some i386 BIOS manufactures are adding this feature
also.  EDD 3.0 information is available at http://www.t13.org.

What I'd like to do is add the PCI location of the SCSI controller to
the information printed in /proc/scsi/scsi, as follows:

Attached devices:
Host: scsi0 Channel: 00 Id: 05 Lun: 00 PCI bus: 1 slot: 6 fn: 0
  Vendor: NEC      Model: CD-ROM DRIVE:466 Rev: 1.06
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00 PCI bus: 0 slot: 2 fn: 1
  Vendor: DELL     Model: PERCRAID RAID5   Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 01 Lun: 00 PCI bus: 0 slot: 2 fn: 1
  Vendor: DELL     Model: PERCRAID Stripe  Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 02


SCSI controllers which are not PCI devices would not display this
extra information.

IDE devices already provide PCI location information in
/proc/ide/ideX/config.

Furthermore, since reading /proc/scsi/scsi doesn't indicate which device
(say, /dev/sda) is which, I'd like an ioctl to query /dev/sda directly and
retrieve the PCI information.  An ioctl already exists to retrieve the
SCSI channel/id/lun information.


To do so, I propose the following:
1) Add a struct pci_dev *pci_dev member to struct Scsi_Host.
2) Add the printing function to scsi_proc.c, dependent on pci_dev != NULL.
3) Add an ioctl to scsi_ioctl.c for querying a device name like /dev/sda
directly for it's PCI information.
4) Add a line such as:
    		host = scsi_register (pHostTmpl, sizeof (mega_host_config));
 		if (!host)
 			goto err_unmap;
+		host->pci_dev = pdev;
  to each SCSI driver which is also a PCI device.


I've developed a proof-of-concept patch against 2.4.4-pre1, available at
http://domsch.com/linux/scsi/linux-2.4.4-pre1-scsi-pci-info-010413.patch,
in which I've modified a number of drivers.

Further work is required to change the non-trivial drivers.  By not
changing a driver, the information is simply not available, but causes no
other problems.  Drivers may be updated as time and need allow.

Additionally, on i386, I'd like to add the EDD 3.0 BIOS calls to
arch/i386/kernel/boot.S and make an edd
device driver to display such information in proc (similar to e820).

If this all works, we can remove lilo.conf guesswork like:
  bios=0x80
  disk=/dev/sda
  bios=0x81
  disk=/dev/hda
because we can query BIOS directly, and we can know exactly from
user-space, and we can actually get the SCSI driver load order
correct (assuming you trust BIOS to tell you the boot drive properly).


As always, I welcome your comments and feedback.

Thanks,
Matt

-- 
Matt Domsch
Sr. Software Engineer
Dell Linux Systems Group
Linux OS Development
www.dell.com/linux