Storage device enumeration script

Storage device enumeration script

[Phil Turmel]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

Hi All,

Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.

I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.

New features:
	Supports non-SCSI storage devices
	Describes layered block devices
		MD raid
		LVM
		generic device mapper
		loop (partial)
	Shows UUIDs
	Shows mountpoints
	Avoids repeating subtrees when enumerating raid devices

I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.

Please give it a whirl.

Phil

[-- Attachment #2: lsdrv --]
[-- Type: text/plain, Size: 14388 bytes --]

#! /usr/bin/python
# -*- coding: utf-8 -*-
#
# lsdrv - Report on a system's disk interfaces and how they are used.
# 
# Copyright (C) 2011 Philip J. Turmel <philip@turmel.org>
#
#	This program is free software; you can redistribute it and/or modify
#	it under the terms of the GNU General Public License as published by
#	the Free Software Foundation, version 2.
#
#	This program is distributed in the hope that it will be useful,
#	but WITHOUT ANY WARRANTY; without even the implied warranty of
#	MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#	GNU General Public License for more details.

import os, io, re
from subprocess import Popen, PIPE

#-------------------
# Handy base for objects as "bags of properties"
# inspired by Peter Norvig http://norvig.com/python-iaq.html
# Unlike the original, this one supplies 'None' instead of an attribute
# error when an explicitly named property has not yet been set.
class Struct(object):
	def __init__(self, **entries):
		self.__dict__.update(entries)
	def __repr__(self, recurse=[]):
		if self in recurse:
			return type(self)
		args = []
		for (k,v) in vars(self).items():
			if isinstance(v, Struct):
				args.append('%s=%s' % (k, v.__repr__(recurse+[self])))
			else:
				args.append('%s=%s' % (k, repr(v)))
		return '%s(%s)' % (type(self), ', '.join(args))
	def clone(self):
		return type(self)(**self.__dict__)
	def __getattr__(self, attr):
		return None

#-------------------
# Spawn an executable and collect its output.  Equivalent to the
# check_output convenience function of the subprocess module introduced
# in python 2.7.
def runx(*args, **kwargs):
	kwargs['stdout'] = PIPE
	kwargs['stderr'] = PIPE
	sub = Popen(*args, **kwargs)
	out, err = sub.communicate()
	return out

#-------------------
# Extract a matched expression from a string buffer
# If a match is found, return the given replace expression.
re1 = re.compile(r'([/:][a-zA-Z]*)0+([0-9])')
re2 = re.compile(r'Serial.+\'(.+)\'')
re3 = re.compile(r'Serial.+:(.+)')
def extractre(regexp, buffer, retexp=r'\1'):
	mo = re.search(regexp, buffer)
	if mo:
		return mo.expand(retexp)
	return None

#-------------------
# Extract shell variable assignments from a multiline string buffer
# This simple implementation returns everything after the equals sign
# as the value, including any quotes.
varsre = re.compile(r'^\s*([a-zA-Z][a-zA-Z0-9_]*)\s*=(.+)$', re.MULTILINE)
def extractvars(buffer):
	vars=dict()
	for mo in varsre.finditer(buffer):
		vars[mo.group(1)] = mo.group(2)
	return vars

#-------------------
# By Seo Sanghyeon.  Some changes by Connelly Barnes and Phil Turmel.
def try_int(s):
    try: return int(s)
    except: return s
natsortre = re.compile(r'(\d+|\D+)')
def natsort_key(s):
	if isinstance(s, str):
		return map(try_int, natsortre.findall(s))
	else:
		try:
			return tuple([natsort_keys(x) for x in s])
		except:
			return s
def natcmp(a, b):
    return cmp(natsort_key(a), natsort_key(b))

#-------------------
# Convert device sizes expressed in kibibytes into human-readable
# sizes with a reasonable power-of-two suffix.
def k2size(k):
	if k<1000:
		return "%4.2fk" % k
	m=k/1024.0
	if m<1000:
		return "%4.2fm" % m
	g=m/1024.0
	if g<1000:
		return "%4.2fg" % g
	t=g/1024.0
	if t<1000:
		return "%4.2ft" % t
	p=t/1024.0
	return "%4.2fp" % p

#-------------------
# Convert device sizes expressed as 512-byte sectors into human-readable
# sizes with a reasonable power-of-two suffix.
def sect2size(sectors):
	return k2size(int(sectors)/2.0)

#-------------------
# Given a sysfs path to the parent of a physical block device, resolve the
# controller path, look it up in the list of known controllers, and return
# the corresponding struct object.  If it's not present in the list, create
# the struct object w/ filled in details.
controllers=dict()
def probe_controller(cpathlink):
	cpath = os.path.realpath(cpathlink)
	if cpath in controllers:
		return controllers[cpath]
	while cpath and not os.path.exists(cpath+'/driver'):
		cpath = os.path.dirname(cpath)
		if cpath in controllers:
			return controllers[cpath]
	if not cpath:
		return None
	cntrlr = Struct(cpath=cpath, units=dict(),
		abbrev=re1.sub(r'\1\2', cpath[12:]),
		driver = os.path.realpath(cpath+'/driver').rsplit('/',1)[-1],
		modpre = io.FileIO(cpath+'/modalias').read().split("\n",1)[0].split(':',1)[0])
	if cntrlr.modpre == 'pci':
		cntrlr.description = runx(['lspci', '-s', cntrlr.abbrev.rsplit('/',1)[-1]]).split("\n",1)[0]
		cntrlr.descriptors = ['PCI', '[%s]' % cntrlr.driver, cntrlr.description]
	elif cntrlr.modpre == 'usb':
		if os.path.exists(cpath+'/busnum'):
			cntrlr.busnum = io.FileIO(cpath+'/busnum').read().split("\n",1)[0]
			cntrlr.devnum = io.FileIO(cpath+'/devnum').read().split("\n",1)[0]
			cntrlr.serial = io.FileIO(cpath+'/serial').read().split("\n",1)[0]
		else:
			parentpath = os.path.dirname(cpath)
			cntrlr.busnum = io.FileIO(parentpath+'/busnum').read().split("\n",1)[0]
			cntrlr.devnum = io.FileIO(parentpath+'/devnum').read().split("\n",1)[0]
			cntrlr.serial = io.FileIO(parentpath+'/serial').read().split("\n",1)[0]
		cntrlr.description = runx(['lsusb', '-s', cntrlr.busnum+':'+cntrlr.devnum]).split("\n",1)[0]
		cntrlr.descriptors = ['USB', '[%s]' % cntrlr.driver, cntrlr.description, '{%s}' % cntrlr.serial]
	else:
		cntrlr.descriptors = ['Controller %s' % cntrlr.abbrev[1:], '[%s]' % cntrlr.driver, cntrlr.description]
	controllers[cpath] = cntrlr
	return cntrlr

#-------------------
# Given a link to a physical block device syspath, resolve the real device
# path, look it up in the list of known physical devices, and return
# the corresponding struct object.  If it's not present in the list,
# create the struct object w/ filled in details, and probe its
# controller.
phydevs=dict()
def probe_device(devpathlink, nodestr):
	devpath = os.path.realpath(devpathlink)
	if devpath in phydevs:
		return phydevs[devpath]
	phy = Struct(dpath=devpath, node=nodestr,
		vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
		model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
	if os.path.exists(devpath+'/unique_id'):
		phy.serial = io.FileIO(devpath+'/unique_id').read().split("\n",1)[0].strip()
	if not phy.serial:
		phy.serial = extractre(re2, runx(['sginfo', '-s', '/dev/block/'+nodestr]))
	if not phy.serial:
		phy.serial = extractre(re3, runx(['smartctl', '-i', '/dev/block/'+nodestr]))
	phy.name = "%s %s" % (os.path.realpath(devpath+'/subsystem').rsplit('/',1)[-1], devpath.rsplit('/',1)[-1])
	phy.controller = probe_controller(os.path.dirname(devpath))
	if phy.controller:
		phy.controller.units[phy.name] = phy
	phydevs[devpath] = phy
	return phy

#-------------------
# Collect block device information and create dictionaries by kernel
# name and by device major:minor.  Probe each block device and try to
# describe the filesystem or other usage.
blockbyname=dict()
blockbynode=dict()
sysclassblock="/sys/class/block/"
for x in os.listdir(sysclassblock):
	nodestr=io.FileIO(sysclassblock+x+'/dev').read().split("\n")[0]
	sizestr=sect2size(io.FileIO(sysclassblock+x+'/size').read().split("\n")[0])
	node = nodestr.split(':',1)
	dev=Struct(name=x, node=nodestr, size=sizestr, major=int(node[0]), minor=int(node[1]), shown=False)
	if os.path.exists(sysclassblock+x+'/device'):
		dev.phy = probe_device(sysclassblock+x+'/device', nodestr)
		if dev.phy:
			dev.phy.block = dev
	if os.path.exists(sysclassblock+x+'/holders'):
		dev.holders = os.listdir(sysclassblock+x+'/holders')
	else:
		dev.holders = []
	if os.path.exists(sysclassblock+x+'/slaves'):
		dev.slaves = os.listdir(sysclassblock+x+'/slaves')
	else:
		dev.slaves = []
	dev.partitions = [y for y in os.listdir(sysclassblock+x) if os.path.exists(sysclassblock+x+'/'+y+'/dev')]
	dev.__dict__.update(extractvars(runx(['blkid', '-p', '-o', 'udev', '/dev/block/'+nodestr])))
	if os.path.exists(sysclassblock+x+'/md'):
		dev.isMD = True
		dev.__dict__.update(extractvars(runx(['mdadm', '--export', '--detail', '/dev/block/'+nodestr])))
	if dev.ID_FS_TYPE == 'linux_raid_member':
		dev.hasMD = True
		dev.__dict__.update(extractvars(runx(['mdadm', '--export', '--examine', '/dev/block/'+nodestr])))
		if dev.holders:
			mddir=sysclassblock+x+'/holders/'+dev.holders[0]+'/md/'
			dev.MD_array_state = io.FileIO(mddir+'array_state').read().split("\n")[0]
			dev.MD_array_size = io.FileIO(mddir+'array_size').read().split("\n")[0]
			dev.MD_slot = io.FileIO(mddir+'dev-'+x+'/slot').read().split("\n")[0]
			dev.MD_state = io.FileIO(mddir+'dev-'+x+'/state').read().split("\n")[0]
			dev.FS = "MD %s (%s/%s) %s %s %s %s" % (dev.MD_LEVEL, dev.MD_slot, int(dev.MD_DEVICES), dev.size, dev.holders[0], dev.MD_array_state, dev.MD_state)
		else:
			dev.FS = "MD %s (%s) %s inactive" % (dev.MD_LEVEL, dev.MD_DEVICES, dev.size)
	elif dev.ID_FS_TYPE and dev.ID_FS_TYPE[0:3] == 'LVM':
		# Placeholder string for inactive physical volumes.  It'll be
		# overwritten when active PVs are scanned.
		dev.FS = "PV %s (inactive)" % dev.ID_FS_TYPE
	elif dev.ID_PART_TABLE_TYPE:
		dev.FS = "Partitioned (%s) %s" % (dev.ID_PART_TABLE_TYPE, dev.size)
	elif dev.ID_FS_TYPE:
		dev.FS = "(%s) %s" % (dev.ID_FS_TYPE, dev.size)
	else:
		dev.FS = "Empty/Unknown %s" % dev.size
	if dev.ID_FS_LABEL:
		dev.FS += " '%s'" % dev.ID_FS_LABEL
	if dev.ID_FS_UUID:
		dev.FS += " {%s}" % dev.ID_FS_UUID
	blockbyname[x] = dev
	blockbynode[nodestr] = dev

#-------------------
# Collect information on mounted file systems and annotate the
# corresponding block device.  Use the block device's major:minor node
# numbers, as the mount list often shows symlinks.
for x in io.FileIO('/proc/mounts').readlines():
	if x[0:5] == '/dev/':
		mdev, mnt = tuple(x.split(' ', 2)[0:2])
		devstat = os.stat(mdev)
		nodestr="%d:%d" % (os.major(devstat.st_rdev), os.minor(devstat.st_rdev))
		if nodestr in blockbynode:
			mntstat = os.statvfs(mnt)
			dev = blockbynode[nodestr]
			dev.mountdev = mdev
			dev.mountpoint = mnt
			dev.mountinfo = mntstat

#-------------------
# Collect information on LVM volumes and groups and annotate the
# corresponding block device.  Use the block device's major:minor node
# numbers, as the mount list often shows symlinks.
vgroups = dict()
for x in runx(['pvs', '-o', 'pv_name,pv_used,pv_size,pv_uuid,vg_name,vg_size,vg_free,vg_uuid', '--noheadings', '--separator', ' ']).split("\n"):
	if x:
		pv_name, pv_used, pv_size, pv_uuid, vg_name, vg_size, vg_free, vg_uuid = tuple(x.strip().split(' ',7))
		devstat = os.stat(pv_name)
		nodestr="%d:%d" % (os.major(devstat.st_rdev), os.minor(devstat.st_rdev))
		if nodestr in blockbynode:
			dev = blockbynode[nodestr]
			dev.vg_name = vg_name
			if not dev.hasLVM:
				dev.hasLVM = True
				dev.pv_used = pv_used
				dev.pv_size = pv_size
				dev.pv_uuid = pv_uuid
				dev.FS = "PV %s %s/%s VG %s %s {%s}" % (dev.ID_FS_TYPE, pv_used, pv_size, vg_name, vg_size, pv_uuid)
			if vg_name in vgroups:
				vgroups[vg_name].PVs += [dev]
			else:
				vgroups[vg_name] = Struct(name=vg_name, size=vg_size, free=vg_free, uuid=vg_uuid, LVs=[], PVs=[dev])

for x in runx(['lvs', '-o', 'vg_name,lv_name,lv_path', '--noheadings', '--separator', ' ']).split("\n"):
	if x:
		vg_name, lv_name, lv_path = tuple(x.strip().split(' ',2))
		devstat = os.stat(lv_path)
		nodestr="%d:%d" % (os.major(devstat.st_rdev), os.minor(devstat.st_rdev))
		if nodestr in blockbynode:
			dev = blockbynode[nodestr]
			dev.isLVM = True
			dev.vg_name = vg_name
			dev.__dict__.update(extractvars(runx(['lvs', '--rows', '-o', 'all', '--nameprefixes', '--noheadings', '--unquoted', lv_path])))
		if vg_name in vgroups:
			vgroups[vg_name].LVs += [dev]
		else:
			vgroups[vg_name] = Struct(name=vg_name, LVs=[dev], PVs=[])

def show_vgroup(indent, vg):
	if vg.shown:
		return
	print "%s  └─Volume Group %s (%s) %s free {%s}" % (indent, vg.name, ','.join([dev.name for dev in vg.PVs]), vg.free, vg.uuid)
	show_blocks(indent+"   ", vg.LVs)
	vg.shown = True

#-------------------
# Given an indent level and list of block device names, recursively describe
# them.
continuation = ('│', '├')
corner = (' ', 'â””')
def show_blocks(indent, blocks):
	blocks = [x for x in blocks if not x.shown]
	for blk in blocks:
		if blk == blocks[-1]:
			branch=corner
		else:
			branch=continuation
		print "%s %s─%s: %s" % (indent, branch[1], blk.name, blk.FS)
		if blk.mountpoint:
			print "%s %s  └─Mounted as %s @ %s" % (indent, branch[0], blk.mountdev, blk.mountpoint)
		elif blk.hasLVM:
			show_vgroup(indent+"   ", vgroups[blk.vg_name])
		else:
			subs = blk.partitions + blk.holders
			subs.sort(natcmp)
			if subs:
				show_blocks("%s %s " % (indent, branch[0]), [blockbyname[x] for x in subs])
		blk.shown = True

#-------------------
# Collect SCSI host / controller pairs from sysfs and create an ordered tree.  Skip
# hosts that have targets, as they will already be in the list.  Add empty physical
# device entries for hosts without targets.
scsidir = "/sys/bus/scsi/devices/"
scsilist = os.listdir(scsidir)
hosts = dict([(int(x[4:]), Struct(n=int(x[4:]), cpath=os.path.dirname(os.path.realpath(scsidir+x)), hpath='/'+x)) for x in scsilist if x[0:4]=='host'])

for n, host in hosts.items():
	cntrlr = probe_controller(host.cpath)
	if cntrlr :
		targets = [x for x in os.listdir(host.cpath+host.hpath) if x[0:6]=='target']
		if not targets:
			phy = Struct(name='scsi %d:x:x:x [Empty]' % host.n)
			cntrlr.units[phy.name] = phy

for cntrlr in controllers.values():
	cntrlr.unitlist = cntrlr.units.keys()
	if cntrlr.unitlist:
		cntrlr.unitlist.sort(natcmp)
		cntrlr.first = cntrlr.unitlist[0]
	else:
		cntrlr.first = ''

tree=[(cntrlr.first, cntrlr) for cntrlr in controllers.values()]
tree.sort(natcmp)

for f, cntrlr in tree:
	print " ".join(cntrlr.descriptors)
	if cntrlr.unitlist:
		cntrlr.units[cntrlr.unitlist[-1]].last = True
	branch = continuation
	for key in cntrlr.unitlist:
		phy = cntrlr.units[key]
		if phy.last:
			branch = corner
		unitdetail = phy.name
		if phy.vendor:
			unitdetail += ' '+phy.vendor
		if phy.model:
			unitdetail += ' '+phy.model
		if phy.serial:
			unitdetail += " {%s}" % phy.serial.strip()
		print ' %s─%s' % (branch[1], unitdetail)
		if phy.block:
			show_blocks(" %s " % branch[0], [phy.block])

unshown = [z.name for z in blockbynode.values() if z.size != '0.00k' and not z.shown]
unshown.sort(natcmp)
if unshown:
	print "Other Block Devices"
	show_blocks("", [blockbyname[x] for x in unshown])

Re: Storage device enumeration script

[Mathias Burén]

On 26 May 2011 04:03, Phil Turmel <philip@turmel.org> wrote:
> Hi All,
>
> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>
> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>
> New features:
>        Supports non-SCSI storage devices
>        Describes layered block devices
>                MD raid
>                LVM
>                generic device mapper
>                loop (partial)
>        Shows UUIDs
>        Shows mountpoints
>        Avoids repeating subtrees when enumerating raid devices
>
> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>
> Please give it a whirl.
>
> Phil
>

Awesome, however:

$ ./lsdrv.py
  File "./lsdrv.py", line 301
    print "%s  └─Volume Group %s (%s) %s free {%s}" % (indent,
vg.name, ','.join([dev.name for dev in vg.PVs]), vg.free, vg.uuid)
                                                      ^
SyntaxError: invalid syntax

Maybe it's a copy-paste thing from my mail that does it. Could you put
it somewhere, like a pastebin?

cheers,
/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

On 05/25/2011 11:10 PM, Mathias Burén wrote:
> On 26 May 2011 04:03, Phil Turmel <philip@turmel.org> wrote:
>> Hi All,
>>
>> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>>
>> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>>
>> New features:
>>        Supports non-SCSI storage devices
>>        Describes layered block devices
>>                MD raid
>>                LVM
>>                generic device mapper
>>                loop (partial)
>>        Shows UUIDs
>>        Shows mountpoints
>>        Avoids repeating subtrees when enumerating raid devices
>>
>> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>>
>> Please give it a whirl.
>>
>> Phil
>>
> 
> Awesome, however:
> 
> $ ./lsdrv.py
>   File "./lsdrv.py", line 301
>     print "%s  └─Volume Group %s (%s) %s free {%s}" % (indent,
> vg.name, ','.join([dev.name for dev in vg.PVs]), vg.free, vg.uuid)
>                                                       ^
> SyntaxError: invalid syntax
> 
> Maybe it's a copy-paste thing from my mail that does it. Could you put
> it somewhere, like a pastebin?

Yeah, should have thought of that first.

http://www.turmel.org/other/lsdrv

HTH,

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Mathias Burén]

On 26 May 2011 04:21, Phil Turmel <philip@turmel.org> wrote:
> On 05/25/2011 11:10 PM, Mathias Burén wrote:
>> On 26 May 2011 04:03, Phil Turmel <philip@turmel.org> wrote:
>>> Hi All,
>>>
>>> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>>>
>>> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>>>
>>> New features:
>>>        Supports non-SCSI storage devices
>>>        Describes layered block devices
>>>                MD raid
>>>                LVM
>>>                generic device mapper
>>>                loop (partial)
>>>        Shows UUIDs
>>>        Shows mountpoints
>>>        Avoids repeating subtrees when enumerating raid devices
>>>
>>> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>>>
>>> Please give it a whirl.
>>>
>>> Phil
>>>
>>
>> Awesome, however:
>>
>> $ ./lsdrv.py
>>   File "./lsdrv.py", line 301
>>     print "%s  └─Volume Group %s (%s) %s free {%s}" % (indent,
>> vg.name, ','.join([dev.name for dev in vg.PVs]), vg.free, vg.uuid)
>>                                                       ^
>> SyntaxError: invalid syntax
>>
>> Maybe it's a copy-paste thing from my mail that does it. Could you put
>> it somewhere, like a pastebin?
>
> Yeah, should have thought of that first.
>
> http://www.turmel.org/other/lsdrv
>
> HTH,
>
> Phil
>

Aha, Archlinux defaults to python3. If I run the script with python2
it works. :)

~/bin $ sudo python2 lsdrv
PCI [ahci] 00:0b.0 SATA controller: nVidia Corporation MCP79 AHCI
Controller (rev b1)
 ├─scsi 0:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
 │  └─sda: Partitioned (dos) 55.90g
 │     ├─sda1: (ext4) 100.00m 'ssd_boot' {ae879f86-73a4-451f-bb6b-e778ad1b57d6}
 │     │  └─Mounted as /dev/sda1 @ /boot
 │     ├─sda2: (swap) 2.00g 'ssd_swap' {a28e32fa-628c-419a-9693-ca88166d230f}
 │     └─sda3: (ext4) 53.80g 'ssd_root' {6e812ed7-01c4-4a76-ae31-7b3d36d847f5}
 │        └─Mounted as /dev/disk/by-label/ssd_root @ /
 ├─scsi 1:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA1022443}
 │  └─sdb: Partitioned (dos) 1.82t
 │     └─sdb1: MD raid6 (1/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 │        └─md0: PV LVM2_member 9.08t/9.08t VG lvstorage 9.08t
{YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2}
 │            └─Volume Group lvstorage (md0) 0 free {
Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI}
 │              └─dm-0: (ext4) 9.08t 'storage'
{0ca82f13-680f-4b0d-a5d0-08c246a838e5}
 │                 └─Mounted as /dev/mapper/lvstorage-storage @ /raid6volume
 ├─scsi 2:0:0:0 ATA WDC WD20EARS-00M {WD-WMAZ20152590}
 │  └─sdc: Partitioned (dos) 1.82t
 │     └─sdc1: MD raid6 (3/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 3:0:0:0 ATA WDC WD20EARS-00M {WD-WMAZ20188479}
 │  └─sdd: Partitioned (dos) 1.82t
 │     └─sdd1: MD raid6 (2/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 4:x:x:x [Empty]
 └─scsi 5:x:x:x [Empty]
PCI [sata_mv] 05:00.0 SCSI storage controller: HighPoint Technologies,
Inc. RocketRAID 230x 4 Port SATA-II Controller (rev 02)
 ├─scsi 6:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA3609190}
 │  └─sde: Partitioned (dos) 1.82t
 │     └─sde1: MD raid6 (6/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 7:0:0:0 ATA SAMSUNG HD204UI {S2HGJ1RZ800964}
 │  └─sdf: Partitioned (dos) 1.82t
 │     └─sdf1: MD raid6 (4/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 8:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA1000331}
 │  └─sdg: Partitioned (dos) 1.82t
 │     └─sdg1: MD raid6 (0/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 └─scsi 9:0:0:0 ATA SAMSUNG HD204UI {S2HGJ1RZ800850}
    └─sdh: Partitioned (dos) 1.82t
       └─sdh1: MD raid6 (5/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}


Sweet.

/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Wed, 25 May 2011 23:21:52 -0400
Phil Turmel <philip@turmel.org> wrote:

> Yeah, should have thought of that first.
> 
> http://www.turmel.org/other/lsdrv

Traceback (most recent call last):
  File "/r/scripts/lsdrv", line 198, in <module>
    dev.phy = probe_device(sysclassblock+x+'/device', nodestr)
  File "/r/scripts/lsdrv", line 170, in probe_device
    vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
  File "/usr/lib/python2.6/io.py", line 631, in __init__
    _fileio._FileIO.__init__(self, name, mode, closefd)
IOError: [Errno 2] No such file or directory:
'/sys/devices/platform/floppy.0/vendor'

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[CoolCold]

On Thu, May 26, 2011 at 7:21 AM, Phil Turmel <philip@turmel.org> wrote:
> On 05/25/2011 11:10 PM, Mathias Burén wrote:
>> On 26 May 2011 04:03, Phil Turmel <philip@turmel.org> wrote:
>>> Hi All,
>>>
>>> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>>>
>>> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>>>
>>> New features:
>>>        Supports non-SCSI storage devices
>>>        Describes layered block devices
>>>                MD raid
>>>                LVM
>>>                generic device mapper
>>>                loop (partial)
>>>        Shows UUIDs
>>>        Shows mountpoints
>>>        Avoids repeating subtrees when enumerating raid devices
>>>
>>> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>>>
>>> Please give it a whirl.
>>>
>>> Phil
>>>
>>
>> Awesome, however:
>>
>> $ ./lsdrv.py
>>   File "./lsdrv.py", line 301
>>     print "%s  └─Volume Group %s (%s) %s free {%s}" % (indent,
>> vg.name, ','.join([dev.name for dev in vg.PVs]), vg.free, vg.uuid)
>>                                                       ^
>> SyntaxError: invalid syntax
>>
>> Maybe it's a copy-paste thing from my mail that does it. Could you put
>> it somewhere, like a pastebin?
>
> Yeah, should have thought of that first.
>
> http://www.turmel.org/other/lsdrv
>
> HTH,

Works in Debian Squeeze, but requires packages pciutils and sg3-utils
to be installed.

On Debian Lenny produces error:
root@gamma2:/tmp# python lsdrv
Traceback (most recent call last):
  File "lsdrv", line 17, in <module>
    import os, io, re
ImportError: No module named io


>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 04:24 AM, CoolCold wrote:
[...]
> On Debian Lenny produces error:
> root@gamma2:/tmp# python lsdrv
> Traceback (most recent call last):
>   File "lsdrv", line 17, in <module>
>     import os, io, re
> ImportError: No module named io

Huh.  The 'io' module is v2.6 and above.  Another reason to make a helper function for reading these files.

Phil

Re: Storage device enumeration script

[Simon McNair]

Hi Phil,
Thanks for doing this work off your own back, I'm sure it'll help a lot 
of people.

My 2p would be to ask if you could have a comment at the start of the 
file with the prerequisite applications/packages named it may help the 
people that it doesn't work for (initially).  The other thing is that 
the virtualbox install of Ubuntu 11.04 that I just completed did not 
contain the LVM2 package.

regards
Simon

On 26/05/2011 13:00, Phil Turmel wrote:
> On 05/26/2011 04:24 AM, CoolCold wrote:
> [...]
>> On Debian Lenny produces error:
>> root@gamma2:/tmp# python lsdrv
>> Traceback (most recent call last):
>>    File "lsdrv", line 17, in<module>
>>      import os, io, re
>> ImportError: No module named io
> Huh.  The 'io' module is v2.6 and above.  Another reason to make a helper function for reading these files.
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message tomajordomo@vger.kernel.org
> More majordomo info athttp://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[CoolCold]

On Tue, May 31, 2011 at 10:51 PM, Simon McNair <simonmcnair@gmail.com> wrote:
> Hi Phil,
> Thanks for doing this work off your own back, I'm sure it'll help a lot of
> people.
>
> My 2p would be to ask if you could have a comment at the start of the file
> with the prerequisite applications/packages named it may help the people
> that it doesn't work for (initially).  The other thing is that the
> virtualbox install of Ubuntu 11.04 that I just completed did not contain the
> LVM2 package.
There is debian packaging for this tool, checkout
https://github.com/pturmel/lsdrv
You should be able build your package with proper dependencies.


>
> regards
> Simon
>
> On 26/05/2011 13:00, Phil Turmel wrote:
>>
>> On 05/26/2011 04:24 AM, CoolCold wrote:
>> [...]
>>>
>>> On Debian Lenny produces error:
>>> root@gamma2:/tmp# python lsdrv
>>> Traceback (most recent call last):
>>>   File "lsdrv", line 17, in<module>
>>>     import os, io, re
>>> ImportError: No module named io
>>
>> Huh.  The 'io' module is v2.6 and above.  Another reason to make a helper
>> function for reading these files.
>>
>> Phil
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message tomajordomo@vger.kernel.org
>> More majordomo info athttp://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

Hi Simon,

On 05/31/2011 05:21 PM, CoolCold wrote:
> On Tue, May 31, 2011 at 10:51 PM, Simon McNair <simonmcnair@gmail.com> wrote:
>> Hi Phil,
>> Thanks for doing this work off your own back, I'm sure it'll help a lot of
>> people.
>>
>> My 2p would be to ask if you could have a comment at the start of the file
>> with the prerequisite applications/packages named it may help the people
>> that it doesn't work for (initially).  The other thing is that the
>> virtualbox install of Ubuntu 11.04 that I just completed did not contain the
>> LVM2 package.
> There is debian packaging for this tool, checkout
> https://github.com/pturmel/lsdrv
> You should be able build your package with proper dependencies.

The latest version (this evening) reports on the utilities that didn't work, so you can look for what you need.  Of course, if you have LVM volumes, but not the utilities, your system won't actually be able to start up those devices.

If you don't have LVM volumes, you can safely ignore the warning.  Hmmm.  Maybe I should suppress that if no LVM PVs are found in the first pass....

G'night!

Phil

RE: Storage device enumeration script

[Leslie Rhorer]

	I get an error right off the bat:

Traceback (most recent call last):
  File "./lsdrv.txt", line 264, in <module>
    for x in runx(['pvs', '-o',
'pv_name,pv_used,pv_size,pv_uuid,vg_name,vg_size,vg_free,vg_uuid',
'--noheadings', '--separator', ' ']).split("\n"):
  File "./lsdrv.txt", line 50, in runx
    sub = Popen(*args, **kwargs)
  File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I'm running Python 2.6.6 and mdadm 3.1.4 running on kernel 2.6.32-5-amd64

Re: Storage device enumeration script

[Mathias Burén]

On 26 May 2011 07:14, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>
>        I get an error right off the bat:
>
> Traceback (most recent call last):
>  File "./lsdrv.txt", line 264, in <module>
>    for x in runx(['pvs', '-o',
> 'pv_name,pv_used,pv_size,pv_uuid,vg_name,vg_size,vg_free,vg_uuid',
> '--noheadings', '--separator', ' ']).split("\n"):
>  File "./lsdrv.txt", line 50, in runx
>    sub = Popen(*args, **kwargs)
>  File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
>    errread, errwrite)
>  File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
>    raise child_exception
> OSError: [Errno 2] No such file or directory
>
> I'm running Python 2.6.6 and mdadm 3.1.4 running on kernel 2.6.32-5-amd64
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I get that in Ubuntu 11.04 on a laptop as well.

/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 02:16 AM, Mathias Burén wrote:
> On 26 May 2011 07:14, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>>
>>        I get an error right off the bat:
>>
>> Traceback (most recent call last):
>>  File "./lsdrv.txt", line 264, in <module>
>>    for x in runx(['pvs', '-o',
>> 'pv_name,pv_used,pv_size,pv_uuid,vg_name,vg_size,vg_free,vg_uuid',
>> '--noheadings', '--separator', ' ']).split("\n"):
>>  File "./lsdrv.txt", line 50, in runx
>>    sub = Popen(*args, **kwargs)
>>  File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
>>    errread, errwrite)
>>  File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
>>    raise child_exception
>> OSError: [Errno 2] No such file or directory
>>
>> I'm running Python 2.6.6 and mdadm 3.1.4 running on kernel 2.6.32-5-amd64
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> I get that in Ubuntu 11.04 on a laptop as well.
> 
> /M

Does a plain install of Ubuntu include the LVM utilities?  Can you run "pvs" directly?

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Storage device enumeration script

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Phil Turmel
> Sent: Thursday, May 26, 2011 6:42 AM
> To: Mathias Burén
> Cc: lrhorer@satx.rr.com; linux-raid@vger.kernel.org
> Subject: Re: Storage device enumeration script
> 
> On 05/26/2011 02:16 AM, Mathias Burén wrote:
> > On 26 May 2011 07:14, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
> >>
> >>        I get an error right off the bat:
> >>
> >> Traceback (most recent call last):
> >>  File "./lsdrv.txt", line 264, in <module>
> >>    for x in runx(['pvs', '-o',
> >> 'pv_name,pv_used,pv_size,pv_uuid,vg_name,vg_size,vg_free,vg_uuid',
> >> '--noheadings', '--separator', ' ']).split("\n"):
> >>  File "./lsdrv.txt", line 50, in runx
> >>    sub = Popen(*args, **kwargs)
> >>  File "/usr/lib/python2.6/subprocess.py", line 623, in __init__
> >>    errread, errwrite)
> >>  File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
> >>    raise child_exception
> >> OSError: [Errno 2] No such file or directory
> >>
> >> I'm running Python 2.6.6 and mdadm 3.1.4 running on kernel 2.6.32-5-
> amd64
> >>
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >
> > I get that in Ubuntu 11.04 on a laptop as well.
> >
> > /M
> 
> Does a plain install of Ubuntu include the LVM utilities?  Can you run
> "pvs" directly?

	I can't speak to Ubuntu, but Debian evidently does not.  I don't
know what "pvs" is, but neither bash nor Python recognize it as a file
anywhere in the path.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Brad Campbell]

On 27/05/11 08:13, Leslie Rhorer wrote:
>>
> 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
> know what "pvs" is, but neither bash nor Python recognize it as a file
> anywhere in the path.
apt-get install lvm2.

Re: Storage device enumeration script

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

On Fri, 27 May 2011 08:16:07 +0800
Brad Campbell <brad@fnarfbargle.com> wrote:

> On 27/05/11 08:13, Leslie Rhorer wrote:
> >>
> > 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
> > know what "pvs" is, but neither bash nor Python recognize it as a file
> > anywhere in the path.
> apt-get install lvm2.

Sure, but lsdrv shouldn't assume lvm2 is installed or require it to be
installed. Not everyone uses LVM, and simply installing it automatically adds
things to initramfs (PV/VG/LV detection?), which can slow down boot-up process.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[Gordon Henderson]

On Fri, 27 May 2011, Roman Mamedov wrote:

> On Fri, 27 May 2011 08:16:07 +0800
> Brad Campbell <brad@fnarfbargle.com> wrote:
>
>> On 27/05/11 08:13, Leslie Rhorer wrote:
>>>>
>>> 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
>>> know what "pvs" is, but neither bash nor Python recognize it as a file
>>> anywhere in the path.
>> apt-get install lvm2.
>
> Sure, but lsdrv shouldn't assume lvm2 is installed or require it to be
> installed. Not everyone uses LVM, and simply installing it automatically adds
> things to initramfs (PV/VG/LV detection?), which can slow down boot-up process.

As well as not using LVM, some of us don't even use udev... Or kernels 
with modules...

Can someone post the output of this utility so I can see what it's 
actually doing?

(I don't have python on all my servers either)

Cheers,

Gordon

Re: Storage device enumeration script

[Torbjørn Skagestad]

[-- Attachment #1: Type: text/plain, Size: 3262 bytes --]

Hi,

Some info can be found here https://github.com/pturmel/lsdrv

The output will typically be like this:

Controller platform [None]
 └─platform floppy.0
    └─fd0: Empty/Unknown 4.00k
PCI [pata_jmicron] 03:00.1 IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)
 ├─scsi 0:x:x:x [Empty]
 └─scsi 2:x:x:x [Empty]
PCI [ahci] 03:00.0 SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)
 ├─scsi 12:0:0:0 ATA THROTTLE
 │  └─sdu: Empty/Unknown 7.51g
 │     ├─sdu1: Empty/Unknown 243.00m
 │     │  └─Mounted as /dev/sdu1 @ /boot
 │     ├─sdu2: Empty/Unknown 1.00k
 │     └─sdu5: Empty/Unknown 7.27g
 │        ├─dm-0: Empty/Unknown 6.90g
 │        │  └─Mounted as /dev/mapper/server-root @ /
 │        └─dm-1: Empty/Unknown 368.00m
 └─scsi 13:0:0:0 ATA WDC WD10EACS-00Z
    └─sdv: Empty/Unknown 931.51g
       └─md1: Empty/Unknown 3.64t
          └─dm-3: Empty/Unknown 3.64t
             └─Mounted as /dev/mapper/big1 @ /mnt/big1
PCI [pcieport] 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
 ├─scsi 15:0:0:0 ATA WDC WD20EARS-00J
 │  └─sdy: Empty/Unknown 1.82t
 │     └─sdy1: Empty/Unknown 1.82t
 │        └─md3: Empty/Unknown 7.28t
 │           └─dm-5: Empty/Unknown 7.28t
 │              └─Mounted as /dev/mapper/big3 @ /mnt/big3
 ├─scsi 15:0:1:0 ATA WDC WD20EARS-00J
 │  └─sdz: Empty/Unknown 1.82t
 │     └─sdz1: Empty/Unknown 1.82t
 ├─scsi 15:0:2:0 ATA WDC WD20EARS-00J
 │  └─sda: Empty/Unknown 1.82t
 │     └─sda1: Empty/Unknown 1.82t
 ├─scsi 15:0:3:0 ATA WDC WD20EARS-00J
 │  └─sdw: Empty/Unknown 1.82t
 │     └─sdw1: Empty/Unknown 1.82t
 └─scsi 15:0:4:0 ATA WDC WD20EARS-00J
    └─sdae: Empty/Unknown 1.82t
       └─sdae1: Empty/Unknown 1.82t
---snip---


On Fri, 2011-05-27 at 11:45 +0100, Gordon Henderson wrote:
> On Fri, 27 May 2011, Roman Mamedov wrote:
> 
> > On Fri, 27 May 2011 08:16:07 +0800
> > Brad Campbell <brad@fnarfbargle.com> wrote:
> >
> >> On 27/05/11 08:13, Leslie Rhorer wrote:
> >>>>
> >>> 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
> >>> know what "pvs" is, but neither bash nor Python recognize it as a file
> >>> anywhere in the path.
> >> apt-get install lvm2.
> >
> > Sure, but lsdrv shouldn't assume lvm2 is installed or require it to be
> > installed. Not everyone uses LVM, and simply installing it automatically adds
> > things to initramfs (PV/VG/LV detection?), which can slow down boot-up process.
> 
> As well as not using LVM, some of us don't even use udev... Or kernels 
> with modules...
> 
> Can someone post the output of this utility so I can see what it's 
> actually doing?
> 
> (I don't have python on all my servers either)
> 
> Cheers,
> 
> Gordon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Torbjørn Skagestad
Idé Til Produkt AS
torborn@itpas.no

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[Gordon Henderson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3278 bytes --]

On Fri, 27 May 2011, Torbjørn Skagestad wrote:

> Hi,
>
> Some info can be found here https://github.com/pturmel/lsdrv
>
> The output will typically be like this:

Ok. Intersting, thanks.

I generllyy know what's inside my servers, but I suppose visualising it is 
good, or having to work on someone elses server...

Gordon


>
> Controller platform [None]
> ??platform floppy.0
>    ??fd0: Empty/Unknown 4.00k
> PCI [pata_jmicron] 03:00.1 IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)
> ??scsi 0:x:x:x [Empty]
> ??scsi 2:x:x:x [Empty]
> PCI [ahci] 03:00.0 SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02)
> ??scsi 12:0:0:0 ATA THROTTLE
> ?  ??sdu: Empty/Unknown 7.51g
> ?     ??sdu1: Empty/Unknown 243.00m
> ?     ?  ??Mounted as /dev/sdu1 @ /boot
> ?     ??sdu2: Empty/Unknown 1.00k
> ?     ??sdu5: Empty/Unknown 7.27g
> ?        ??dm-0: Empty/Unknown 6.90g
> ?        ?  ??Mounted as /dev/mapper/server-root @ /
> ?        ??dm-1: Empty/Unknown 368.00m
> ??scsi 13:0:0:0 ATA WDC WD10EACS-00Z
>    ??sdv: Empty/Unknown 931.51g
>       ??md1: Empty/Unknown 3.64t
>          ??dm-3: Empty/Unknown 3.64t
>             ??Mounted as /dev/mapper/big1 @ /mnt/big1
> PCI [pcieport] 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
> ??scsi 15:0:0:0 ATA WDC WD20EARS-00J
> ?  ??sdy: Empty/Unknown 1.82t
> ?     ??sdy1: Empty/Unknown 1.82t
> ?        ??md3: Empty/Unknown 7.28t
> ?           ??dm-5: Empty/Unknown 7.28t
> ?              ??Mounted as /dev/mapper/big3 @ /mnt/big3
> ??scsi 15:0:1:0 ATA WDC WD20EARS-00J
> ?  ??sdz: Empty/Unknown 1.82t
> ?     ??sdz1: Empty/Unknown 1.82t
> ??scsi 15:0:2:0 ATA WDC WD20EARS-00J
> ?  ??sda: Empty/Unknown 1.82t
> ?     ??sda1: Empty/Unknown 1.82t
> ??scsi 15:0:3:0 ATA WDC WD20EARS-00J
> ?  ??sdw: Empty/Unknown 1.82t
> ?     ??sdw1: Empty/Unknown 1.82t
> ??scsi 15:0:4:0 ATA WDC WD20EARS-00J
>    ??sdae: Empty/Unknown 1.82t
>       ??sdae1: Empty/Unknown 1.82t
> ---snip---
>
>
> On Fri, 2011-05-27 at 11:45 +0100, Gordon Henderson wrote:
>> On Fri, 27 May 2011, Roman Mamedov wrote:
>>
>>> On Fri, 27 May 2011 08:16:07 +0800
>>> Brad Campbell <brad@fnarfbargle.com> wrote:
>>>
>>>> On 27/05/11 08:13, Leslie Rhorer wrote:
>>>>>>
>>>>> 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
>>>>> know what "pvs" is, but neither bash nor Python recognize it as a file
>>>>> anywhere in the path.
>>>> apt-get install lvm2.
>>>
>>> Sure, but lsdrv shouldn't assume lvm2 is installed or require it to be
>>> installed. Not everyone uses LVM, and simply installing it automatically adds
>>> things to initramfs (PV/VG/LV detection?), which can slow down boot-up process.
>>
>> As well as not using LVM, some of us don't even use udev... Or kernels
>> with modules...
>>
>> Can someone post the output of this utility so I can see what it's
>> actually doing?
>>
>> (I don't have python on all my servers either)
>>
>> Cheers,
>>
>> Gordon
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> -- 
> Torbjørn Skagestad
> Idé Til Produkt AS
> torborn@itpas.no
>

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 11:58 PM, Roman Mamedov wrote:
> On Fri, 27 May 2011 08:16:07 +0800
> Brad Campbell <brad@fnarfbargle.com> wrote:
> 
>> On 27/05/11 08:13, Leslie Rhorer wrote:
>>>>
>>> 	I can't speak to Ubuntu, but Debian evidently does not.  I don't
>>> know what "pvs" is, but neither bash nor Python recognize it as a file
>>> anywhere in the path.
>> apt-get install lvm2.
> 
> Sure, but lsdrv shouldn't assume lvm2 is installed or require it to be
> installed. Not everyone uses LVM, and simply installing it automatically adds
> things to initramfs (PV/VG/LV detection?), which can slow down boot-up process.

With the latest contribution from Torbjørn, lsdrv will now report the missing utility, then continue.  If one of the devices is in fact an LVM element, the block device recursion code should still show the basic relationships, without the volume group details.  Testing that case would be appreciated.

https://github.com/pturmel/lsdrv

If you find more bugs, or have other suggestions, using the issue tracker and the wiki on github would minimize the chatter on linux-raid.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[CoolCold]

May be setup some github repo for this?

On Thu, May 26, 2011 at 7:03 AM, Phil Turmel <philip@turmel.org> wrote:
> Hi All,
>
> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>
> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>
> New features:
>        Supports non-SCSI storage devices
>        Describes layered block devices
>                MD raid
>                LVM
>                generic device mapper
>                loop (partial)
>        Shows UUIDs
>        Shows mountpoints
>        Avoids repeating subtrees when enumerating raid devices
>
> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>
> Please give it a whirl.
>
> Phil
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[John Robinson]

On 26/05/2011 04:03, Phil Turmel wrote:
> Hi All,
>
> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
>
> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
>
> New features:
> 	Supports non-SCSI storage devices
> 	Describes layered block devices
> 		MD raid
> 		LVM
> 		generic device mapper
> 		loop (partial)
> 	Shows UUIDs
> 	Shows mountpoints
> 	Avoids repeating subtrees when enumerating raid devices
>
> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
>
> Please give it a whirl.

On CentOS 5:

[root@beast lsdrv]# ./lsdrv
Traceback (most recent call last):
   File "./lsdrv", line 17, in ?
     import os, io, re
ImportError: No module named io
[root@beast lsdrv]# python -V
Python 2.4.3

Maybe this is too old, so I got Python 2.6.5 from EPEL. Trying again:

[root@beast lsdrv]# python2.6 lsdrv
Traceback (most recent call last):
   File "./lsdrv", line 192, in <module>
     for x in os.listdir(sysclassblock):
OSError: [Errno 2] No such file or directory: '/sys/class/block/'

So I changed sysclassblock to '/sys/block/' to suit EL5's 2.6.18 kernel 
(so it might be nice ;-) if you could auto-detect which was present):

[root@beast lsdrv]# python2.6 lsdrv
Traceback (most recent call last):
   File "lsdrv", line 198, in <module>
     dev.phy = probe_device(sysclassblock+x+'/device', nodestr)
   File "lsdrv", line 170, in probe_device
     vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
   File "/usr/lib64/python2.6/io.py", line 631, in __init__
     _fileio._FileIO.__init__(self, name, mode, closefd)
IOError: [Errno 2] No such file or directory: 
'/sys/devices/platform/floppy.0/vendor'

This looks like the same error Roman had.

Cheers,

John.

Re: Storage device enumeration script

[Torbjørn Skagestad]

[-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --]

Hi,

Great tool, thanks for sharing.

I had to add some error handling to get it to work properly.
Currently it runs on Ubuntu 10.04, 10.10 and 11.04.

Added the check John Robinson asked for as well.

Attached patch for those interested.


--
Thanks
Torbjørn

On Wed, 2011-05-25 at 23:03 -0400, Phil Turmel wrote:
> Hi All,
> 
> Last November, I shared a shell script that helped me keep track of the specific hot-swap drives I had in the various slots of my servers.  Although encouraged by Roman and John, I declined to make a project out of it.
> 
> I've since kicked it around some more, and thought a bit about supporting more than just the SCSI subsystem.  The latest and greatest is still built around some standard executables:  blkid, lspci, lsusb, sginfo, and smartctl.  The original was similar to "lsscsi", but with controller details and device serial numbers.
> 
> New features:
> 	Supports non-SCSI storage devices
> 	Describes layered block devices
> 		MD raid
> 		LVM
> 		generic device mapper
> 		loop (partial)
> 	Shows UUIDs
> 	Shows mountpoints
> 	Avoids repeating subtrees when enumerating raid devices
> 
> I struggled with the last item, until I gave up on bash.  I needed to pass data to subroutines by reference, and bash is sorely lacking in that area.  The new script is in python.  I'm releasing this one under the GPL version 2.
> 
> Please give it a whirl.
> 
> Phil

-- 
Torbjørn Skagestad
Idé Til Produkt AS
torborn@itpas.no

[-- Attachment #1.2: lsdrv.patch --]
[-- Type: text/x-patch, Size: 1821 bytes --]

--- lsdrv	2011-05-26 11:06:44.000000000 +0200
+++ lsdrv.py	2011-05-26 11:34:50.000000000 +0200
@@ -47,7 +47,12 @@
 def runx(*args, **kwargs):
 	kwargs['stdout'] = PIPE
 	kwargs['stderr'] = PIPE
-	sub = Popen(*args, **kwargs)
+	try:
+		sub = Popen(*args, **kwargs)
+	except OSError as e:
+		print "Unable to execute " + str(args[0][0])
+		print e
+		exit()
 	out, err = sub.communicate()
 	return out
 
@@ -166,9 +171,12 @@
 	devpath = os.path.realpath(devpathlink)
 	if devpath in phydevs:
 		return phydevs[devpath]
-	phy = Struct(dpath=devpath, node=nodestr,
-		vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
-		model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
+	try:
+		phy = Struct(dpath=devpath, node=nodestr,
+			vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
+			model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
+	except IOError:
+		return None
 	if os.path.exists(devpath+'/unique_id'):
 		phy.serial = io.FileIO(devpath+'/unique_id').read().split("\n",1)[0].strip()
 	if not phy.serial:
@@ -189,6 +197,8 @@
 blockbyname=dict()
 blockbynode=dict()
 sysclassblock="/sys/class/block/"
+if(not os.path.exists(sysclassblock)):
+	sysclassblock = "/sys/block/"
 for x in os.listdir(sysclassblock):
 	nodestr=io.FileIO(sysclassblock+x+'/dev').read().split("\n")[0]
 	sizestr=sect2size(io.FileIO(sysclassblock+x+'/size').read().split("\n")[0])
@@ -250,7 +260,10 @@
 		devstat = os.stat(mdev)
 		nodestr="%d:%d" % (os.major(devstat.st_rdev), os.minor(devstat.st_rdev))
 		if nodestr in blockbynode:
-			mntstat = os.statvfs(mnt)
+			try:
+				mntstat = os.statvfs(mnt)
+			except OSError:
+				mntstat = None
 			dev = blockbynode[nodestr]
 			dev.mountdev = mdev
 			dev.mountpoint = mnt

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[John Robinson]

On 26/05/2011 10:45, Torbjørn Skagestad wrote:
> Hi,
>
> Great tool, thanks for sharing.
>
> I had to add some error handling to get it to work properly.
> Currently it runs on Ubuntu 10.04, 10.10 and 11.04.
>
> Added the check John Robinson asked for as well.
>
> Attached patch for those interested.

Thanks, Torbjørn!

Getting there:

[root@beast lsdrv]# python2.6 lsdrv
PCI [pata_marvell] 03:00.0 IDE interface: Marvell Technology Group Ltd. 
88SE6121 SATA II Controller (rev b2)
  └─scsi 0:0:0:0 HL-DT-ST DVD-RAM GH22NP20
     └─sr0: Empty/Unknown 1.00g
PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 
Family) SATA AHCI Controller
  ├─scsi 2:0:0:0 ATA Hitachi HDS72101
  │  └─sda: Empty/Unknown 931.51g
Traceback (most recent call last):
   File "lsdrv", line 387, in <module>
     show_blocks(" %s " % branch[0], [phy.block])
   File "lsdrv", line 339, in show_blocks
     show_blocks("%s %s " % (indent, branch[0]), [blockbyname[x] for x 
in subs])
KeyError: 'sda1'

Now, something's not getting picked up about sda. Looking at Mathias' 
"sweet" output, it's not coping with the (DOS) partition table. Another 
variation on my kernel's /sys or still to old a Python or ...?

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 05:59 AM, John Robinson wrote:
> On 26/05/2011 10:45, Torbjørn Skagestad wrote:
>> Hi,
>>
>> Great tool, thanks for sharing.
>>
>> I had to add some error handling to get it to work properly.
>> Currently it runs on Ubuntu 10.04, 10.10 and 11.04.
>>
>> Added the check John Robinson asked for as well.
>>
>> Attached patch for those interested.
> 
> Thanks, Torbjørn!
> 
> Getting there:
> 
> [root@beast lsdrv]# python2.6 lsdrv
> PCI [pata_marvell] 03:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II Controller (rev b2)
>  └─scsi 0:0:0:0 HL-DT-ST DVD-RAM GH22NP20
>     └─sr0: Empty/Unknown 1.00g
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
>  ├─scsi 2:0:0:0 ATA Hitachi HDS72101
>  │  └─sda: Empty/Unknown 931.51g
> Traceback (most recent call last):
>   File "lsdrv", line 387, in <module>
>     show_blocks(" %s " % branch[0], [phy.block])
>   File "lsdrv", line 339, in show_blocks
>     show_blocks("%s %s " % (indent, branch[0]), [blockbyname[x] for x in subs])
> KeyError: 'sda1'
> 
> Now, something's not getting picked up about sda. Looking at Mathias' "sweet" output, it's not coping with the (DOS) partition table. Another variation on my kernel's /sys or still to old a Python or ...?

Hmmm.  I'll set up another CentOS VM.  If I recall correctly, that kernel has the original block devices in folders named '.../block:sda1' instead of '.../block/sda1'.  I'll have to identify this in the initial sweep through sysfs.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Torbjørn Skagestad]

[-- Attachment #1: Type: text/plain, Size: 2052 bytes --]

On Thu, 2011-05-26 at 07:49 -0400, Phil Turmel wrote:
> On 05/26/2011 05:59 AM, John Robinson wrote:
> > On 26/05/2011 10:45, Torbjørn Skagestad wrote:
> >> Hi,
> >>
> >> Great tool, thanks for sharing.
> >>
> >> I had to add some error handling to get it to work properly.
> >> Currently it runs on Ubuntu 10.04, 10.10 and 11.04.
> >>
> >> Added the check John Robinson asked for as well.
> >>
> >> Attached patch for those interested.
> > 
> > Thanks, Torbjørn!
> > 
> > Getting there:
> > 
> > [root@beast lsdrv]# python2.6 lsdrv
> > PCI [pata_marvell] 03:00.0 IDE interface: Marvell Technology Group Ltd. 88SE6121 SATA II Controller (rev b2)
> >  └─scsi 0:0:0:0 HL-DT-ST DVD-RAM GH22NP20
> >     └─sr0: Empty/Unknown 1.00g
> > PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
> >  ├─scsi 2:0:0:0 ATA Hitachi HDS72101
> >  │  └─sda: Empty/Unknown 931.51g
> > Traceback (most recent call last):
> >   File "lsdrv", line 387, in <module>
> >     show_blocks(" %s " % branch[0], [phy.block])
> >   File "lsdrv", line 339, in show_blocks
> >     show_blocks("%s %s " % (indent, branch[0]), [blockbyname[x] for x in subs])
> > KeyError: 'sda1'
> > 
> > Now, something's not getting picked up about sda. Looking at Mathias' "sweet" output, it's not coping with the (DOS) partition table. Another variation on my kernel's /sys or still to old a Python or ...?
> 
> Hmmm.  I'll set up another CentOS VM.  If I recall correctly, that kernel has the original block devices in folders named '.../block:sda1' instead of '.../block/sda1'.  I'll have to identify this in the initial sweep through sysfs.
> 

For CentOS 5.4 (2.6.18-128.1.14) it is /sys/block/sda/sda[1-9]


> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Torbjørn Skagestad
Idé Til Produkt AS
torborn@itpas.no

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[John Robinson]

On 26/05/2011 10:59, John Robinson wrote:
[...]
> [root@beast lsdrv]# python2.6 lsdrv
> PCI [pata_marvell] 03:00.0 IDE interface: Marvell Technology Group Ltd.
> 88SE6121 SATA II Controller (rev b2)
> └─scsi 0:0:0:0 HL-DT-ST DVD-RAM GH22NP20
> └─sr0: Empty/Unknown 1.00g
> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10
> Family) SATA AHCI Controller
> ├─scsi 2:0:0:0 ATA Hitachi HDS72101
> │ └─sda: Empty/Unknown 931.51g
> Traceback (most recent call last):
> File "lsdrv", line 387, in <module>
> show_blocks(" %s " % branch[0], [phy.block])
> File "lsdrv", line 339, in show_blocks
> show_blocks("%s %s " % (indent, branch[0]), [blockbyname[x] for x in subs])
> KeyError: 'sda1'
>
> Now, something's not getting picked up about sda. Looking at Mathias'
> "sweet" output, it's not coping with the (DOS) partition table. Another
> variation on my kernel's /sys or still to old a Python or ...?

Still seeing this. I added a print command so I can see that 
dev.partitions is being populated successfully. I'm not entirely sure 
where dev.ID_ etc are supposed to be coming from, but if it's that 
`blkid -p -o udev /dev/block/8:0` then I'm afraid CentOS 5's blkid 
doesn't understand the -p or -o udev options, it doesn't produce any 
output for whole drives with partition tables, and there isn't a 
/dev/block directory. It's blkid 1.0.0 from e2fsprogs 1.39-23.el5_5.1.

If that knocks CentOS 5 support on the head then so be it...

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[John Robinson]

On 27/05/2011 10:15, John Robinson wrote:
[...]
> I'm not entirely sure where dev.ID_ etc are supposed to be coming
> from, but if it's that `blkid -p -o udev /dev/block/8:0` then I'm
> afraid CentOS 5's blkid doesn't understand the -p or -o udev options,
> it doesn't produce any output for whole drives with partition tables,
> and there isn't a /dev/block directory. It's blkid 1.0.0 from
> e2fsprogs 1.39-23.el5_5.1.
>
> If that knocks CentOS 5 support on the head then so be it...

Hmm, udevinfo might be of some use. Still doesn't say it's found a DOS
partition table, but it does get you e.g. ID_FS_TYPE=linux_raid_member 
and perhaps `file -s` will tell you there's DOS partition table (sort of).

Cheers,

John.

Re: Storage device enumeration script

[Phil Turmel]

On 05/27/2011 05:44 AM, John Robinson wrote:
> On 27/05/2011 10:15, John Robinson wrote:
> [...]
>> I'm not entirely sure where dev.ID_ etc are supposed to be coming
>> from, but if it's that `blkid -p -o udev /dev/block/8:0` then I'm
>> afraid CentOS 5's blkid doesn't understand the -p or -o udev options,
>> it doesn't produce any output for whole drives with partition tables,
>> and there isn't a /dev/block directory. It's blkid 1.0.0 from
>> e2fsprogs 1.39-23.el5_5.1.
>>
>> If that knocks CentOS 5 support on the head then so be it...
> 
> Hmm, udevinfo might be of some use. Still doesn't say it's found a DOS
> partition table, but it does get you e.g. ID_FS_TYPE=linux_raid_member and perhaps `file -s` will tell you there's DOS partition table (sort of).

I'll look into this when I have a new CentOS 5 VM installed on my laptop.  I do want lsdrv to work with all of the CentOS 5 releases.

Phil

Re: Storage device enumeration script

[Phil Turmel]

Hi John,

On 05/27/2011 07:23 AM, Phil Turmel wrote:
> On 05/27/2011 05:44 AM, John Robinson wrote:
>> On 27/05/2011 10:15, John Robinson wrote:
>> [...]
>>> I'm not entirely sure where dev.ID_ etc are supposed to be coming
>>> from, but if it's that `blkid -p -o udev /dev/block/8:0` then I'm
>>> afraid CentOS 5's blkid doesn't understand the -p or -o udev options,
>>> it doesn't produce any output for whole drives with partition tables,
>>> and there isn't a /dev/block directory. It's blkid 1.0.0 from
>>> e2fsprogs 1.39-23.el5_5.1.
>>>
>>> If that knocks CentOS 5 support on the head then so be it...
>>
>> Hmm, udevinfo might be of some use. Still doesn't say it's found a DOS
>> partition table, but it does get you e.g. ID_FS_TYPE=linux_raid_member and perhaps `file -s` will tell you there's DOS partition table (sort of).
> 
> I'll look into this when I have a new CentOS 5 VM installed on my laptop.  I do want lsdrv to work with all of the CentOS 5 releases.

I've been playing with lsdrv in a CentOS 5 VM, and found a number of items to address.  The result has been pushed to github, with a detailed description.  Please give it a whirl.

https://github.com/pturmel/lsdrv

(Further bug reports should be posted there...  trying to keep the noise down on linux-raid.)

Phil

Re: Storage device enumeration script

[Phil Turmel]

Thanks Torbjørn !

I tend to short the error-checking when I'm putting something together...  Clearly, this script was less baked than I thought.

On 05/26/2011 05:45 AM, Torbjørn Skagestad wrote:
> @@ -166,9 +171,12 @@
>  	devpath = os.path.realpath(devpathlink)
>  	if devpath in phydevs:
>  		return phydevs[devpath]
> -	phy = Struct(dpath=devpath, node=nodestr,
> -		vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
> -		model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
> +	try:
> +		phy = Struct(dpath=devpath, node=nodestr,
> +			vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
> +			model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
> +	except IOError:
> +		return None
>  	if os.path.exists(devpath+'/unique_id'):
>  		phy.serial = io.FileIO(devpath+'/unique_id').read().split("\n",1)[0].strip()
>  	if not phy.serial:

This hunk will have to be done differently.  "phy" needs to be set without vendor & model if they can't be read.  I'll make a helper function to read the first line of a file, or return None.

On 05/26/2011 04:11 AM, CoolCold wrote:
> May be setup some github repo for this?
> 

Yes, I'll do this later today.  Obviously needed. :(

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Torbjørn Skagestad]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

On Thu, 2011-05-26 at 07:39 -0400, Phil Turmel wrote:
> Thanks Torbjørn !
> 
> I tend to short the error-checking when I'm putting something together...  Clearly, this script was less baked than I thought.
> 
> On 05/26/2011 05:45 AM, Torbjørn Skagestad wrote:
> > @@ -166,9 +171,12 @@
> >  	devpath = os.path.realpath(devpathlink)
> >  	if devpath in phydevs:
> >  		return phydevs[devpath]
> > -	phy = Struct(dpath=devpath, node=nodestr,
> > -		vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
> > -		model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
> > +	try:
> > +		phy = Struct(dpath=devpath, node=nodestr,
> > +			vendor=io.FileIO(devpath+'/vendor').read().split("\n",1)[0].strip(),
> > +			model=io.FileIO(devpath+'/model').read().split("\n",1)[0].strip())
> > +	except IOError:
> > +		return None
> >  	if os.path.exists(devpath+'/unique_id'):
> >  		phy.serial = io.FileIO(devpath+'/unique_id').read().split("\n",1)[0].strip()
> >  	if not phy.serial:
> 
> This hunk will have to be done differently.  "phy" needs to be set without vendor & model if they can't be read.  I'll make a helper function to read the first line of a file, or return None.

That makes sense.
Guess I was too trigger happy here and ignored some devices in the
process.
> 
> On 05/26/2011 04:11 AM, CoolCold wrote:
> > May be setup some github repo for this?
> > 
> 
> Yes, I'll do this later today.  Obviously needed. :(
> 
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Torbjørn Skagestad
Idé Til Produkt AS
torborn@itpas.no

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 07:39 AM, Phil Turmel wrote:
> On 05/26/2011 04:11 AM, CoolCold wrote:
>> May be setup some github repo for this?
>>
> 
> Yes, I'll do this later today.  Obviously needed. :(

Done.  With part of Torbjørn's patch and further fixes as proper commits.

https://github.com/pturmel/lsdrv

if you just want the latest script:

https://github.com/pturmel/lsdrv/raw/HEAD/lsdrv

Bug reports welcome. Patches, too.

I'm especially interested in users with hardware raid controllers...  not all of them expose their devices through the SCSI subsystem.  I've an older HP SmartArray here to play with, so I know it basically works.  I'm interested in enumerating the physical drives hooked to these controller subsystems.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Mathias Burén]

On 26 May 2011 18:46, Phil Turmel <philip@turmel.org> wrote:
> On 05/26/2011 07:39 AM, Phil Turmel wrote:
>> On 05/26/2011 04:11 AM, CoolCold wrote:
>>> May be setup some github repo for this?
>>>
>>
>> Yes, I'll do this later today.  Obviously needed. :(
>
> Done.  With part of Torbjørn's patch and further fixes as proper commits.
>
> https://github.com/pturmel/lsdrv
>
> if you just want the latest script:
>
> https://github.com/pturmel/lsdrv/raw/HEAD/lsdrv
>
> Bug reports welcome. Patches, too.
>
> I'm especially interested in users with hardware raid controllers...  not all of them expose their devices through the SCSI subsystem.  I've an older HP SmartArray here to play with, so I know it basically works.  I'm interested in enumerating the physical drives hooked to these controller subsystems.
>
> Phil
>

I think all looks good now:

 ~/bin $ sudo python2 ./lsdrv
PCI [ahci] 00:0b.0 SATA controller: nVidia Corporation MCP79 AHCI
Controller (rev b1)
 ├─scsi 0:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
 │  └─sda: Partitioned (dos) 55.90g
 │     ├─sda1: (ext4) 100.00m 'ssd_boot' {ae879f86-73a4-451f-bb6b-e778ad1b57d6}
 │     │  └─Mounted as /dev/sda1 @ /boot
 │     ├─sda2: (swap) 2.00g 'ssd_swap' {a28e32fa-628c-419a-9693-ca88166d230f}
 │     └─sda3: (ext4) 53.80g 'ssd_root' {6e812ed7-01c4-4a76-ae31-7b3d36d847f5}
 │        └─Mounted as /dev/disk/by-label/ssd_root @ /
 ├─scsi 1:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA1022443}
 │  └─sdb: Partitioned (dos) 1.82t
 │     └─sdb1: MD raid6 (1/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 │        └─md0: PV LVM2_member 9.08t/9.08t VG lvstorage 9.08t
{YLEUKB-klxF-X3gF-6dG3-DL4R-xebv-6gKQc2}
 │            └─Volume Group lvstorage (md0) 0 free {
Xd0HTM-azdN-v9kJ-C7vD-COcU-Cnn8-6AJ6hI}
 │              └─dm-0: (ext4) 9.08t 'storage'
{0ca82f13-680f-4b0d-a5d0-08c246a838e5}
 │                 └─Mounted as /dev/mapper/lvstorage-storage @ /raid6volume
 ├─scsi 2:0:0:0 ATA WDC WD20EARS-00M {WD-WMAZ20152590}
 │  └─sdc: Partitioned (dos) 1.82t
 │     └─sdc1: MD raid6 (3/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 3:0:0:0 ATA WDC WD20EARS-00M {WD-WMAZ20188479}
 │  └─sdd: Partitioned (dos) 1.82t
 │     └─sdd1: MD raid6 (2/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 4:x:x:x [Empty]
 └─scsi 5:x:x:x [Empty]
PCI [sata_mv] 05:00.0 SCSI storage controller: HighPoint Technologies,
Inc. RocketRAID 230x 4 Port SATA-II Controller (rev 02)
 ├─scsi 6:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA3609190}
 │  └─sde: Partitioned (dos) 1.82t
 │     └─sde1: MD raid6 (6/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 7:0:0:0 ATA SAMSUNG HD204UI {S2HGJ1RZ800964}
 │  └─sdf: Partitioned (dos) 1.82t
 │     └─sdf1: MD raid6 (4/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 ├─scsi 8:0:0:0 ATA WDC WD20EARS-00M {WD-WCAZA1000331}
 │  └─sdg: Partitioned (dos) 1.82t
 │     └─sdg1: MD raid6 (0/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}
 └─scsi 9:0:0:0 ATA SAMSUNG HD204UI {S2HGJ1RZ800850}
    └─sdh: Partitioned (dos) 1.82t
       └─sdh1: MD raid6 (5/7) 1.82t md0 clean in_sync 'ion:0'
{e6595c64-b3ae-90b3-f011-33ac3f402d20}

Nice.

/M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On Thu, 26 May 2011 13:46:59 -0400
Phil Turmel <philip@turmel.org> wrote:

> if you just want the latest script:
> 
> https://github.com/pturmel/lsdrv/raw/HEAD/lsdrv

I already reported this earlier, but still in this version:

Traceback (most recent call last):
  File "./lsdrv", line 274, in <module>
    probe_block('/sys/block/'+x)
  File "./lsdrv", line 226, in probe_block
    dev.phy = probe_device(blkpath+'/device', nodestr)
  File "./lsdrv", line 193, in probe_device
    vendor=fileline1(devpath+'/vendor'),
  File "./lsdrv", line 49, in fileline1
    fh = open(filename, 'r')
IOError: [Errno 2] No such file or directory:
'/sys/devices/platform/floppy.0/vendor'

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[Phil Turmel]

Hi Roman,

On 05/26/2011 01:54 PM, Roman Mamedov wrote:
> On Thu, 26 May 2011 13:46:59 -0400
> Phil Turmel <philip@turmel.org> wrote:
> 
>> if you just want the latest script:
>>
>> https://github.com/pturmel/lsdrv/raw/HEAD/lsdrv
> 
> I already reported this earlier, but still in this version:
> 
> Traceback (most recent call last):
>   File "./lsdrv", line 274, in <module>
>     probe_block('/sys/block/'+x)
>   File "./lsdrv", line 226, in probe_block
>     dev.phy = probe_device(blkpath+'/device', nodestr)
>   File "./lsdrv", line 193, in probe_device
>     vendor=fileline1(devpath+'/vendor'),
>   File "./lsdrv", line 49, in fileline1
>     fh = open(filename, 'r')
> IOError: [Errno 2] No such file or directory:
> '/sys/devices/platform/floppy.0/vendor'

I used the wrong exception type.  Fix pushed.  Try again?

Phil

Re: Storage device enumeration script

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 2788 bytes --]

On Thu, 26 May 2011 14:02:41 -0400
Phil Turmel <philip@turmel.org> wrote:

> Hi Roman,
> 
> On 05/26/2011 01:54 PM, Roman Mamedov wrote:
> > On Thu, 26 May 2011 13:46:59 -0400
> > Phil Turmel <philip@turmel.org> wrote:
> > 
> >> if you just want the latest script:
> >>
> >> https://github.com/pturmel/lsdrv/raw/HEAD/lsdrv
> > 
> > I already reported this earlier, but still in this version:
> > 
> > Traceback (most recent call last):
> >   File "./lsdrv", line 274, in <module>
> >     probe_block('/sys/block/'+x)
> >   File "./lsdrv", line 226, in probe_block
> >     dev.phy = probe_device(blkpath+'/device', nodestr)
> >   File "./lsdrv", line 193, in probe_device
> >     vendor=fileline1(devpath+'/vendor'),
> >   File "./lsdrv", line 49, in fileline1
> >     fh = open(filename, 'r')
> > IOError: [Errno 2] No such file or directory:
> > '/sys/devices/platform/floppy.0/vendor'
> 
> I used the wrong exception type.  Fix pushed.  Try again?

Now it locks up with 100% CPU load and no output, I waited for a couple of
minutes. On Ctrl-C:

^CTraceback (most recent call last):
  File "./lsdrv", line 274, in <module>
    probe_block('/sys/block/'+x)

$ ls /sys/block/
etherd!e1.5  etherd!e2.1  md0  md2  sda  sdc  sde  sdg
etherd!e1.6  fd0          md1  md4  sdb  sdd  sdf

The first two devices are actually down at this moment, maybe that's the
reason? Still I'd expect not 100% CPU load by lsdrv, but 0% CPU and 100%
iowait in this case.

Output of the old (bash) lsdrv:

Controller device @ pci0000:00/0000:00:06.0 [pata_amd]
  IDE interface: nVidia Corporation CK804 IDE (rev f2)
    host8: [Empty]
    host9: [Empty]
Controller device @ pci0000:00/0000:00:07.0 [sata_nv]
  IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
    host6: /dev/sdd ATA Hitachi HDS5C302 {SN: ..............}
    host7: /dev/sde ATA WDC WD15EADS-00S {SN: ..............}
Controller device @ pci0000:00/0000:00:08.0 [sata_nv]
  IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
    host10: /dev/sdf ATA WDC WD20EADS-00S {SN: ..............}
    host11: /dev/sdg ATA WDC WD20EADS-00S {SN: ..............}
Controller device @ pci0000:00/0000:00:0d.0/0000:02:00.0 [ahci]
  SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s
controller (rev 10) host4: /dev/sdc ATA Hitachi HDS5C302 {SN: ..............}
    host5: [Empty]
Controller device @ pci0000:00/0000:00:0e.0/0000:01:00.0 [sata_mv]
  SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port
SATA-II (rev 02) host0: [Empty]
    host1: [Empty]
    host2: /dev/sda ATA ST31000528AS {SN: ..............}
    host3: /dev/sdb ATA Hitachi HDS72202 {SN: ..............}


-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 02:12 PM, Roman Mamedov wrote:
[...]
> Now it locks up with 100% CPU load and no output, I waited for a couple of
> minutes. On Ctrl-C:
> 
> ^CTraceback (most recent call last):
>   File "./lsdrv", line 274, in <module>
>     probe_block('/sys/block/'+x)
> 
> $ ls /sys/block/
> etherd!e1.5  etherd!e2.1  md0  md2  sda  sdc  sde  sdg
> etherd!e1.6  fd0          md1  md4  sdb  sdd  sdf
> 
> The first two devices are actually down at this moment, maybe that's the
> reason? Still I'd expect not 100% CPU load by lsdrv, but 0% CPU and 100%
> iowait in this case.

Sounds like an infinite loop, or infinite recursion.  Could you apply the temporary patch below so I can see how far the probing got?

> Output of the old (bash) lsdrv:
> 
> Controller device @ pci0000:00/0000:00:06.0 [pata_amd]
>   IDE interface: nVidia Corporation CK804 IDE (rev f2)
>     host8: [Empty]
>     host9: [Empty]
> Controller device @ pci0000:00/0000:00:07.0 [sata_nv]
>   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
>     host6: /dev/sdd ATA Hitachi HDS5C302 {SN: ..............}
>     host7: /dev/sde ATA WDC WD15EADS-00S {SN: ..............}
> Controller device @ pci0000:00/0000:00:08.0 [sata_nv]
>   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
>     host10: /dev/sdf ATA WDC WD20EADS-00S {SN: ..............}
>     host11: /dev/sdg ATA WDC WD20EADS-00S {SN: ..............}
> Controller device @ pci0000:00/0000:00:0d.0/0000:02:00.0 [ahci]
>   SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s
> controller (rev 10) host4: /dev/sdc ATA Hitachi HDS5C302 {SN: ..............}
>     host5: [Empty]
> Controller device @ pci0000:00/0000:00:0e.0/0000:01:00.0 [sata_mv]
>   SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port
> SATA-II (rev 02) host0: [Empty]
>     host1: [Empty]
>     host2: /dev/sda ATA ST31000528AS {SN: ..............}
>     host3: /dev/sdb ATA Hitachi HDS72202 {SN: ..............}

The old code never attempted to recurse into the layers of block devices, so it can't have recursion problems.

Phil

8<-------------------------

diff --git a/lsdrv b/lsdrv
index d1caaf8..37728c1 100755
--- a/lsdrv
+++ b/lsdrv
@@ -145,6 +145,7 @@ def sect2size(sectors):
 # the struct object w/ filled in details.
 controllers=dict()
 def probe_controller(cpathlink):
+	print "Probing controller %s" % cpathlink
 	cpath = os.path.realpath(cpathlink)
 	if cpath in controllers:
 		return controllers[cpath]
@@ -186,6 +187,7 @@ def probe_controller(cpathlink):
 # controller.
 phydevs=dict()
 def probe_device(devpathlink, nodestr):
+	print "Probing device %s" % devpathlink
 	devpath = os.path.realpath(devpathlink)
 	if devpath in phydevs:
 		return phydevs[devpath]
@@ -214,6 +216,7 @@ def probe_device(devpathlink, nodestr):
 blockbyname=dict()
 blockbynode=dict()
 def probe_block(blocklink):
+	print "Probing block %s" % blocklink
 	name=blocklink.rsplit('/', 1)[-1]
 	if name in blockbyname:
 		return

Re: Storage device enumeration script

[Torbjørn Skagestad]

Hi,

I'm hitting this as well.It seems to be triggered by my non-existing
floppy device.

As there is no real floppy device connected, no driver is loaded, so we
never see a /driver directory. This loop then goes all the way up to /,
where it loops forever at 100% cpu.


from probe_controller()
151 while cpath and not os.path.exists(cpath+'/driver'):
152		cpath = os.path.dirname(cpath)
153		if cpath in controllers:
154			return controllers[cpath]


On Thu, 2011-05-26 at 14:22 -0400, Phil Turmel wrote:
> On 05/26/2011 02:12 PM, Roman Mamedov wrote:
> [...]
> > Now it locks up with 100% CPU load and no output, I waited for a couple of
> > minutes. On Ctrl-C:
> > 
> > ^CTraceback (most recent call last):
> >   File "./lsdrv", line 274, in <module>
> >     probe_block('/sys/block/'+x)
> > 
> > $ ls /sys/block/
> > etherd!e1.5  etherd!e2.1  md0  md2  sda  sdc  sde  sdg
> > etherd!e1.6  fd0          md1  md4  sdb  sdd  sdf
> > 
> > The first two devices are actually down at this moment, maybe that's the
> > reason? Still I'd expect not 100% CPU load by lsdrv, but 0% CPU and 100%
> > iowait in this case.
> 
> Sounds like an infinite loop, or infinite recursion.  Could you apply the temporary patch below so I can see how far the probing got?
> 
> > Output of the old (bash) lsdrv:
> > 
> > Controller device @ pci0000:00/0000:00:06.0 [pata_amd]
> >   IDE interface: nVidia Corporation CK804 IDE (rev f2)
> >     host8: [Empty]
> >     host9: [Empty]
> > Controller device @ pci0000:00/0000:00:07.0 [sata_nv]
> >   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
> >     host6: /dev/sdd ATA Hitachi HDS5C302 {SN: ..............}
> >     host7: /dev/sde ATA WDC WD15EADS-00S {SN: ..............}
> > Controller device @ pci0000:00/0000:00:08.0 [sata_nv]
> >   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
> >     host10: /dev/sdf ATA WDC WD20EADS-00S {SN: ..............}
> >     host11: /dev/sdg ATA WDC WD20EADS-00S {SN: ..............}
> > Controller device @ pci0000:00/0000:00:0d.0/0000:02:00.0 [ahci]
> >   SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s
> > controller (rev 10) host4: /dev/sdc ATA Hitachi HDS5C302 {SN: ..............}
> >     host5: [Empty]
> > Controller device @ pci0000:00/0000:00:0e.0/0000:01:00.0 [sata_mv]
> >   SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port
> > SATA-II (rev 02) host0: [Empty]
> >     host1: [Empty]
> >     host2: /dev/sda ATA ST31000528AS {SN: ..............}
> >     host3: /dev/sdb ATA Hitachi HDS72202 {SN: ..............}
> 
> The old code never attempted to recurse into the layers of block devices, so it can't have recursion problems.
> 
> Phil
> 
> 8<-------------------------
> 
> diff --git a/lsdrv b/lsdrv
> index d1caaf8..37728c1 100755
> --- a/lsdrv
> +++ b/lsdrv
> @@ -145,6 +145,7 @@ def sect2size(sectors):
>  # the struct object w/ filled in details.
>  controllers=dict()
>  def probe_controller(cpathlink):
> +	print "Probing controller %s" % cpathlink
>  	cpath = os.path.realpath(cpathlink)
>  	if cpath in controllers:
>  		return controllers[cpath]
> @@ -186,6 +187,7 @@ def probe_controller(cpathlink):
>  # controller.
>  phydevs=dict()
>  def probe_device(devpathlink, nodestr):
> +	print "Probing device %s" % devpathlink
>  	devpath = os.path.realpath(devpathlink)
>  	if devpath in phydevs:
>  		return phydevs[devpath]
> @@ -214,6 +216,7 @@ def probe_device(devpathlink, nodestr):
>  blockbyname=dict()
>  blockbynode=dict()
>  def probe_block(blocklink):
> +	print "Probing block %s" % blocklink
>  	name=blocklink.rsplit('/', 1)[-1]
>  	if name in blockbyname:
>  		return

-- 
Torbjørn Skagestad
Ide Til Produkt A/S

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[CoolCold]

I guess we should move off this list, may be continue on github comments.

On Thu, May 26, 2011 at 10:42 PM, Torbjørn Skagestad <torbjorn@itpas.no> wrote:
> Hi,
>
> I'm hitting this as well.It seems to be triggered by my non-existing
> floppy device.
>
> As there is no real floppy device connected, no driver is loaded, so we
> never see a /driver directory. This loop then goes all the way up to /,
> where it loops forever at 100% cpu.
>
>
> from probe_controller()
> 151 while cpath and not os.path.exists(cpath+'/driver'):
> 152             cpath = os.path.dirname(cpath)
> 153             if cpath in controllers:
> 154                     return controllers[cpath]
>
>
> On Thu, 2011-05-26 at 14:22 -0400, Phil Turmel wrote:
>> On 05/26/2011 02:12 PM, Roman Mamedov wrote:
>> [...]
>> > Now it locks up with 100% CPU load and no output, I waited for a couple of
>> > minutes. On Ctrl-C:
>> >
>> > ^CTraceback (most recent call last):
>> >   File "./lsdrv", line 274, in <module>
>> >     probe_block('/sys/block/'+x)
>> >
>> > $ ls /sys/block/
>> > etherd!e1.5  etherd!e2.1  md0  md2  sda  sdc  sde  sdg
>> > etherd!e1.6  fd0          md1  md4  sdb  sdd  sdf
>> >
>> > The first two devices are actually down at this moment, maybe that's the
>> > reason? Still I'd expect not 100% CPU load by lsdrv, but 0% CPU and 100%
>> > iowait in this case.
>>
>> Sounds like an infinite loop, or infinite recursion.  Could you apply the temporary patch below so I can see how far the probing got?
>>
>> > Output of the old (bash) lsdrv:
>> >
>> > Controller device @ pci0000:00/0000:00:06.0 [pata_amd]
>> >   IDE interface: nVidia Corporation CK804 IDE (rev f2)
>> >     host8: [Empty]
>> >     host9: [Empty]
>> > Controller device @ pci0000:00/0000:00:07.0 [sata_nv]
>> >   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
>> >     host6: /dev/sdd ATA Hitachi HDS5C302 {SN: ..............}
>> >     host7: /dev/sde ATA WDC WD15EADS-00S {SN: ..............}
>> > Controller device @ pci0000:00/0000:00:08.0 [sata_nv]
>> >   IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
>> >     host10: /dev/sdf ATA WDC WD20EADS-00S {SN: ..............}
>> >     host11: /dev/sdg ATA WDC WD20EADS-00S {SN: ..............}
>> > Controller device @ pci0000:00/0000:00:0d.0/0000:02:00.0 [ahci]
>> >   SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s
>> > controller (rev 10) host4: /dev/sdc ATA Hitachi HDS5C302 {SN: ..............}
>> >     host5: [Empty]
>> > Controller device @ pci0000:00/0000:00:0e.0/0000:01:00.0 [sata_mv]
>> >   SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port
>> > SATA-II (rev 02) host0: [Empty]
>> >     host1: [Empty]
>> >     host2: /dev/sda ATA ST31000528AS {SN: ..............}
>> >     host3: /dev/sdb ATA Hitachi HDS72202 {SN: ..............}
>>
>> The old code never attempted to recurse into the layers of block devices, so it can't have recursion problems.
>>
>> Phil
>>
>> 8<-------------------------
>>
>> diff --git a/lsdrv b/lsdrv
>> index d1caaf8..37728c1 100755
>> --- a/lsdrv
>> +++ b/lsdrv
>> @@ -145,6 +145,7 @@ def sect2size(sectors):
>>  # the struct object w/ filled in details.
>>  controllers=dict()
>>  def probe_controller(cpathlink):
>> +     print "Probing controller %s" % cpathlink
>>       cpath = os.path.realpath(cpathlink)
>>       if cpath in controllers:
>>               return controllers[cpath]
>> @@ -186,6 +187,7 @@ def probe_controller(cpathlink):
>>  # controller.
>>  phydevs=dict()
>>  def probe_device(devpathlink, nodestr):
>> +     print "Probing device %s" % devpathlink
>>       devpath = os.path.realpath(devpathlink)
>>       if devpath in phydevs:
>>               return phydevs[devpath]
>> @@ -214,6 +216,7 @@ def probe_device(devpathlink, nodestr):
>>  blockbyname=dict()
>>  blockbynode=dict()
>>  def probe_block(blocklink):
>> +     print "Probing block %s" % blocklink
>>       name=blocklink.rsplit('/', 1)[-1]
>>       if name in blockbyname:
>>               return
>
> --
> Torbjørn Skagestad
> Ide Til Produkt A/S
>
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storage device enumeration script

[Phil Turmel]

On 05/26/2011 02:58 PM, CoolCold wrote:
> I guess we should move off this list, may be continue on github comments.

I enabled the wiki on github for now.  At some point I'll create a dev list.

https://github.com/pturmel/lsdrv/wiki

And I've created a issue for the floppy drive problem(s).

Phil