Re: AS2105-based enclosure size issues with >2TB HDDs

Re: AS2105-based enclosure size issues with >2TB HDDs

[Oliver Neukum]

On Mon, 2014-08-25 at 10:58 +0000, Alfredo Dal Ava Junior wrote:

> - 1TB and 2TB: READ_CAPACITY_10 returns correct size value
> - 3TB and 4TB: READ_CAPACITY_10 returns size in a 2TB modulus
> 
> If we fix capacity size by reporting (READ_CAPACITY_10 + MODULO_2TB), the result
>  will be invalid when user plugs a <2TB HDD. An idea (bring by Oliver) is:

It is worse than that. Pretty soon a 4.7TB disk will also be plausible.

> first guess reading last sector using modulus result to check if size is valid.

It is necessary that a virgin disk also be handled correctly.
We cannot use the partition table (besides it being a layering
violation)

It would propose (in pseudo code)

if (read_capacity_16(device) < 0) {
	lower_limit = read_capacity_10(device);
	for (i = 1; i++; i < SANITY_LIMIT) {
		err = read_sector(device, lower_limit + i * 2TB-SAFETY);
		if (err == OUT_OF_RANGE)
			break;
	}
	if (i < SANITY_LIMIT)
		return (i - 1) * 2TB + lower_limit;
	else
		return ERROR;

	Regards
		Oliver


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AS2105-based enclosure size issues with >2TB HDDs

[Alan Stern]

On Mon, 25 Aug 2014, Oliver Neukum wrote:

> On Mon, 2014-08-25 at 10:58 +0000, Alfredo Dal Ava Junior wrote:
> 
> > - 1TB and 2TB: READ_CAPACITY_10 returns correct size value
> > - 3TB and 4TB: READ_CAPACITY_10 returns size in a 2TB modulus
> > 
> > If we fix capacity size by reporting (READ_CAPACITY_10 + MODULO_2TB), the result
> >  will be invalid when user plugs a <2TB HDD. An idea (bring by Oliver) is:
> 
> It is worse than that. Pretty soon a 4.7TB disk will also be plausible.
> 
> > first guess reading last sector using modulus result to check if size is valid.
> 
> It is necessary that a virgin disk also be handled correctly.
> We cannot use the partition table (besides it being a layering
> violation)
> 
> It would propose (in pseudo code)
> 
> if (read_capacity_16(device) < 0) {
> 	lower_limit = read_capacity_10(device);
> 	for (i = 1; i++; i < SANITY_LIMIT) {
> 		err = read_sector(device, lower_limit + i * 2TB-SAFETY);
> 		if (err == OUT_OF_RANGE)
> 			break;
> 	}
> 	if (i < SANITY_LIMIT)
> 		return (i - 1) * 2TB + lower_limit;
> 	else
> 		return ERROR;

Don't forget that lots of disks go crazy if you try to read from a 
nonexistent block, that is, one beyond the end of the disk.

IMO, this bug cannot be worked around in any reasonable manner.  The 
device simply cannot handle disks larger than 2 TB.

Alan Stern

RES: AS2105-based enclosure size issues with >2TB HDDs

[Alfredo Dal Ava Junior]

On Mon, 25 Aug 2014, Alan Stern wrote:
> 
> Don't forget that lots of disks go crazy if you try to read from a nonexistent
> block, that is, one beyond the end of the disk.
> IMO, this bug cannot be worked around in any reasonable manner.  The
> device simply cannot handle disks larger than 2 TB.


This device works well on Windows 7 if HDD is already partitioned. Sounds like Win7 gnores the READ_CAPACITY value on a partitioned HDD. It shows 4TB on disk manager, but will fall back to 1.8TB if I remove the partition.
Could we do the same? Would be possible to signalize to upper layers that capacity is not accurate (or return zero) on this device, and tell GPT handlers to bypass it's partition_size vs disk size consistency check?

Thanks,
Alfredo
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RES: AS2105-based enclosure size issues with >2TB HDDs

[James Bottomley]

On Mon, 2014-08-25 at 18:48 +0000, Alfredo Dal Ava Junior wrote:
> On Mon, 25 Aug 2014, Alan Stern wrote:
> > 
> > Don't forget that lots of disks go crazy if you try to read from a nonexistent
> > block, that is, one beyond the end of the disk.
> > IMO, this bug cannot be worked around in any reasonable manner.  The
> > device simply cannot handle disks larger than 2 TB.
> 
> 
> This device works well on Windows 7 if HDD is already partitioned.

So how did the partition get on there at the correct size in the first
place?  Even under windows partition managers believe the disk size they
get from the system if the disk is blank.

>  Sounds like Win7 gnores the READ_CAPACITY value on a partitioned HDD.
> It shows 4TB on disk manager, but will fall back to 1.8TB if I remove
> the partition.

I assume for those of us on linux-scsi who don't have the start of this
thread, you already tried read capacity(16) and it has this same
problem?

> Could we do the same?

Hm, allowing users to set desired capacity by overriding the partition
size ... I'm sure that's not going to cause support problems ...

>  Would be possible to signalize to upper layers that capacity is not
> accurate (or return zero) on this device, and tell GPT handlers to
> bypass it's partition_size vs disk size consistency check?

If we can find a heuristic to set the correct capacity in the kernel,
then we may be able to fix this ... but as Alain says, it looks hard.
We can't ask userspace to hack tools for broken devices.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alan Stern]

On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:

> On Mon, 25 Aug 2014, Alan Stern wrote:
> > 
> > Don't forget that lots of disks go crazy if you try to read from a nonexistent
> > block, that is, one beyond the end of the disk.
> > IMO, this bug cannot be worked around in any reasonable manner.  The
> > device simply cannot handle disks larger than 2 TB.
> 
> 
> This device works well on Windows 7 if HDD is already partitioned.
> Sounds like Win7 gnores the READ_CAPACITY value on a partitioned HDD.
> It shows 4TB on disk manager, but will fall back to 1.8TB if I remove
> the partition.

That's right.  I don't know why Windows behaves that way.

> Could we do the same? Would be possible to signalize to upper layers
> that capacity is not accurate (or return zero) on this device, and
> tell GPT handlers to bypass it's partition_size vs disk size
> consistency check?

There is no way to do this, as far as I know.  But I'm not an expert in 
this area.  Maybe you can figure out a way to add this capability.

(But then what happens if the size stored in the partition table really
is larger than the disk's capacity?)

Alan Stern

RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alfredo Dal Ava Junior]

On Mon, 15 Aug 2014 James Bottomley wrote:

> So how did the partition get on there at the correct size in the first place?
> Even under windows partition managers believe the disk size they get from
> the system if the disk is blank.

The HDD can be partitioned outside the enclosure, when connected directly to one SATA port on motherboard. READ_CAPACITY(16) will return properly when talking directly to the HDD.

> I assume for those of us on linux-scsi who don't have the start of this thread,
> you already tried read capacity(16) and it has this same problem?

Sorry, I forgot to include linux-scsi. On this device, READ_CAPACITY_16 fails 100% of times as unsupported command, then  READ_CAPACITY_10 has a distinct behavior depending on HDD size:

1TB and 2TB: READ_CAPACITY_10 returns correct value
3TB and 4TB: READ_CAPACITY_10 returns in a 2TB modulus

> Hm, allowing users to set desired capacity by overriding the partition size ...
> I'm sure that's not going to cause support problems ...

Well, it is causing problems anyway... from user perspective, it's a Linux  compatibility issue, as it works "fine" on Windows. I'm not an expert, but I'm wondering that if usb-storage could set capacity as "UNDETERMINED"/ Zero  (or keep using the readcapacity_10 as it as with some flag signalizing it as inaccurate), EFI partition check would be able to ignore size and look for secondary GPT where it really is.

[]'s
Alfredo

RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alfredo Dal Ava Junior]

On Mon, 25 Aug 2014 Alan Stern wrote:
> 
> On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:
> 
> That's right.  I don't know why Windows behaves that way.

Please look this output from diskpart (Windows):

DISKPART> list partition

  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
  Partition 1    Primary           3726 GB    17 KB

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          298 GB      0 B
* Disk 1    Online         1678 GB      0 B        *

DISKPART> select disk 1

Disk 1 is now the selected disk.

DISKPART> list partition

  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
  Partition 1    Primary           3726 GB    17 KB

> > Could we do the same? Would be possible to signalize to upper layers
> > that capacity is not accurate (or return zero) on this device, and
> > tell GPT handlers to bypass it's partition_size vs disk size
> > consistency check?
> 
> There is no way to do this, as far as I know.  But I'm not an expert in this area.
> Maybe you can figure out a way to add this capability.
> 
> (But then what happens if the size stored in the partition table really is larger
> than the disk's capacity?)

It's the first time I touch this code, but I will learn from the code and try to find it out... but still in hope to find a clean solution...
If size stored on partition table is really larger than disk, my guess it will cause I/O errors, and that some disks may get crazy like you pointed. 
Do you think disk could cause kernel instability? I think it is acceptable if no consequences to kernel stability, since it is a "specific-device" workaround. 

[]'s
Alfredo
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alan Stern]

On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:

> Well, it is causing problems anyway... from user perspective, it's a
> Linux compatibility issue, as it works "fine" on Windows. I'm not an
> expert, but I'm wondering that if usb-storage could set capacity as
> "UNDETERMINED"/ Zero (or keep using the readcapacity_10 as it as with
> some flag signalizing it as inaccurate), EFI partition check would be
> able to ignore size and look for secondary GPT where it really is.

Part of the problem is that usb-storage has no way to know that
anything strange is going on.  It's normal for READ CAPACITY(16) to
fail (this depend on the SCSI level), and it's normal for the READ
CAPACITY(10) to report a value less than 2 TB.

Really there is only one way to know whether the actual capacity is 
larger than the reported capacity, and that is by trying to read blocks 
beyond the reported capacity -- a dangerous test that many drives do 
not like.  (And in most cases a futile test.  If a device doesn't 
support READ CAPACITY(16), how likely is it to support READ(16)?)

Yes, in theory you can believe the value in the partition table in 
preference to the reported capacity.  But what if that value is wrong?  
And how do you tell partition-manager programs what the capacity should 
be when they modify or set up the initial partition table?

Attaching the drive over a SATA connection when you want to partition
it isn't a very satisfactory solution.  After all, if you have a SATA
connection available then why would you be using a USB enclosure in the
first place?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RES: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alfredo Dal Ava Junior]

On Mon, 25 Aug 2014 Alan Stern wrote:
> Part of the problem is that usb-storage has no way to know that anything
> strange is going on.  It's normal for READ CAPACITY(16) to fail (this depend on
> the SCSI level), and it's normal for the READ
> CAPACITY(10) to report a value less than 2 TB.
> Really there is only one way to know whether the actual capacity is larger
> than the reported capacity, and that is by trying to read blocks beyond the
> reported capacity -- a dangerous test that many drives do not like.  (And in
> most cases a futile test.  If a device doesn't support READ CAPACITY(16), how
> likely is it to support READ(16)?)
> Yes, in theory you can believe the value in the partition table in preference to
> the reported capacity.  But what if that value is wrong?
> And how do you tell partition-manager programs what the capacity should be
> when they modify or set up the initial partition table?

We may add this device to UNUSUAL_DEV list, to keep using READ_CAPACITY(10)  and 
pass  some flag to bypass EFI GPT partition check. I'm almost sure that kernel will be able
to mount the partition if we can make it available on /proc/partition. This would make this 
device behaves like it does on Windows 7.

I see this as best effort workaround for a cheap buggy hardware, like many others on
UNUSUAL_DEV list

> Attaching the drive over a SATA connection when you want to partition it
> isn't a very satisfactory solution.  After all, if you have a SATA connection
> available then why would you be using a USB enclosure in the first place?

You may want do a backup or plug it in a laptop, for example.

[]'s
Alfredo

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Oliver Neukum]

On Mon, 2014-08-25 at 16:21 -0400, Alan Stern wrote:
> On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:
> 
> > Well, it is causing problems anyway... from user perspective, it's a
> > Linux compatibility issue, as it works "fine" on Windows. I'm not an
> > expert, but I'm wondering that if usb-storage could set capacity as
> > "UNDETERMINED"/ Zero (or keep using the readcapacity_10 as it as
> with
> > some flag signalizing it as inaccurate), EFI partition check would
> be
> > able to ignore size and look for secondary GPT where it really is.
> 
> Part of the problem is that usb-storage has no way to know that
> anything strange is going on.  It's normal for READ CAPACITY(16) to
> fail (this depend on the SCSI level), and it's normal for the READ
> CAPACITY(10) to report a value less than 2 TB.

Just set US_FL_NEEDS_CAP16. If READ CAPACITY(16) fails in that case,
it is clear that something is wrong. It must be set or READ CAPACITY(10)
alone would be taken as giving a valid answer.

At that time we are sure that the drive will be unusable unless we
determine the correct capacity, so we don't make matters worse if we
crash it.

Is there an easy way for Alfredo to determine what happens if we
read beyond the end?

	Regards
		Oliver

RE: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[David Laight]

From: Alan Stern
> On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:
> 
> > Well, it is causing problems anyway... from user perspective, it's a
> > Linux compatibility issue, as it works "fine" on Windows. I'm not an
> > expert, but I'm wondering that if usb-storage could set capacity as
> > "UNDETERMINED"/ Zero (or keep using the readcapacity_10 as it as with
> > some flag signalizing it as inaccurate), EFI partition check would be
> > able to ignore size and look for secondary GPT where it really is.
> 
> Part of the problem is that usb-storage has no way to know that
> anything strange is going on.  It's normal for READ CAPACITY(16) to
> fail (this depend on the SCSI level), and it's normal for the READ
> CAPACITY(10) to report a value less than 2 TB.

Could the code try READ CAPACITY(16) first?

> Really there is only one way to know whether the actual capacity is
> larger than the reported capacity, and that is by trying to read blocks
> beyond the reported capacity -- a dangerous test that many drives do
> not like.  (And in most cases a futile test.  If a device doesn't
> support READ CAPACITY(16), how likely is it to support READ(16)?)
> 
> Yes, in theory you can believe the value in the partition table in
> preference to the reported capacity.  But what if that value is wrong?
> And how do you tell partition-manager programs what the capacity should
> be when they modify or set up the initial partition table?

I've a feeling that, historically at least, windows believes the partition table.
I remember some CF cards that locked up when I tried to read the 'device info'
sector, and others (apparently identical) that had the 32bit sector size misaligned.
These were 'major manufacturer' cards as well.

	David

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Oliver Neukum]

On Tue, 2014-08-26 at 09:58 +0000, David Laight wrote:
> > Part of the problem is that usb-storage has no way to know that
> > anything strange is going on.  It's normal for READ CAPACITY(16) to
> > fail (this depend on the SCSI level), and it's normal for the READ
> > CAPACITY(10) to report a value less than 2 TB.
> 
> Could the code try READ CAPACITY(16) first?

Yes. It does already. That fails as the device doesn't support
this version. In a way we are discussing error handling here.

	Regards
		Oliver


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[David Laight]

From Oliver Neukum [mailto:oneukum@suse.de]
> On Tue, 2014-08-26 at 09:58 +0000, David Laight wrote:
> > > Part of the problem is that usb-storage has no way to know that
> > > anything strange is going on.  It's normal for READ CAPACITY(16) to
> > > fail (this depend on the SCSI level), and it's normal for the READ
> > > CAPACITY(10) to report a value less than 2 TB.
> >
> > Could the code try READ CAPACITY(16) first?
> 
> Yes. It does already. That fails as the device doesn't support
> this version. In a way we are discussing error handling here.

I read more of the thread later (getting outluck to sort mails in
any sensible way is almost impossible.)

I'm sort of surprised that the 16byte reads work if the 16byte
read capacity doesn't.

I wonder what the manufacturer would saw in response the bug where
windows shows the incorrect size when trying to partition the disk?

Such bugs ought to fail 'fitness for purpose' - so, in the UK, the
shop would have to replace the 'faulty' goods.

	David

RES: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alfredo Dal Ava Junior]

On Mon, 26 Aug 2014,  David Leight wrote:

> I wonder what the manufacturer would saw in response the bug where
> windows shows the incorrect size when trying to partition the disk?

I contacted enclosure manufacturer (Welland) some weeks ago, they are supposed to escalate my questions to engineering and I'm still waiting for reply.  Interesting thing is that I read AS1051 on the chip, but it is detected as AS2105 (maybe same VID and PID for different chips). I contacted ASMedia (chip bridge manufacturer), they replied that ASM1051 supports 48bit LBAs and that some FW feature and FW SPEC depends on manufacture's request. They added that this chip entered EOL this year and I have to contact Welland for support.
Anyway, it show explain way we see distinct behavior for same the chip (and same VID&PID).

[]'s
Alfredo

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alan Stern]

On Mon, 25 Aug 2014, Oliver Neukum wrote:

> On Mon, 2014-08-25 at 16:21 -0400, Alan Stern wrote:
> > On Mon, 25 Aug 2014, Alfredo Dal Ava Junior wrote:
> > 
> > > Well, it is causing problems anyway... from user perspective, it's a
> > > Linux compatibility issue, as it works "fine" on Windows. I'm not an
> > > expert, but I'm wondering that if usb-storage could set capacity as
> > > "UNDETERMINED"/ Zero (or keep using the readcapacity_10 as it as
> > with
> > > some flag signalizing it as inaccurate), EFI partition check would
> > be
> > > able to ignore size and look for secondary GPT where it really is.
> > 
> > Part of the problem is that usb-storage has no way to know that
> > anything strange is going on.  It's normal for READ CAPACITY(16) to
> > fail (this depend on the SCSI level), and it's normal for the READ
> > CAPACITY(10) to report a value less than 2 TB.
> 
> Just set US_FL_NEEDS_CAP16. If READ CAPACITY(16) fails in that case,
> it is clear that something is wrong. It must be set or READ CAPACITY(10)
> alone would be taken as giving a valid answer.

You have committed a mental layering violation.  :-)

US_FL_NEEDS_CAP16 is a usb-storage flag.  But the capacity
determination is made by the sd driver, which relies on the SCSI
try_rc_10_first flag.  If that flag isn't set, sd tries READ 
CAPACITY(16) and then falls back to READ CAPACITY(10) if an error 
occurs.  That's what happened here.

I don't think we want to add another SCSI flag to say that READ
CAPACITY(10) is unreliable.

> At that time we are sure that the drive will be unusable unless we
> determine the correct capacity, so we don't make matters worse if we
> crash it.

Given the difficulty of determining the true capacity, I see two
alternatives.  We could set the capacity to a ridiculously large value
(like 1 billion TB), or we could leave the capacity set to the low
value and disable the "block within bounds" checks.  Neither of these
is attractive and they both have issues of their own -- although the 
second is close to what Windows does.

(For example, udev often tries to read the last sectors of a new drive, 
looking for a GPT or RAID signature.  That won't work if the capacity 
is set to some random value.)

> Is there an easy way for Alfredo to determine what happens if we
> read beyond the end?

He could fiddle with sd_read_capacity().  Change sdkp->capacity to 
something bigger than the disk's actual size and then try reading 
beyond the actual end.

However, I don't see much point in this.  Any changes we make will have 
to work with every drive, not just Alfredo's.

Alan Stern

Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

This is almost certainly a form of the problem reported in
"AS2105-based enclosure size issues with >2TB HDDs".  I'm repeating my
original message here so linux-usb can see it, and so it can be
connected to the older thread.  I'll address it in another message.

I've appended James Bottomley's response, to avoid creating an
additional message.

Dale
----------
Date:	Wed, 20 Aug 2014 16:18:00 -0400
From: worley-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org (Dale R. Worley)
To: linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Problem with USB-to-SATA adapters

I don't know if this is the correct place for this problem, but you
people can probably tell me the correct place.

I have two "USB to SATA adapter" dongles.  In general, they work fine.
However I've discovered that one of them handles large (1 TB and
above) drives correctly and one does not.  In particular, when I use
the unsuccessful one, neither fdisk nor gdisk reports the disk size
correctly, and the kernel does not read the partition table.

More confusingly, both dongles work correctly with large disks on
Windows 7 Professional.

Both of the following cases are when a 3.0 TB drive is attached.

The successful device has the "Diablo" brand.  The most interesting
messages in the log when I plug in the USB are (the complete message
set is appended below):

Aug 18 21:56:53 hobgoblin kernel: [  294.229462] usb 2-1.3: New USB device found, idVendor=152d, idProduct=2338
Aug 18 21:56:53 hobgoblin kernel: [  294.229475] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=5
Aug 18 21:56:53 hobgoblin kernel: [  294.229482] usb 2-1.3: Product: USB to ATA/ATAPI bridge
Aug 18 21:56:53 hobgoblin kernel: [  294.229488] usb 2-1.3: Manufacturer: JMicron
Aug 18 21:56:53 hobgoblin kernel: [  294.229495] usb 2-1.3: SerialNumber: 000001D91CC4
Aug 18 21:56:54 hobgoblin kernel: [  295.236576] sd 7:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
Aug 18 21:56:54 hobgoblin kernel: [  295.236955] sd 7:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)

The unsuccessful device has the "Ez-Connect" brand.  The most
interesting messages in the log when I plug in the USB are (the
complete message set is appended below):

Aug 18 21:54:06 hobgoblin kernel: [  127.674374] usb 2-2: new high-speed USB device number 4 using ehci-pci
Aug 18 21:54:06 hobgoblin kernel: [  127.791128] usb 2-2: New USB device found, idVendor=05e3, idProduct=0718
Aug 18 21:54:06 hobgoblin kernel: [  127.791135] usb 2-2: New USB device strings: Mfr=0, Product=1, SerialNumber=2
Aug 18 21:54:06 hobgoblin kernel: [  127.791138] usb 2-2: Product: USB Storage
Aug 18 21:54:06 hobgoblin kernel: [  127.791142] usb 2-2: SerialNumber: 000000000033
Aug 18 21:54:07 hobgoblin kernel: [  128.847269] sd 6:0:0:0: [sdb] 1565565872 512-byte logical blocks: (801 GB/746 GiB)

Dale

----------------------------------------------------------------------
For the Diablo adapter:

Aug 18 21:56:53 hobgoblin kernel: [  294.153343] usb 2-1.3: new high-speed USB device number 5 using ehci-pci
Aug 18 21:56:53 hobgoblin kernel: [  294.229462] usb 2-1.3: New USB device found, idVendor=152d, idProduct=2338
Aug 18 21:56:53 hobgoblin kernel: [  294.229475] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=5
Aug 18 21:56:53 hobgoblin kernel: [  294.229482] usb 2-1.3: Product: USB to ATA/ATAPI bridge
Aug 18 21:56:53 hobgoblin kernel: [  294.229488] usb 2-1.3: Manufacturer: JMicron
Aug 18 21:56:53 hobgoblin kernel: [  294.229495] usb 2-1.3: SerialNumber: 000001D91CC4
Aug 18 21:56:53 hobgoblin kernel: [  294.231159] usb-storage 2-1.3:1.0: USB Mass Storage device detected
Aug 18 21:56:53 hobgoblin kernel: [  294.233951] scsi7 : usb-storage 2-1.3:1.0
Aug 18 21:56:53 hobgoblin mtp-probe: checking bus 2, device 5: "/sys/devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1.3"
Aug 18 21:56:53 hobgoblin mtp-probe: bus: 2, device: 5 was not an MTP device
Aug 18 21:56:54 hobgoblin kernel: [  295.235735] scsi 7:0:0:0: Direct-Access     WDC WD30 EZRX-00SPEB0          PQ: 0 ANSI: 5
Aug 18 21:56:54 hobgoblin kernel: [  295.236209] sd 7:0:0:0: Attached scsi generic sg2 type 0
Aug 18 21:56:54 hobgoblin kernel: [  295.236576] sd 7:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
Aug 18 21:56:54 hobgoblin kernel: [  295.236955] sd 7:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
Aug 18 21:56:54 hobgoblin kernel: [  295.237957] sd 7:0:0:0: [sdb] Write Protect is off
Aug 18 21:56:54 hobgoblin kernel: [  295.238957] sd 7:0:0:0: [sdb] No Caching mode page found
Aug 18 21:56:54 hobgoblin kernel: [  295.238962] sd 7:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:56:54 hobgoblin kernel: [  295.239956] sd 7:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
Aug 18 21:56:54 hobgoblin kernel: [  295.242334] sd 7:0:0:0: [sdb] No Caching mode page found
Aug 18 21:56:54 hobgoblin kernel: [  295.242339] sd 7:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:56:54 hobgoblin kernel: [  295.305631] GPT:Primary header thinks Alt. header is not at the end of the disk.
Aug 18 21:56:54 hobgoblin kernel: [  295.305638] GPT:1565565871 != 5860533167
Aug 18 21:56:54 hobgoblin kernel: [  295.305639] GPT:Alternate GPT header not at the end of the disk.
Aug 18 21:56:54 hobgoblin kernel: [  295.305640] GPT:1565565871 != 5860533167
Aug 18 21:56:54 hobgoblin kernel: [  295.305642] GPT: Use GNU Parted to correct GPT errors.
Aug 18 21:56:54 hobgoblin kernel: [  295.305653]  sdb: sdb1 sdb2 sdb3 sdb4
Aug 18 21:56:54 hobgoblin kernel: [  295.308235] sd 7:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
Aug 18 21:56:54 hobgoblin kernel: [  295.310974] sd 7:0:0:0: [sdb] No Caching mode page found
Aug 18 21:56:54 hobgoblin kernel: [  295.310980] sd 7:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:56:54 hobgoblin kernel: [  295.310984] sd 7:0:0:0: [sdb] Attached SCSI disk
----------------------------------------------------------------------
For the Ez-Connect adapter:

Aug 18 21:54:06 hobgoblin kernel: [  127.674374] usb 2-2: new high-speed USB device number 4 using ehci-pci
Aug 18 21:54:06 hobgoblin kernel: [  127.791128] usb 2-2: New USB device found, idVendor=05e3, idProduct=0718
Aug 18 21:54:06 hobgoblin kernel: [  127.791135] usb 2-2: New USB device strings: Mfr=0, Product=1, SerialNumber=2
Aug 18 21:54:06 hobgoblin kernel: [  127.791138] usb 2-2: Product: USB Storage
Aug 18 21:54:06 hobgoblin kernel: [  127.791142] usb 2-2: SerialNumber: 000000000033
Aug 18 21:54:06 hobgoblin mtp-probe: checking bus 2, device 4: "/sys/devices/pci0000:00/0000:00:1d.7/usb2/2-2"
Aug 18 21:54:06 hobgoblin mtp-probe: bus: 2, device: 4 was not an MTP device
Aug 18 21:54:06 hobgoblin kernel: [  127.839825] usb-storage 2-2:1.0: USB Mass Storage device detected
Aug 18 21:54:06 hobgoblin kernel: [  127.839917] scsi6 : usb-storage 2-2:1.0
Aug 18 21:54:06 hobgoblin kernel: [  127.840721] usbcore: registered new interface driver usb-storage
Aug 18 21:54:07 hobgoblin kernel: [  128.844942] scsi 6:0:0:0: Direct-Access     WDC WD30 EZRX-00SPEB0     0041 PQ: 0 ANSI: 0
Aug 18 21:54:07 hobgoblin kernel: [  128.846146] sd 6:0:0:0: Attached scsi generic sg2 type 0
Aug 18 21:54:07 hobgoblin kernel: [  128.847269] sd 6:0:0:0: [sdb] 1565565872 512-byte logical blocks: (801 GB/746 GiB)
Aug 18 21:54:07 hobgoblin kernel: [  128.848303] sd 6:0:0:0: [sdb] Write Protect is off
Aug 18 21:54:07 hobgoblin kernel: [  128.849389] sd 6:0:0:0: [sdb] No Caching mode page found
Aug 18 21:54:07 hobgoblin kernel: [  128.849398] sd 6:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:54:07 hobgoblin kernel: [  128.853028] sd 6:0:0:0: [sdb] No Caching mode page found
Aug 18 21:54:07 hobgoblin kernel: [  128.853038] sd 6:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:54:08 hobgoblin kernel: [  129.638416]  sdb: sdb1 sdb2 sdb3
Aug 18 21:54:08 hobgoblin kernel: [  129.641877] sd 6:0:0:0: [sdb] No Caching mode page found
Aug 18 21:54:08 hobgoblin kernel: [  129.641884] sd 6:0:0:0: [sdb] Assuming drive cache: write through
Aug 18 21:54:08 hobgoblin kernel: [  129.641889] sd 6:0:0:0: [sdb] Attached SCSI disk
----------------------------------------------------------------------
Subject: Re: Problem with USB-to-SATA adapters
From: James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>
Date: Wed, 20 Aug 2014 17:45:48 -0500

[clipped]

So the basic problem here seems to be that this adapter has wrapped the
INQUIRY response instead of giving us the the conventional response that
means my device is too big.  This could be because the processing engine
on the card doesn't understand 16 byte commands (which are required for
addressing capacity > 2TB) or it could just be a firmware SCSI emulator
mistake.  The best way of telling might be to use the SG_IO interface to
send a read capacity 16.  sg_readcap --16 should be able to do this.

If you get a sensible response, chances are we should just quirk this
device to apply RC16 first.  If something goes wrong, it's likely
unfixably broken for devices > 2TB

James
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[James Bottomley]

On Tue, 2014-08-26 at 15:39 -0400, Dale R. Worley wrote:
> This is almost certainly a form of the problem reported in
> "AS2105-based enclosure size issues with >2TB HDDs".  I'm repeating my
> original message here so linux-usb can see it, and so it can be
> connected to the older thread.  I'll address it in another message.

Did you try read capacity 16 on it?  What happened?  (the AS2105 rejects
read capacity 16, so there's no reliable way to deduce the capacity of
drives over 2TB).

James

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Oliver Neukum]

On Tue, 2014-08-26 at 10:47 -0400, Alan Stern wrote:
> On Mon, 25 Aug 2014, Oliver Neukum wrote:

> > Just set US_FL_NEEDS_CAP16. If READ CAPACITY(16) fails in that case,
> > it is clear that something is wrong. It must be set or READ CAPACITY(10)
> > alone would be taken as giving a valid answer.
> 
> You have committed a mental layering violation.  :-)

Indeed.

> US_FL_NEEDS_CAP16 is a usb-storage flag.  But the capacity
> determination is made by the sd driver, which relies on the SCSI
> try_rc_10_first flag.  If that flag isn't set, sd tries READ 
> CAPACITY(16) and then falls back to READ CAPACITY(10) if an error 
> occurs.  That's what happened here.
> 
> I don't think we want to add another SCSI flag to say that READ
> CAPACITY(10) is unreliable.

Why not? It would only be friendly to tell the upper layer
of a malfunction if we know about it.

> > At that time we are sure that the drive will be unusable unless we
> > determine the correct capacity, so we don't make matters worse if we
> > crash it.
> 
> Given the difficulty of determining the true capacity, I see two
> alternatives.  We could set the capacity to a ridiculously large value
> (like 1 billion TB), or we could leave the capacity set to the low
> value and disable the "block within bounds" checks.  Neither of these
> is attractive and they both have issues of their own -- although the 
> second is close to what Windows does.

That seems to be the most attractive solution to me.

> (For example, udev often tries to read the last sectors of a new drive, 
> looking for a GPT or RAID signature.  That won't work if the capacity 
> is set to some random value.)

Yes, but clipping has its own dangers. Suppose you use the medium
without a partition table.

	Regards
		Oliver



--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RES: RES: AS2105-based enclosure size issues with >2TB HDDs

[Alan Stern]

On Wed, 27 Aug 2014, Oliver Neukum wrote:

> > I don't think we want to add another SCSI flag to say that READ
> > CAPACITY(10) is unreliable.
> 
> Why not? It would only be friendly to tell the upper layer
> of a malfunction if we know about it.

To what end?  What will the upper layer do with this information?

> > Given the difficulty of determining the true capacity, I see two
> > alternatives.  We could set the capacity to a ridiculously large value
> > (like 1 billion TB), or we could leave the capacity set to the low
> > value and disable the "block within bounds" checks.  Neither of these
> > is attractive and they both have issues of their own -- although the 
> > second is close to what Windows does.
> 
> That seems to be the most attractive solution to me.

I'm skeptical that you can convince the SCSI and block-layer developers 
about this.  Maybe they'll accept it if it is applied only to USB 
transports...

> > (For example, udev often tries to read the last sectors of a new drive, 
> > looking for a GPT or RAID signature.  That won't work if the capacity 
> > is set to some random value.)
> 
> Yes, but clipping has its own dangers. Suppose you use the medium
> without a partition table.

What would Windows do?  In the absence of a partition table, it would 
believe the value from READ CAPACITY, right?  Isn't that the same as 
clipping?

Alan Stern

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

> From: James Bottomley <James.Bottomley@HansenPartnership.com>

> Did you try read capacity 16 on it?  What happened?  (the AS2105 rejects
> read capacity 16, so there's no reliable way to deduce the capacity of
> drives over 2TB).

OK, I had to track down which package contains sg_readcap.

The adapter that fails gives this output:

    # sg_readcap --16 /dev/sdb
    bad field in READ CAPACITY (16) cdb including unsupported service action

The adapter that succeeds gives this output:

    # sg_readcap --16 /dev/sdc
    Read Capacity results:
       Protection: prot_en=0, p_type=0, p_i_exponent=0
       Logical block provisioning: lbpme=0, lbprz=0
       Last logical block address=5860533167 (0x15d50a3af), Number of logical blocks=5860533168
       Logical block length=512 bytes
       Logical blocks per physical block exponent=0
       Lowest aligned logical block address=0
    Hence:
       Device size: 3000592982016 bytes, 2861588.5 MiB, 3000.59 GB

Dale

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

What I find interesting is that Windows (at least, Windows 7
Professional) seems to be able to handle the deficient adapter.  What
I'd like to do is log the disk commands during the mounting sequence,
preferably at both the SCSI and USB layers.  Then at least we'll know
exactly what the driver is doing.  Are there kernel options to do
this?

Unfortunately, I don't know any way to log what Windows is doing with
the drive.

Dale
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[James Bottomley]

On Wed, 2014-08-27 at 15:21 -0400, Dale R. Worley wrote:
> > From: James Bottomley <James.Bottomley@HansenPartnership.com>
> 
> > Did you try read capacity 16 on it?  What happened?  (the AS2105 rejects
> > read capacity 16, so there's no reliable way to deduce the capacity of
> > drives over 2TB).
> 
> OK, I had to track down which package contains sg_readcap.
> 
> The adapter that fails gives this output:
> 
>     # sg_readcap --16 /dev/sdb
>     bad field in READ CAPACITY (16) cdb including unsupported service action

OK, so that's definitely the tame taxonomy.

James

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Wed, 27 Aug 2014, Dale R. Worley wrote:

> What I find interesting is that Windows (at least, Windows 7
> Professional) seems to be able to handle the deficient adapter.

So does Linux.  The difference is that Windows believes the values in
the partition table in preference to what the hardware says, whereas 
Linux believes the hardware in preference to the partition table.

Thus, if the hardware says the disk contains 0.8 TB and the partition 
table says the first partition contains 2.8 TB, Windows will try to 
access all 2.8 TB but Linux will complain that the partition entry is 
invalid (because the partition extends beyond the end of the disk).

If you try to repartition the drive under Windows using the deficient 
adapter, you'll see that the problem still exists.  It just doesn't 
show up during normal use.

>  What
> I'd like to do is log the disk commands during the mounting sequence,
> preferably at both the SCSI and USB layers.  Then at least we'll know
> exactly what the driver is doing.  Are there kernel options to do
> this?

You can record the USB transfers by using usbmon (see
Documentation/usb/usbmon.txt).  The trace will include all the
important SCSI data, so you don't need to record anything at the SCSI
layer.

This isn't really necessary, though.  We already know what the driver 
is doing.

> Unfortunately, I don't know any way to log what Windows is doing with
> the drive.

My experiments with Windows show that it does essentially the same
things as Linux does.  The important difference is not what commands
and data get sent, but whether the OS pays attention to the result.

Alan Stern

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

> From: Alan Stern <stern@rowland.harvard.edu>

> If you try to repartition the drive under Windows using the deficient 
> adapter, you'll see that the problem still exists.  It just doesn't 
> show up during normal use.

So in summary, the Windows workaround is icky, but it allows any use
but repartitioning to be one on the attached disk.

Dale

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Matthew Dharm]

Is there an 'easy' way to override the detected size of a storage
device from userspace?  If we had that, someone could write a helper
application which looked for this particular fubar and try to Do The
Right Thing(tm), or at least offer the user some options.

Matt

On Fri, Aug 29, 2014 at 2:07 PM, Dale R. Worley <worley@alum.mit.edu> wrote:
>> From: Alan Stern <stern@rowland.harvard.edu>
>
>> If you try to repartition the drive under Windows using the deficient
>> adapter, you'll see that the problem still exists.  It just doesn't
>> show up during normal use.
>
> So in summary, the Windows workaround is icky, but it allows any use
> but repartitioning to be one on the attached disk.
>
> Dale
> --
> To unsubscribe from this list: send the line "unsubscribe linux-usb" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Matthew Dharm
Maintainer, USB Mass Storage driver for Linux

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Fri, 29 Aug 2014, Matthew Dharm wrote:

> Is there an 'easy' way to override the detected size of a storage
> device from userspace?  If we had that, someone could write a helper
> application which looked for this particular fubar and try to Do The
> Right Thing(tm), or at least offer the user some options.

You mean, force a Media Change event and override the capacity reported 
by the hardware?  I'm not aware of any API for doing that, although it 
probably wouldn't be too hard to add one.

How would the user know what value to put in for the capacity?  Unless 
the drive had been hooked up to a different computer and the user 
manually noted the correct capacity and typed it in, it would have to 
be guesswork.

Alan Stern

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Douglas Gilbert]

On 14-08-30 05:15 PM, Alan Stern wrote:
> On Fri, 29 Aug 2014, Matthew Dharm wrote:
>
>> Is there an 'easy' way to override the detected size of a storage
>> device from userspace?  If we had that, someone could write a helper
>> application which looked for this particular fubar and try to Do The
>> Right Thing(tm), or at least offer the user some options.
>
> You mean, force a Media Change event and override the capacity reported
> by the hardware?  I'm not aware of any API for doing that, although it
> probably wouldn't be too hard to add one.
>
> How would the user know what value to put in for the capacity?  Unless
> the drive had been hooked up to a different computer and the user
> manually noted the correct capacity and typed it in, it would have to
> be guesswork.

Might another possibility be using the SAT layer to issue
the appropriate ATA command via the SCSI ATA PASS-THROUGH
(12 or 16) command to find out the disk's size. This might
be a possible strategy if READ CAPACITY(10) yields 0xffffffff
for the last sector's LBA and the follow-up READ CAPACITY(16)
fails or yields a truncated value.

Doug Gilbert


BTW Been looking at a USB-to-SATA adapter that uses the
UAS(P) transport. I thought nothing could have worse
SCSI compliance than USB mass storage devices. I was
wrong ...

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Matthew Dharm]

On Sat, Aug 30, 2014 at 2:15 PM, Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org> wrote:
> On Fri, 29 Aug 2014, Matthew Dharm wrote:
>
>> Is there an 'easy' way to override the detected size of a storage
>> device from userspace?  If we had that, someone could write a helper
>> application which looked for this particular fubar and try to Do The
>> Right Thing(tm), or at least offer the user some options.
>
> You mean, force a Media Change event and override the capacity reported
> by the hardware?  I'm not aware of any API for doing that, although it
> probably wouldn't be too hard to add one.
>
> How would the user know what value to put in for the capacity?  Unless
> the drive had been hooked up to a different computer and the user
> manually noted the correct capacity and typed it in, it would have to
> be guesswork.

I didn't say it would be easy to figure out the right value, but at
least it would be possible.

I was thinking of something that could notice a USB device which is
formatted NTFS and has a partition table and filesystem that indicates
a much bigger capacity than what the drive reports.  Under this
circumstances, you could do something like pop-up a dialog box saying
"this drive is confused -- is it 2TB or 3TB?"

Well, maybe that would say "Drive capacity is not consistent with
partition table.  This can happen with certain USB drives designed for
use with Windows.  Override drive capacity (emulating Windows)?"

You could imagine increasing complex heuristics to try to detect this
scenario.  Even without an automated helper program to do it, if there
was a sysfs interface then when we got the periodic e-mails reporting
this same type of problem, we could offer a quick-and-clean solution.

Matt


-- 
Matthew Dharm
Maintainer, USB Mass Storage driver for Linux
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Sat, 30 Aug 2014, Douglas Gilbert wrote:

> On 14-08-30 05:15 PM, Alan Stern wrote:
> > On Fri, 29 Aug 2014, Matthew Dharm wrote:
> >
> >> Is there an 'easy' way to override the detected size of a storage
> >> device from userspace?  If we had that, someone could write a helper
> >> application which looked for this particular fubar and try to Do The
> >> Right Thing(tm), or at least offer the user some options.
> >
> > You mean, force a Media Change event and override the capacity reported
> > by the hardware?  I'm not aware of any API for doing that, although it
> > probably wouldn't be too hard to add one.
> >
> > How would the user know what value to put in for the capacity?  Unless
> > the drive had been hooked up to a different computer and the user
> > manually noted the correct capacity and typed it in, it would have to
> > be guesswork.
> 
> Might another possibility be using the SAT layer to issue
> the appropriate ATA command via the SCSI ATA PASS-THROUGH
> (12 or 16) command to find out the disk's size.

That might work.  Not all USB mass-storage devices support ATA 
pass-through but some of them do.

> This might
> be a possible strategy if READ CAPACITY(10) yields 0xffffffff
> for the last sector's LBA and the follow-up READ CAPACITY(16)
> fails or yields a truncated value.

Yes.  The problem in this case is that READ CAPACITY(10) yields a 
reasonable value (not 0xffffffff), so READ CAPACITY(16) never gets 
sent.  And if it was sent, the device wouldn't handle it anyway.

> BTW Been looking at a USB-to-SATA adapter that uses the
> UAS(P) transport. I thought nothing could have worse
> SCSI compliance than USB mass storage devices. I was
> wrong ...

Well, a USB-to-SATA adapter _is_ a USB mass-storage device.  What 
vendor and product?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

> From: Alan Stern <stern@rowland.harvard.edu>
> 
> On Fri, 29 Aug 2014, Matthew Dharm wrote:
> > Is there an 'easy' way to override the detected size of a storage
> > device from userspace?  If we had that, someone could write a helper
> > application which looked for this particular fubar and try to Do The
> > Right Thing(tm), or at least offer the user some options.
> 
> You mean, force a Media Change event and override the capacity reported 
> by the hardware?  I'm not aware of any API for doing that, although it 
> probably wouldn't be too hard to add one.
> 
> How would the user know what value to put in for the capacity?  Unless 
> the drive had been hooked up to a different computer and the user 
> manually noted the correct capacity and typed it in, it would have to 
> be guesswork.

The value would most sanely be extracted from the partition table.
(After verifying that the partition table looks correct.)  That seems
to be what Windows does, and it seems to work consistently enough for
Windows to trust that method.  Or at least, it could take the disk
size to be the end of the last partition, which would at least make
all the partitions accessible.

As somebody else hinted at, the userspace program could check the USB
device against a list of device types known to have this problem.

It could even verify that the SCSI-reported size matches the size
reported by the partition table (modulo two-to-the-whatever) (at least
for GPT tables, I don't know if MBR tables report the disk size).

Do we have any way of knowing what algorithm Windows uses in this
situation?

Dale

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Wed, 3 Sep 2014, Dale R. Worley wrote:

> > From: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
> > 
> > On Fri, 29 Aug 2014, Matthew Dharm wrote:
> > > Is there an 'easy' way to override the detected size of a storage
> > > device from userspace?  If we had that, someone could write a helper
> > > application which looked for this particular fubar and try to Do The
> > > Right Thing(tm), or at least offer the user some options.
> > 
> > You mean, force a Media Change event and override the capacity reported 
> > by the hardware?  I'm not aware of any API for doing that, although it 
> > probably wouldn't be too hard to add one.
> > 
> > How would the user know what value to put in for the capacity?  Unless 
> > the drive had been hooked up to a different computer and the user 
> > manually noted the correct capacity and typed it in, it would have to 
> > be guesswork.
> 
> The value would most sanely be extracted from the partition table.
> (After verifying that the partition table looks correct.)  That seems
> to be what Windows does, and it seems to work consistently enough for
> Windows to trust that method.  Or at least, it could take the disk
> size to be the end of the last partition, which would at least make
> all the partitions accessible.

If there is a partition table.  It might be worthwhile to try an ATA 
pass-through command as well.

> As somebody else hinted at, the userspace program could check the USB
> device against a list of device types known to have this problem.
> 
> It could even verify that the SCSI-reported size matches the size
> reported by the partition table (modulo two-to-the-whatever) (at least
> for GPT tables, I don't know if MBR tables report the disk size).

They don't.  Just the start and end of each partition.

> Do we have any way of knowing what algorithm Windows uses in this
> situation?

Ask Microsoft.  I suspect you're not likely to get an answer, though.

Anyway, I can try writing a patch to add this capability.  We'll see if 
it can solve your problem.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RES: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alfredo Dal Ava Junior]

On Sat, 30 Aug 2014, Matthew Dharm wrote 
> I was thinking of something that could notice a USB device which is formatted
> NTFS and has a partition table and filesystem that indicates a much bigger
> capacity than what the drive reports.  Under this circumstances, you could do
> something like pop-up a dialog box saying "this drive is confused -- is it 2TB or
> 3TB?"
> 
> Well, maybe that would say "Drive capacity is not consistent with partition
> table.  This can happen with certain USB drives designed for use with
> Windows.  Override drive capacity (emulating Windows)?"
> 
> You could imagine increasing complex heuristics to try to detect this scenario.
> Even without an automated helper program to do it, if there was a sysfs
> interface then when we got the periodic e-mails reporting this same type of
> problem, we could offer a quick-and-clean solution.


Hi Matt,

I did small hack to skip is_pte_valid() on efi.c  and now I have sdc1 partition listed on /proc/partition, but I hit other issue: ntfs-3g mount userspace tool that comes with Fedora now fails with "Failed to read last sector (7814037100): Invalid argument".  I also tried to override disk capacity, but SD driver fails with "Invalid command failure" and "Illegal Request" (I'll investigate it later)

Many places rely on disk capacity value, I think emulate Windows behavior for these HDD docks will not be an easy task.... 

[]'s
Alfredo

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

> From: Alan Stern <stern@rowland.harvard.edu>

> Anyway, I can try writing a patch to add this capability.  We'll see if 
> it can solve your problem.

Unfortunately, I think there is genuine value in such a hack.  E.g.,
I've got two USB-to-SATA adapters.  One works correctly.  One does
not.  But at this point, I can't attach both of my high-capacity disks
to the laptop at the same time, which makes it difficult to
bulk-transfer files from one to the other.

If, for the deficient adapter, the kernel assumed the disk size was
the end of the last partition (or the backup GPT partition table, if
that follows), then I could use it for everything other than
repartitioning.  And that would make life easier.

Thanks for trying,

Dale

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[James Bottomley]

On Wed, 2014-09-03 at 15:05 -0400, Alan Stern wrote:
> On Wed, 3 Sep 2014, Dale R. Worley wrote:
> 
> > > From: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
> > > 
> > > On Fri, 29 Aug 2014, Matthew Dharm wrote:
> > > > Is there an 'easy' way to override the detected size of a storage
> > > > device from userspace?  If we had that, someone could write a helper
> > > > application which looked for this particular fubar and try to Do The
> > > > Right Thing(tm), or at least offer the user some options.
> > > 
> > > You mean, force a Media Change event and override the capacity reported 
> > > by the hardware?  I'm not aware of any API for doing that, although it 
> > > probably wouldn't be too hard to add one.
> > > 
> > > How would the user know what value to put in for the capacity?  Unless 
> > > the drive had been hooked up to a different computer and the user 
> > > manually noted the correct capacity and typed it in, it would have to 
> > > be guesswork.
> > 
> > The value would most sanely be extracted from the partition table.
> > (After verifying that the partition table looks correct.)  That seems
> > to be what Windows does, and it seems to work consistently enough for
> > Windows to trust that method.  Or at least, it could take the disk
> > size to be the end of the last partition, which would at least make
> > all the partitions accessible.
> 
> If there is a partition table.  It might be worthwhile to try an ATA 
> pass-through command as well.
> 
> > As somebody else hinted at, the userspace program could check the USB
> > device against a list of device types known to have this problem.
> > 
> > It could even verify that the SCSI-reported size matches the size
> > reported by the partition table (modulo two-to-the-whatever) (at least
> > for GPT tables, I don't know if MBR tables report the disk size).
> 
> They don't.  Just the start and end of each partition.
> 
> > Do we have any way of knowing what algorithm Windows uses in this
> > situation?
> 
> Ask Microsoft.  I suspect you're not likely to get an answer, though.
> 
> Anyway, I can try writing a patch to add this capability.  We'll see if 
> it can solve your problem.

Before we embark on elaborate hacks, why don't we just make the capacity
writeable (by root) in sysfs from userspace (will require block change)?
We can then encode all the nasty heuristics (including gpt reading) in
userspace as a udev rule.

James



--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Wed, 3 Sep 2014, James Bottomley wrote:

> Before we embark on elaborate hacks, why don't we just make the capacity
> writeable (by root) in sysfs from userspace (will require block change)?
> We can then encode all the nasty heuristics (including gpt reading) in
> userspace as a udev rule.

That's what I'm working on.  Except that I don't know where to do it in
the block layer, so for now I'm adding the attribute to sd.c.

Where in the block layer would the right place be?  We want this to
apply only to entire devices, not to individual partitions.

Alan Stern

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[James Bottomley]

On Wed, 2014-09-03 at 16:30 -0400, Alan Stern wrote:
> On Wed, 3 Sep 2014, James Bottomley wrote:
> 
> > Before we embark on elaborate hacks, why don't we just make the capacity
> > writeable (by root) in sysfs from userspace (will require block change)?
> > We can then encode all the nasty heuristics (including gpt reading) in
> > userspace as a udev rule.
> 
> That's what I'm working on.  Except that I don't know where to do it in
> the block layer, so for now I'm adding the attribute to sd.c.
> 
> Where in the block layer would the right place be?  We want this to
> apply only to entire devices, not to individual partitions.

The bottom layer for this is part0.nr_sects which is the size attribute
you see in the block sysfs.  However, it looks like we keep a separate
value in sdkp, but we don't ever seem to use it (except to see if the
capacity has changed).

So this could be done in two ways: add a writeable capacity attribute in
sd.c, as you were originally thinking (it would call set_capacity() on
write and that would update the block layer) or make the size parameter
writeable.

This is how you would do the block layer one below, but there's no way
to inform the lower layer (so it better not have any information cached
that it makes use of).

James

---
diff --git a/block/genhd.c b/block/genhd.c
index 791f419..a114636 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -980,7 +980,7 @@ static DEVICE_ATTR(range, S_IRUGO, disk_range_show, NULL);
 static DEVICE_ATTR(ext_range, S_IRUGO, disk_ext_range_show, NULL);
 static DEVICE_ATTR(removable, S_IRUGO, disk_removable_show, NULL);
 static DEVICE_ATTR(ro, S_IRUGO, disk_ro_show, NULL);
-static DEVICE_ATTR(size, S_IRUGO, part_size_show, NULL);
+static DEVICE_ATTR(size, S_IRUGO|S_IWUSR, part_size_show, part_size_store);
 static DEVICE_ATTR(alignment_offset, S_IRUGO, disk_alignment_offset_show, NULL);
 static DEVICE_ATTR(discard_alignment, S_IRUGO, disk_discard_alignment_show,
 		   NULL);
diff --git a/block/partition-generic.c b/block/partition-generic.c
index 789cdea..d0cc418 100644
--- a/block/partition-generic.c
+++ b/block/partition-generic.c
@@ -87,6 +87,20 @@ ssize_t part_size_show(struct device *dev,
 	return sprintf(buf, "%llu\n",(unsigned long long)part_nr_sects_read(p));
 }
 
+ssize_t part_size_store(struct device *dev, struct device_attribute *attr,
+			const char *buf, size_t count)
+{
+	struct hd_struct *p = dev_to_part(dev);
+	u64 size;
+
+	if (count > 0 && sscanf(buf, "%llu", &size) > 0)
+		part_nr_sects_write(p, size);
+	else
+		return -EINVAL;
+
+	return count;
+}
+
 static ssize_t part_ro_show(struct device *dev,
 			    struct device_attribute *attr, char *buf)
 {
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index ec274e0..c9b3473 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -628,6 +628,9 @@ extern void blk_unregister_region(dev_t devt, unsigned long range);
 
 extern ssize_t part_size_show(struct device *dev,
 			      struct device_attribute *attr, char *buf);
+extern ssize_t part_size_store(struct device *dev,
+			       struct device_attribute *attr,
+			       const char *buf, size_t count);
 extern ssize_t part_stat_show(struct device *dev,
 			      struct device_attribute *attr, char *buf);
 extern ssize_t part_inflight_show(struct device *dev,


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Dale R. Worley]

> From: James Bottomley <James.Bottomley@HansenPartnership.com>

> Before we embark on elaborate hacks, why don't we just make the capacity
> writeable (by root) in sysfs from userspace (will require block change)?
> We can then encode all the nasty heuristics (including gpt reading) in
> userspace as a udev rule.

Looking in from the outside, this makes sense to me.  All the nasty
hueristics can be put in userspace -- perhaps as even a
special-purpose program, where we can directly warn the user as to
what he's getting himself into.  (I do not demand that this all work
automatically.)  And the hueristics can be improved easily, without
kernel changes.

The only gotcha that I see is that once the recorded device size is
changed, the kernel may now consider the partition table to be valid,
and should now parse it and set up the special device numbers for the
partitions.  So that needs to get triggered properly.

I suppose there is some complexity if the block-handling layer already
has blocks cached and the device size is reduced below the addresses
of the cached blocks.  But as long as the kernel doesn't crash or
write blocks in incorrect places, I don't see that as a problem -- if
you set the device size as less than the block numbers you've already
written to, that's your problem.

Dale

Re: Problem with USB-to-SATA adapters (was: AS2105-based enclosure size issues with >2TB HDDs)

[Alan Stern]

On Wed, 3 Sep 2014, James Bottomley wrote:

> On Wed, 2014-09-03 at 16:30 -0400, Alan Stern wrote:
> > On Wed, 3 Sep 2014, James Bottomley wrote:
> > 
> > > Before we embark on elaborate hacks, why don't we just make the capacity
> > > writeable (by root) in sysfs from userspace (will require block change)?
> > > We can then encode all the nasty heuristics (including gpt reading) in
> > > userspace as a udev rule.
> > 
> > That's what I'm working on.  Except that I don't know where to do it in
> > the block layer, so for now I'm adding the attribute to sd.c.
> > 
> > Where in the block layer would the right place be?  We want this to
> > apply only to entire devices, not to individual partitions.
> 
> The bottom layer for this is part0.nr_sects which is the size attribute
> you see in the block sysfs.  However, it looks like we keep a separate
> value in sdkp, but we don't ever seem to use it (except to see if the
> capacity has changed).
> 
> So this could be done in two ways: add a writeable capacity attribute in
> sd.c, as you were originally thinking (it would call set_capacity() on
> write and that would update the block layer) or make the size parameter
> writeable.

Here's my patch to the sd driver.  It seems to work -- writing to the
capacity_override attribute does change the device size.  But then you
have to force the kernel to re-read the partition table manually, for
example by running

	blockdev --rereadpt /dev/sdX

because I couldn't see any reasonable way to make this happen 
automatically.

Alan Stern



Index: usb-3.17/drivers/scsi/sd.h
===================================================================
--- usb-3.17.orig/drivers/scsi/sd.h
+++ usb-3.17/drivers/scsi/sd.h
@@ -66,6 +66,7 @@ struct scsi_disk {
 	struct gendisk	*disk;
 	atomic_t	openers;
 	sector_t	capacity;	/* size in 512-byte sectors */
+	sector_t	capacity_override;	/* in native-size blocks */
 	u32		max_xfer_blocks;
 	u32		max_ws_blocks;
 	u32		max_unmap_blocks;
Index: usb-3.17/drivers/scsi/sd.c
===================================================================
--- usb-3.17.orig/drivers/scsi/sd.c
+++ usb-3.17/drivers/scsi/sd.c
@@ -477,6 +477,38 @@ max_write_same_blocks_store(struct devic
 }
 static DEVICE_ATTR_RW(max_write_same_blocks);
 
+static ssize_t
+capacity_override_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct scsi_disk *sdkp = to_scsi_disk(dev);
+
+	return sprintf(buf, "%llu\n",
+			(unsigned long long) sdkp->capacity_override);
+}
+
+static ssize_t
+capacity_override_store(struct device *dev, struct device_attribute *attr,
+		const char *buf, size_t count)
+{
+	struct scsi_disk *sdkp = to_scsi_disk(dev);
+	unsigned long long cap;
+	int err;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EACCES;
+
+	err = kstrtoull(buf, 10, &cap);
+	if (err)
+		return err;
+
+	sdkp->capacity_override = cap;
+	revalidate_disk(sdkp->disk);
+
+	return count;
+}
+static DEVICE_ATTR_RW(capacity_override);
+
 static struct attribute *sd_disk_attrs[] = {
 	&dev_attr_cache_type.attr,
 	&dev_attr_FUA.attr,
@@ -489,6 +521,7 @@ static struct attribute *sd_disk_attrs[]
 	&dev_attr_provisioning_mode.attr,
 	&dev_attr_max_write_same_blocks.attr,
 	&dev_attr_max_medium_access_timeouts.attr,
+	&dev_attr_capacity_override.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(sd_disk);
@@ -2158,6 +2191,13 @@ sd_read_capacity(struct scsi_disk *sdkp,
 	struct scsi_device *sdp = sdkp->device;
 	sector_t old_capacity = sdkp->capacity;
 
+	/* Did the user override the reported capacity? */
+	if (!sdkp->first_scan && sdkp->capacity_override) {
+		sector_size = sdkp->device->sector_size;
+		sdkp->capacity = sdkp->capacity_override;
+		goto got_data;
+	}
+
 	if (sd_try_rc16_first(sdp)) {
 		sector_size = read_capacity_16(sdkp, sdp, buffer);
 		if (sector_size == -EOVERFLOW)