Large disk drives

Large disk drives

[Dale R. Worley]

Was there any resolution as to how large disk drives would be handled
if their interface did not support the "capacity" request that would
tell how large they were?

Or as an alternative, is there any way to avoid buying USB-SCSI
interfaces that do not support the large-capacity request?
Unfortunately, such devices work OK with Windows (since Windows trusts
what the partition table says), you can't just say to the salesperson
"It has to work on drives over 3 TB."

Dale
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[James Bottomley]

On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote:
> Was there any resolution as to how large disk drives would be handled
> if their interface did not support the "capacity" request that would
> tell how large they were?

Realistically no ... unless someone comes up with a reliable heuristic
to give us the size.

> Or as an alternative, is there any way to avoid buying USB-SCSI
> interfaces that do not support the large-capacity request?
> Unfortunately, such devices work OK with Windows (since Windows trusts
> what the partition table says), you can't just say to the salesperson
> "It has to work on drives over 3 TB."

This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity
plus 2TB, but the same won't happen for a 5TB device.  Believing the
partition table gives us a chicken and egg problem because something
still has to get the partition table on to the device.

I don't think "don't buy something that doesn't work" is a hugely
unreasonable response to this.

James

> Dale
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Large disk drives

[Alan Stern]

On Tue, 4 Nov 2014, James Bottomley wrote:

> On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote:
> > Was there any resolution as to how large disk drives would be handled
> > if their interface did not support the "capacity" request that would
> > tell how large they were?
> 
> Realistically no ... unless someone comes up with a reliable heuristic
> to give us the size.

I posted a patch to allow the user to override the reported capacity:

	http://marc.info/?l=linux-scsi&m=140993840113445&w=2

Nobody responded to it.

> > Unfortunately, such devices work OK with Windows (since Windows trusts
> > what the partition table says), you can't just say to the salesperson
> > "It has to work on drives over 3 TB."
> 
> This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity
> plus 2TB, but the same won't happen for a 5TB device.  Believing the
> partition table gives us a chicken and egg problem because something
> still has to get the partition table on to the device.
> 
> I don't think "don't buy something that doesn't work" is a hugely
> unreasonable response to this.

The problem is knowing beforehand whether it will work.  Once you buy 
the device and can test it, returning it is annoying and time-consuming 
at best.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Dale R. Worley]

Replying to two messages at once:

> Date: Tue, 4 Nov 2014 11:14:39 -0500 (EST)
> From: Alan Stern <stern-nwvwT67g6+6dFdvTe/nMLpVzexx5G7lz@public.gmane.org>
> cc: "Dale R. Worley" <worley-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org>, <linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
>         <linux-usb-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> 
> On Tue, 4 Nov 2014, James Bottomley wrote:
> 
> > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote:
> > > Was there any resolution as to how large disk drives would be handled
> > > if their interface did not support the "capacity" request that would
> > > tell how large they were?
> > 
> > Realistically no ... unless someone comes up with a reliable heuristic
> > to give us the size.

I can understand why the linux-scsi project would not want to take up
what is really a hack to work around a hardware deficiency.

> I posted a patch to allow the user to override the reported capacity:
> 
> 	http://marc.info/?l=linux-scsi&m=140993840113445&w=2
> 
> Nobody responded to it.
> 
> > > Unfortunately, such devices work OK with Windows (since Windows trusts
> > > what the partition table says), you can't just say to the salesperson
> > > "It has to work on drives over 3 TB."
> > 
> > This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity
> > plus 2TB, but the same won't happen for a 5TB device.  Believing the
> > partition table gives us a chicken and egg problem because something
> > still has to get the partition table on to the device.
> > 
> > I don't think "don't buy something that doesn't work" is a hugely
> > unreasonable response to this.
> 
> The problem is knowing beforehand whether it will work.  Once you buy 
> the device and can test it, returning it is annoying and time-consuming 
> at best.

Or as I would phrase it, How do you turn "Don't buy something that
doesn't work!" into an algorithm?  That is:  I'm standing in
MicroCenter, holding a box in my hand that contains a USB-to-SCSI
adapter.  How do I determine whether or not it supports large disks?
The problem being that it *works with Windows*, so I can't just ask
the friendly salesperson, Does this work with 3 terabyte disks?

In my case, the Diablotek device works, while the Kingwin EZ-Connect
does not, despite being labeled in essentially the same way.  (Neither
says that Linux is supported.)

I admit that my problem may be my deficiency in dealing with the
retail situation, as I'm a much better software engineer than shopper.

Dale
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[James Bottomley]

On Tue, 2014-11-04 at 11:14 -0500, Alan Stern wrote:
> On Tue, 4 Nov 2014, James Bottomley wrote:
> 
> > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote:
> > > Was there any resolution as to how large disk drives would be handled
> > > if their interface did not support the "capacity" request that would
> > > tell how large they were?
> > 
> > Realistically no ... unless someone comes up with a reliable heuristic
> > to give us the size.
> 
> I posted a patch to allow the user to override the reported capacity:
> 
> 	http://marc.info/?l=linux-scsi&m=140993840113445&w=2
> 
> Nobody responded to it.

Sorry, meant to.  In principle I'm OK with this as the lever for the
hack (largely because it means we don't need to do anything) but will
the distributions support it?

> > > Unfortunately, such devices work OK with Windows (since Windows trusts
> > > what the partition table says), you can't just say to the salesperson
> > > "It has to work on drives over 3 TB."
> > 
> > This is a stopgap: your 3TB drive can be guessed as the 16 bit capacity
> > plus 2TB, but the same won't happen for a 5TB device.  Believing the
> > partition table gives us a chicken and egg problem because something
> > still has to get the partition table on to the device.
> > 
> > I don't think "don't buy something that doesn't work" is a hugely
> > unreasonable response to this.
> 
> The problem is knowing beforehand whether it will work.  Once you buy 
> the device and can test it, returning it is annoying and time-consuming 
> at best.

OK, but I still don't understand how windows gets the partition table on
there in the first place ... that must surely be some sort of guess the
disk size hack.

James


> Alan Stern
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Large disk drives

[Alan Stern]

On Wed, 5 Nov 2014, James Bottomley wrote:

> On Tue, 2014-11-04 at 11:14 -0500, Alan Stern wrote:
> > On Tue, 4 Nov 2014, James Bottomley wrote:
> > 
> > > On Mon, 2014-11-03 at 16:06 -0500, Dale R. Worley wrote:
> > > > Was there any resolution as to how large disk drives would be handled
> > > > if their interface did not support the "capacity" request that would
> > > > tell how large they were?
> > > 
> > > Realistically no ... unless someone comes up with a reliable heuristic
> > > to give us the size.
> > 
> > I posted a patch to allow the user to override the reported capacity:
> > 
> > 	http://marc.info/?l=linux-scsi&m=140993840113445&w=2
> > 
> > Nobody responded to it.
> 
> Sorry, meant to.  In principle I'm OK with this as the lever for the
> hack (largely because it means we don't need to do anything) but will
> the distributions support it?

While I can't speak for the distributions, it's reasonable to assume 
that if something becomes part of the upstream kernel then they will 
include it.

> OK, but I still don't understand how windows gets the partition table on
> there in the first place ... that must surely be some sort of guess the
> disk size hack.

It's simpler than that: The drive is attached directly to the computer
(i.e., via SATA rather than USB) when the partition table is created.  
With no USB-SATA bridge chip to mess things up, there's no problem
determining the correct capacity.

Alan Stern

Re: Large disk drives

[Boaz Harrosh]

On 11/05/2014 06:34 PM, Alan Stern wrote:
<>
> 
> It's simpler than that: The drive is attached directly to the computer
> (i.e., via SATA rather than USB) when the partition table is created.  
> With no USB-SATA bridge chip to mess things up, there's no problem
> determining the correct capacity.
> 

Right!

I think it should be very simple to, just as we send a
READ_CAPACITY16 / READ_CAPACITY32 / READ_CAPACITY64 or is
it a GET_CODE_PAGE ? whatever we send today we can also
have READ_CAPACITY_PART() which will send a READ of
sector 0 the raw PC_COMMAND way and run it through some
partition analyzer. We only need to support gpt or msdos
partitions I think in this case.

Then if READ_CAPACITY16 returns all ffff(s) and
READ_CAPACITY32 is not supported and/or blacklisted,
then yes attempt a READ_CAPACITY_PART() which should
be sam-2 compatible.

It should not take longer than a weekend afternoon over
cup of Machha. If any one sends a bad device my way, I'll
do it. But anyone should be able to code something so simple.

> Alan Stern
> 

Cheers
Boaz

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Christoph Hellwig]

On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote:
> > Sorry, meant to.  In principle I'm OK with this as the lever for the
> > hack (largely because it means we don't need to do anything) but will
> > the distributions support it?
> 
> While I can't speak for the distributions, it's reasonable to assume 
> that if something becomes part of the upstream kernel then they will 
> include it.

Btw, we do have precedence for looking at partition tables from SCSI
code with scsi_partsize(), so the layering violation of looking at
EFI labels for disks sizes wouldn't be something entirely new even
if we did it in kernel space.

Re: Large disk drives

[James Bottomley]

On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote:
> On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote:
> > > Sorry, meant to.  In principle I'm OK with this as the lever for the
> > > hack (largely because it means we don't need to do anything) but will
> > > the distributions support it?
> > 
> > While I can't speak for the distributions, it's reasonable to assume 
> > that if something becomes part of the upstream kernel then they will 
> > include it.
> 
> Btw, we do have precedence for looking at partition tables from SCSI
> code with scsi_partsize(), so the layering violation of looking at
> EFI labels for disks sizes wouldn't be something entirely new even
> if we did it in kernel space.

We really don't want to make the decision within the kernel of whether
we believe the partition size or the disk capacity.   For these disk
problems we need it to be the former, but if we choose that always,
we'll get weird results on mispartitioned devices.

The usual rule is no policy in the kernel and which to choose is policy,
so just export the knob (as Alan's patch does) and then let userspace
decide.

James


--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Boaz Harrosh]

On 11/06/2014 12:30 PM, James Bottomley wrote:
> On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote:
>> On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote:
>>>> Sorry, meant to.  In principle I'm OK with this as the lever for the
>>>> hack (largely because it means we don't need to do anything) but will
>>>> the distributions support it?
>>>
>>> While I can't speak for the distributions, it's reasonable to assume 
>>> that if something becomes part of the upstream kernel then they will 
>>> include it.
>>
>> Btw, we do have precedence for looking at partition tables from SCSI
>> code with scsi_partsize(), so the layering violation of looking at
>> EFI labels for disks sizes wouldn't be something entirely new even
>> if we did it in kernel space.
> 
> We really don't want to make the decision within the kernel of whether
> we believe the partition size or the disk capacity.   For these disk
> problems we need it to be the former, but if we choose that always,
> we'll get weird results on mispartitioned devices.
> 
> The usual rule is no policy in the kernel and which to choose is policy,
> so just export the knob (as Alan's patch does) and then let userspace
> decide.
> 

I do not think it is a matter of policy.

>From what I understand the situation is that read_capacity_10() which is
32bit, Max 2T byte disks. fails with 0xffffffff and read_capacity_16() (64bit)
is not supported because of wrong scsi-version or because it is blacklisted.
(or device returns not-supported)

Now If sd_read_capacity() succeed then off-course we choose it. What I'm saying
is if read-capacity fails, then should we attempt a new read_capacity_part()?

And sure a user-mode knob if he wants to make policy, like you said, is fine
by me.

But just the simple case of read-capacity failure should we then?

> James

Thanks
Boaz

Re: Large disk drives

[Alan Stern]

On Thu, 6 Nov 2014, Boaz Harrosh wrote:

> On 11/06/2014 12:30 PM, James Bottomley wrote:

> > We really don't want to make the decision within the kernel of whether
> > we believe the partition size or the disk capacity.   For these disk
> > problems we need it to be the former, but if we choose that always,
> > we'll get weird results on mispartitioned devices.
> > 
> > The usual rule is no policy in the kernel and which to choose is policy,
> > so just export the knob (as Alan's patch does) and then let userspace
> > decide.

Another alternative would be to export a policy setting for whether or 
not to take the capacity from the partition table.  That would be more 
convenient for users, because then they wouldn't have to figure out the 
number of blocks in the drive.  On the other hand, it wouldn't work so 
well if there is no partition table and the user wants to create one.

> I do not think it is a matter of policy.
> 
> From what I understand the situation is that read_capacity_10() which is
> 32bit, Max 2T byte disks. fails with 0xffffffff and read_capacity_16() (64bit)
> is not supported because of wrong scsi-version or because it is blacklisted.
> (or device returns not-supported)

No, that is not the problem.  read_capacity_10() succeeds but gives an 
incorrect value.  read_capacity_16() fails.

> Now If sd_read_capacity() succeed then off-course we choose it. What I'm saying
> is if read-capacity fails, then should we attempt a new read_capacity_part()?

sd_read_capacity() succeeds, but the value it gets is wrong.

> And sure a user-mode knob if he wants to make policy, like you said, is fine
> by me.
> 
> But just the simple case of read-capacity failure should we then?

That's a separate question.  As far as I know, the case you are 
describing has not come up.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Boaz Harrosh]

On 11/06/2014 05:53 PM, Alan Stern wrote:
>> But just the simple case of read-capacity failure should we then?
> 
> That's a separate question.  As far as I know, the case you are 
> describing has not come up.
> 

BTW: what we should do is when the partition parser at the block layer
see that the partition capacity as written in the partition-table is
bigger then the capacity reported for the device we can put a fat
message at dmesg with both sizes and user can decide.
And/or add an extra field at block attributes for part_capacity. It
is info that is known and parsed for today.

> Alan Stern

Thanks
Boaz

RE: Large disk drives

[David Laight]

From: Boaz Harrosh
> On 11/06/2014 05:53 PM, Alan Stern wrote:
> >> But just the simple case of read-capacity failure should we then?
> >
> > That's a separate question.  As far as I know, the case you are
> > describing has not come up.
> >
> 
> BTW: what we should do is when the partition parser at the block layer
> see that the partition capacity as written in the partition-table is
> bigger then the capacity reported for the device we can put a fat
> message at dmesg with both sizes and user can decide.

One possibility is to try to read the last sector of the actual
partition. If it succeeds there are 2 possibilities:

1) The disk is as bit as the partition table indicates.
2) The high bit(s) of the sector number have been masked.
(or the disk locks up)

In many cases the latter can be verified by reading the other sector.
But if the media is new/erased then all the sectors will be 0xff
and you'd have to do a write to ensure there was no sector aliasing.

	David

Re: Large disk drives

[James Bottomley]

On Thu, 2014-11-06 at 17:08 +0000, David Laight wrote:
> From: Boaz Harrosh
> > On 11/06/2014 05:53 PM, Alan Stern wrote:
> > >> But just the simple case of read-capacity failure should we then?
> > >
> > > That's a separate question.  As far as I know, the case you are
> > > describing has not come up.
> > >
> > 
> > BTW: what we should do is when the partition parser at the block layer
> > see that the partition capacity as written in the partition-table is
> > bigger then the capacity reported for the device we can put a fat
> > message at dmesg with both sizes and user can decide.
> 
> One possibility is to try to read the last sector of the actual
> partition. If it succeeds there are 2 possibilities:

These are USB devices (bridges) not well behaved SCSI devices.  The last
time we tried to poke USB devices beyond their defined capacity (the USB
capacity off by one problem) the firmware on some crashed and we
eventually gave up.  That's why, unless we can find simple, functional
heuristics for the kernel, it's safer to punt to userspace.

James

> 1) The disk is as bit as the partition table indicates.
> 2) The high bit(s) of the sector number have been masked.
> (or the disk locks up)
> 
> In many cases the latter can be verified by reading the other sector.
> But if the media is new/erased then all the sectors will be 0xff
> and you'd have to do a write to ensure there was no sector aliasing.
> 
> 	David
> 
> NrybXǧv^)޺{.n+{"{ay\x1dʇڙ,j\afhz\x1ew\fj:+vwjm\azZ+ݢj"!


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[James Bottomley]

On Thu, 2014-11-06 at 16:33 +0200, Boaz Harrosh wrote:
> On 11/06/2014 12:30 PM, James Bottomley wrote:
> > On Wed, 2014-11-05 at 11:30 -0800, Christoph Hellwig wrote:
> >> On Wed, Nov 05, 2014 at 11:34:11AM -0500, Alan Stern wrote:
> >>>> Sorry, meant to.  In principle I'm OK with this as the lever for the
> >>>> hack (largely because it means we don't need to do anything) but will
> >>>> the distributions support it?
> >>>
> >>> While I can't speak for the distributions, it's reasonable to assume 
> >>> that if something becomes part of the upstream kernel then they will 
> >>> include it.
> >>
> >> Btw, we do have precedence for looking at partition tables from SCSI
> >> code with scsi_partsize(), so the layering violation of looking at
> >> EFI labels for disks sizes wouldn't be something entirely new even
> >> if we did it in kernel space.
> > 
> > We really don't want to make the decision within the kernel of whether
> > we believe the partition size or the disk capacity.   For these disk
> > problems we need it to be the former, but if we choose that always,
> > we'll get weird results on mispartitioned devices.
> > 
> > The usual rule is no policy in the kernel and which to choose is policy,
> > so just export the knob (as Alan's patch does) and then let userspace
> > decide.
> > 
> 
> I do not think it is a matter of policy.
> 
> From what I understand the situation is that read_capacity_10() which is
> 32bit, Max 2T byte disks. fails with 0xffffffff and read_capacity_16() (64bit)
> is not supported because of wrong scsi-version or because it is blacklisted.
> (or device returns not-supported)

Actually, no, RC(10) returns the lowest 32 bits of the actual result.
Which is out of spec, but hey, this is a USB bridge ...

> Now If sd_read_capacity() succeed then off-course we choose it.

Um, the problem is we can't tell ... sd_read_capacity() returns a
number, it just may be wrong by a factor of 2TB.

>  What I'm saying
> is if read-capacity fails, then should we attempt a new read_capacity_part()?

We can't tell we have a failure.  We can tell if the partition capacity
and the read capacity differ that one of them must be wrong ...

> And sure a user-mode knob if he wants to make policy, like you said, is fine
> by me.
> 
> But just the simple case of read-capacity failure should we then?

We don't have a failure.  This is the problem.  Determining that a
problem exists 

James

Re: Large disk drives

[Boaz Harrosh]

On 11/06/2014 05:54 PM, James Bottomley wrote:
> 
> We don't have a failure.  This is the problem.  Determining that a
> problem exists 
> 

OK Sorry. I assumed the bridge is smart enough to do nothing,

ie READ_CAPACITY_10 is passed as is via sata to the device that
actually supports READ_CAPACITY_16, as I understand then the 
actual good drive is not suppose to send size-modulue-2G in
response to READ_CAPACITY_10. Should it ?

Then the bridge just sends that back to me, now if I send
READ_CAPACITY_16 the bridge will return NOT-SUPPORTED because
it is unexpected.

But what are you saying that the bridge was smart enough to
do READ_CAPACITY_16 get a 64bit value from the drive and then
return the lower 32bit to me ? Really ? I would not imagine
in the life of me someone so dumb. And surly it is against
any spec.

Are you sure ? I think you are wrong I think the guy reported
that he can only see 2T out of his 3T drive which means the
bridge returned 0xffffffff, exactly 2T

> James
> 
> 

But hey, I guess stupidity is limitless, this one here pushing
real far.

Cheers
Boaz

Re: Large disk drives

[Alan Stern]

On Thu, 6 Nov 2014, Boaz Harrosh wrote:

> On 11/06/2014 05:54 PM, James Bottomley wrote:
> > 
> > We don't have a failure.  This is the problem.  Determining that a
> > problem exists 
> > 
> 
> OK Sorry. I assumed the bridge is smart enough to do nothing,
> 
> ie READ_CAPACITY_10 is passed as is via sata to the device that
> actually supports READ_CAPACITY_16, as I understand then the 
> actual good drive is not suppose to send size-modulue-2G in
> response to READ_CAPACITY_10. Should it ?

The bridge does not send SCSI commands to the drive; it sends ATA 
commands.

When the bridge receives a READ CAPACITY(10) command from the host 
computer, if the drive's actual capacity is >= 2^32 blocks then the 
bridge is supposed to send back a result buffer containing 0xffffffff.  
Not the actual capacity modulo 2^32.

> Then the bridge just sends that back to me, now if I send
> READ_CAPACITY_16 the bridge will return NOT-SUPPORTED because
> it is unexpected.

Yes.

> But what are you saying that the bridge was smart enough to
> do READ_CAPACITY_16 get a 64bit value from the drive and then
> return the lower 32bit to me ? Really ? I would not imagine
> in the life of me someone so dumb. And surly it is against
> any spec.

The bridge does not send READ CAPACITY(16) to the drive.  It sends an 
ATA command -- and it probably sends the same command for READ 
CAPACITY(10).

> Are you sure ? I think you are wrong I think the guy reported
> that he can only see 2T out of his 3T drive which means the
> bridge returned 0xffffffff, exactly 2T

We are sure.  Dale reported that the size he got was less than 2 TB, 
and it was equal to the actual capacity modulo 2^32.  You can check for 
yourself; here's the original email:

	http://marc.info/?l=linux-scsi&m=140908235510961&w=2

Alan Stern

Re: Large disk drives

[Dale R. Worley]

> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> 
> We really don't want to make the decision within the kernel of whether
> we believe the partition size or the disk capacity.   For these disk
> problems we need it to be the former, but if we choose that always,
> we'll get weird results on mispartitioned devices.
> 
> The usual rule is no policy in the kernel and which to choose is policy,
> so just export the knob (as Alan's patch does) and then let userspace
> decide.

I can see some significant advantages to setting up the hack in
user-space, above the architectural ones:

- We can get the user to confirm how big we think the disk is --
because its size is printed on the label of the disk.  And if the
partition table is unreadable in some way, the user can manually enter
the size.

- We can warn the user that the data path to his disk is deficient in
not supporting which ever SCSI request, so that he knows to upgrade
his hardware.

There is one thing that seems like it might be a problem:  We have to
ensure that the SCSI driver can read the partition tables (in the
standard locations) even if it doesn't know how big the disk is.
Which leads me to wonder what happens if one reads /dev/sdX until one
hits end-of file.  People have written that we don't want to read the
disk from locations beyond end-of-data because some disks react badly
to out-of-range reads.  But if that is so in general, there would be
problems simply copying /dev/sdX.  (Indeed, if all disks gave a proper
error for out-of-range reads, a bisection search would find the size
of the disk easily enough.)

Dale

Re: Large disk drives

[Alan Stern]

On Thu, 6 Nov 2014, Dale R. Worley wrote:

> There is one thing that seems like it might be a problem:  We have to
> ensure that the SCSI driver can read the partition tables (in the
> standard locations) even if it doesn't know how big the disk is.

A DOS partition table is stored in the first 512 bytes of the disk.  A
GPT partition table occupies the first 33 or so blocks.  (It also has a
copy occupying the last few blocks, but if you don't know how large the
disk is, that's not much help.)

> Which leads me to wonder what happens if one reads /dev/sdX until one
> hits end-of file.  People have written that we don't want to read the
> disk from locations beyond end-of-data because some disks react badly
> to out-of-range reads.  But if that is so in general, there would be
> problems simply copying /dev/sdX.  (Indeed, if all disks gave a proper
> error for out-of-range reads, a bisection search would find the size
> of the disk easily enough.)

Most drives will work fine if you try to read beyond the end.  You'll 
just get an appropriate error return.

But some devices (typically, cheap consumer-grade USB devices) go 
wacky.  Often enough, they crash or hang.  Some of them may get going 
again in response to a reset; others have to be powered off to recover.

Alan Stern

Re: Large disk drives

[Theodore Ts'o]

On Wed, Nov 05, 2014 at 05:07:48PM +0100, James Bottomley wrote:
> 
> OK, but I still don't understand how windows gets the partition table on
> there in the first place ... that must surely be some sort of guess the
> disk size hack.

99% of the time the partition table was provided by the drive
manufacturer; I would assume they have a way of doing that as a part
of their manufacturing processes that don't require an OS individually
running the equivalent of fdisk or gdisk on said drive.

		    	       	  	   - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Dale R. Worley]

> From: "Theodore Ts'o" <tytso-3s7WtUTddSA@public.gmane.org>
> 
> On Wed, Nov 05, 2014 at 05:07:48PM +0100, James Bottomley wrote:
> > 
> > OK, but I still don't understand how windows gets the partition table on
> > there in the first place ... that must surely be some sort of guess the
> > disk size hack.
> 
> 99% of the time the partition table was provided by the drive
> manufacturer; I would assume they have a way of doing that as a part
> of their manufacturing processes that don't require an OS individually
> running the equivalent of fdisk or gdisk on said drive.

I would expect it as part of the manufacturer's testing process.
After all, the disks are going to be plugged into a machine whose
whole purpose is to run the disk through its paces, and the machine
knows in advance exactly how big the disk is.

Dale
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Dale R. Worley]

> From: Alan Stern <stern@rowland.harvard.edu>

> I posted a patch to allow the user to override the reported capacity:
> 
> 	http://marc.info/?l=linux-scsi&m=140993840113445&w=2

I see the patch, and I feel confident I could install it if I needed
to.  What command do I execute to "write to the capacity_override
attribute", before I do "blockdev --rereadpt /dev/sdX"?

Dale

Re: Large disk drives

[Alan Stern]

On Wed, 5 Nov 2014, Dale R. Worley wrote:

> > From: Alan Stern <stern@rowland.harvard.edu>
> 
> > I posted a patch to allow the user to override the reported capacity:
> > 
> > 	http://marc.info/?l=linux-scsi&m=140993840113445&w=2
> 
> I see the patch, and I feel confident I could install it if I needed
> to.  What command do I execute to "write to the capacity_override
> attribute", before I do "blockdev --rereadpt /dev/sdX"?

As root:

	echo N >/sys/class/scsi_disk/D:D:D:D/capacity_override

where N is the number of blocks and D:D:D:D is the device name
corresponding to the sdX disk.  Or if you prefer, an equivalent path
would be

	/sys/block/sdX/device/scsi_disk/D:D:D:D/capacity_override

and here you wouldn't have to guess at the value of D:D:D:D because it 
would be the only entry in the /sys/block/sdX/device/scsi_disk 
directory.

Alan Stern

Re: Large disk drives

[Norman Diamond]

If READ_CAPACITY_10 returns something that looks valid but might be off
by a multiple of 2TB, and READ_CAPACITY_16 fails, what do we really want
to do when we read the partition table?
 
If the partition table indicates that everything fits in the capacity
returned by READ_CAPACITY_10, great, it isn't a large disk drive and we
can do all 10-byte commands.
 
If the partition table indicates that the disk drive really is large and
the USB bridge has problems, I'd say we'd better not touch the drive any
further (with possible exceptions).  If the USB bridge fails
READ_CAPACITY_16 but the USB bridge pretends to succeed with READ_16 and
WRITE_16, well maybe it wraps around at 2TB and writes the wrong sector
on the drive.  If READ_CAPACITY_16 fails, I wouldn't trust the USB bridge
with any 16-byte command.  If the partition table indicates that the
drive is larger, the user had better be told to find a different USB
bridge or use a real SATA or SCSI controller.
 
A possible exception is that if a USB bridge is well tested and confirmed
to handle READ_16 and WRITE_16 correctly while failing READ_CAPACITY_16,
an antiquirk could be configured to say we rely on the bridge.
 
By the way, I've seen some USB bridges that lie about whether they
performed various SAT commands (ATA passthrough), but told the truth
about performing an ATA IDENTIFY DEVICE through SAT.  So we could attempt
ATA passthrough with an IDENTIFY DEVICE command, and if it happens to
return a sane looking result, we could comparre it to the bridge's own
translation of READ_CAPACITY_10 and READ_CAPACITY_16.  If the passthrough
yielded a sane result and the capacity doesn't match, we know not to
trust the bridge with 16-byte commands (except if tested, as mentioned).
In such a case, we don't even have to look at the partition table.  Of
course if the ATA passthrough fails or if the result is garbage then
abandon this test and maybe look at the partition table.
 
By the way, it's not just USB bridges.  I have a particularly obnoxious
ExpressCard with an eSATA connection.  Where the SATA drive operates on
512-byte LBAs (whether real or emulated doesn't matter), the eSATA bridge
translates groups of 8 of them into emulated 4096-byte LBAs.  This should
make it possible for a 3TB drive to be used when the PC's drivers don't
know how to use ATA commands on sector numbers bigger than 32 bits, as
long as the PC's drivers can handle 4096-byte sectors, right?  Well, no.
This eSATA bridge's firmware wraps around at 2TB.  The vendor went to
all this effort just to screw up and maximize the amount of damage that
occurs by hiding their screw up.  I don't know if eBay's warranty covers
this kind of defect, but it wouldn't be worth paying for postage to
return this POS.  I should have bought the Brooklyn Bridge instead.
To compound the oddity, the same ExpressCard contains a working USB 3.0
bridge.
 
(Sorry I can't construct headers to get this message threaded correctly
in the mailing list.)
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Large disk drives

[David Laight]

From: Norman Diamond
...
> By the way, I've seen some USB bridges that lie about whether they
> performed various SAT commands (ATA passthrough), but told the truth
> about performing an ATA IDENTIFY DEVICE through SAT.  So we could attempt
> ATA passthrough with an IDENTIFY DEVICE command, and if it happens to
> return a sane looking result, we could comparre it to the bridge's own
> translation of READ_CAPACITY_10 and READ_CAPACITY_16.  If the passthrough
> yielded a sane result and the capacity doesn't match, we know not to
> trust the bridge with 16-byte commands (except if tested, as mentioned).
> In such a case, we don't even have to look at the partition table.  Of
> course if the ATA passthrough fails or if the result is garbage then
> abandon this test and maybe look at the partition table.
...

Don't count on devices answering ATA IDENTIFY correctly either.
I had some CF cards from one of the main 'labels' that reported
an incorrectly layed out identify response, and 2 others (almost
identical) that locked solid when the request was issued.
We tried to send them back, but they were returned with a nice
shiny new MBR table.

	David



--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Large disk drives

[Norman Diamond]

From: David Laight
> 
> From: Norman Diamond
> ...
>>  By the way, I've seen some USB bridges that lie about whether they
>>  performed various SAT commands (ATA passthrough), but told the truth
>>  about performing an ATA IDENTIFY DEVICE through SAT.  So we could attempt
>>  ATA passthrough with an IDENTIFY DEVICE command, and if it happens to
>>  return a sane looking result, we could comparre it to the bridge's own
>>  translation of READ_CAPACITY_10 and READ_CAPACITY_16.  If the passthrough
>>  yielded a sane result and the capacity doesn't match, we know not to
>>  trust the bridge with 16-byte commands (except if tested, as mentioned).
>>  In such a case, we don't even have to look at the partition table.  Of
>>  course if the ATA passthrough fails or if the result is garbage then
>>  abandon this test and maybe look at the partition table.
> ...
> 
> Don't count on devices answering ATA IDENTIFY correctly either.
 
Of course.  That's why you quoted me saying this:
 
>> Of
>>  course if the ATA passthrough fails or if the result is garbage then
>>  abandon this test and maybe look at the partition table. 
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html