Re: 64-bit kdev_t - just for playing

Re: 64-bit kdev_t - just for playing

[Badari Pulavarty]

> I'm right here campaigning loudly for a larger dev_t. I intend 
> to never, ever make assumptions about dev_t. In fact, I'd rather not 
> deal with dev_t. But I do need a way to map 4k or 8k or 16k disks. 
> now. 
>
> Joel

 Hi Joel,

I have been playing with supporting 4000 disks on IA32 machines.
There are bunch of issues we need to resolve before we could
do that.

I am using scsi_debug to simulate 4000 disks. (Ofcourse, I had
to hack "sd" to support more than 256 disks). Anyway, I noticed
that I lost almost 350MB of my lowmem, when I simulated 4000 disks.
We are working on most of these. But there are userlevel issues
to be resolved. Here is the list ...

1) deadline_drq, blkdev_request consume 80 MB of low memory.
      - Jens is looking at it. He is working on a patch to allocate
requests dynamically.

2) sysfs inode use up 50 MB of low memory
        - 4000 disks without partitions create (4000 * 35) = 140,000 inodes in 
/sysfs.  So, it uses 50 MB of lowmem. 

3) dcache is eating up 25 MB of low memory.

4) kmalloc() slabs are consuming 55 MB. We are in the process
of identifying the heavy consumers and fixing them.
 	- Jens is fixing hash table size issues with io-schedulers.
	- I have patch to allocate "hd_struct" dynamically. So, if your disks
          does not have any partitions, you don't use any more memory.    

5) glibc and utility issues - lots of stuff are broken
    (Need a new libc)
        - mknod
        - ls
        - raw command
        - etc..

I have not done any IO on these yet. When I mount all of these and do
IO on them, we might see new issues. So with all these, I will be doubtful
if we can ever reach 16k disks on IA32.

Thanks,
Badari

Re: 64-bit kdev_t - just for playing

[William Lee Irwin III]

On Mon, Mar 31, 2003 at 03:41:50PM -0800, Badari Pulavarty wrote:
> 2) sysfs inode use up 50 MB of low memory
>         - 4000 disks without partitions create (4000 * 35) = 140,000 inodes in 
> /sysfs.  So, it uses 50 MB of lowmem. 
> 3) dcache is eating up 25 MB of low memory.

These are actually the same thing (the inode pinning references are
actually held by the dentries IIRC). shaggy and I are brewing up
something to show to gregkh and mochel in the near future.


-- wli

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Mon, Mar 31, 2003 at 03:41:50PM -0800, Badari Pulavarty wrote:
> I have been playing with supporting 4000 disks on IA32 machines.
> There are bunch of issues we need to resolve before we could
> do that.

	That's the conversation I'm trying to kickstart.

> I am using scsi_debug to simulate 4000 disks. (Ofcourse, I had
> to hack "sd" to support more than 256 disks). Anyway, I noticed
> that I lost almost 350MB of my lowmem, when I simulated 4000 disks.
> We are working on most of these. But there are userlevel issues
> to be resolved. Here is the list ...

	Wow, this is cool.  Thanks for telling me about this.

> I have not done any IO on these yet. When I mount all of these and do
> IO on them, we might see new issues. So with all these, I will be doubtful
> if we can ever reach 16k disks on IA32.

	We're going to have to find a way.  IA32 is going to be around
for long enough, I think.  Easily 8k disks, as soon as the folks who are
doing 4k disks today want to multipath.

Joel

-- 

Life's Little Instruction Book #226

	"When someone hugs you, let them be the first to let go."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 31 Mar 2003, Badari Pulavarty wrote:

> I have been playing with supporting 4000 disks on IA32 machines.
> There are bunch of issues we need to resolve before we could
> do that.
> 
> I am using scsi_debug to simulate 4000 disks. (Ofcourse, I had
> to hack "sd" to support more than 256 disks).

Could you please post your changes to sd.c and anything relevant to it?
Thanks.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Badari Pulavarty]

[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]

On Wednesday 02 April 2003 04:18 am, Roman Zippel wrote:
> Hi,
>
> On Mon, 31 Mar 2003, Badari Pulavarty wrote:
> > I have been playing with supporting 4000 disks on IA32 machines.
> > There are bunch of issues we need to resolve before we could
> > do that.
> >
> > I am using scsi_debug to simulate 4000 disks. (Ofcourse, I had
> > to hack "sd" to support more than 256 disks).
>
> Could you please post your changes to sd.c and anything relevant to it?
> Thanks.
>
> bye, Roman

Roman,

Here is the patch for sd to allow more than 256 disks.
There are few issues with the patch that need to be resolved.

1) With the patch I get 16 bits for minor. Since 4 bits are used for
partition, we get 12 bits to represent disks. So each major can support
2^12 = 4096 disks. Disks 0 - 4095 are mapped to major=8, 
disks 4096 - 8191 to major = 65 and so on..

This means ..

(i) I need to create nodes in /dev/ to match new <major, minor> for 
these disks.  Currently "mknod" is broken due to glibc issues with dev_t.

(ii) We need to worry about backward compatibility. For example:
17th disk used to have <65, 0>. Now its major, minor is <8, 256>.
So /dev/ entires need to be re-created to match these, everytime
you reboot 2.4/2.5 etc. Greg KH udev might fix this for us. 

2) Do we still need 16 majors for disks ?

We could change my patch to work around major/minor assignment
issues and maintain backward compatibility. 

Thanks,
Badari




[-- Attachment #2: sd.patch --]
[-- Type: text/x-diff, Size: 1353 bytes --]

--- linux/drivers/scsi/sd.c	Thu Mar 20 15:06:00 2003
+++ linux.new/drivers/scsi/sd.c	Fri Mar 21 11:50:54 2003
@@ -56,7 +56,9 @@
  * Remaining dev_t-handling stuff
  */
 #define SD_MAJORS	16
-#define SD_DISKS	(SD_MAJORS << 4)
+#define SD_DISKS_PER_MAJOR_SHIFT	(KDEV_MINOR_BITS - 4)
+#define SD_DISKS_PER_MAJOR	(1 << SD_DISKS_PER_MAJOR_SHIFT)
+#define SD_DISKS	(SD_MAJORS << SD_DISKS_PER_MAJOR_SHIFT)
 
 /*
  * Time out in seconds for disks and Magneto-opticals (which are slower).
@@ -1328,17 +1330,23 @@ static int sd_attach(struct scsi_device 
 	sdkp->index = index;
 
 	gd->de = sdp->de;
-	gd->major = sd_major(index >> 4);
-	gd->first_minor = (index & 15) << 4;
+	gd->major = sd_major(index >> SD_DISKS_PER_MAJOR_SHIFT);
+	gd->first_minor = (index & (SD_DISKS_PER_MAJOR - 1)) << 4;
 	gd->minors = 16;
 	gd->fops = &sd_fops;
 
-	if (index >= 26) {
+	if (index < 26) {
+		sprintf(gd->disk_name, "sd%c", 'a' + index % 26);
+	} else if (index < (26*27)) {
 		sprintf(gd->disk_name, "sd%c%c",
 			'a' + index/26-1,'a' + index % 26);
 	} else {
-		sprintf(gd->disk_name, "sd%c", 'a' + index % 26);
-	}
+		const unsigned int m1 = (index/ 26 - 1) / 26 - 1;
+		const unsigned int m2 = (index / 26 - 1) % 26;
+		const unsigned int m3 = index % 26;
+		sprintf(gd->disk_name, "sd%c%c%c", 
+			'a' + m1, 'a' + m2, 'a' + m3);
+	}	
 
 	sd_init_onedisk(sdkp, gd);
 

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <200304020931.38671.pbadari@us.ibm.com>
By author:    Badari Pulavarty <pbadari@us.ibm.com>
In newsgroup: linux.dev.kernel
> 
> Roman,
> 
> Here is the patch for sd to allow more than 256 disks.
> There are few issues with the patch that need to be resolved.
> 
> 1) With the patch I get 16 bits for minor. Since 4 bits are used for
> partition, we get 12 bits to represent disks. So each major can support
> 2^12 =3D 4096 disks. Disks 0 - 4095 are mapped to major=3D8,=20
> disks 4096 - 8191 to major =3D 65 and so on..
> 
> This means ..
> 
> (i) I need to create nodes in /dev/ to match new <major, minor> for=20
> these disks.  Currently "mknod" is broken due to glibc issues with dev_t.
> 
> (ii) We need to worry about backward compatibility. For example:
> 17th disk used to have <65, 0>. Now its major, minor is <8, 256>.
> So /dev/ entires need to be re-created to match these, everytime
> you reboot 2.4/2.5 etc. Greg KH udev might fix this for us.=20
> 
> 2) Do we still need 16 majors for disks ?
> 

No, we don't.  On the other hand, we really should change to 64
partitions/disk, same as for non-SCSI disks.  16 really is too small.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[David Lang]

when you think of huge raid arrays that present themselves to the system
as a single drive even 64 partitions can be limiting.

I don't really know what an appropriate limit would be, this may be
something that has a default setting for static dev entries and they
dynamic assignments can get more

David Lang

On 2 Apr 2003, H. Peter Anvin wrote:

> Date: 2 Apr 2003 14:03:58 -0800
> From: H. Peter Anvin <hpa@zytor.com>
> To: linux-kernel@vger.kernel.org
> Subject: Re: 64-bit kdev_t - just for playing
>
> Followup to:  <200304020931.38671.pbadari@us.ibm.com>
> By author:    Badari Pulavarty <pbadari@us.ibm.com>
> In newsgroup: linux.dev.kernel
> >
> > Roman,
> >
> > Here is the patch for sd to allow more than 256 disks.
> > There are few issues with the patch that need to be resolved.
> >
> > 1) With the patch I get 16 bits for minor. Since 4 bits are used for
> > partition, we get 12 bits to represent disks. So each major can support
> > 2^12 =3D 4096 disks. Disks 0 - 4095 are mapped to major=3D8,=20
> > disks 4096 - 8191 to major =3D 65 and so on..
> >
> > This means ..
> >
> > (i) I need to create nodes in /dev/ to match new <major, minor> for=20
> > these disks.  Currently "mknod" is broken due to glibc issues with dev_t.
> >
> > (ii) We need to worry about backward compatibility. For example:
> > 17th disk used to have <65, 0>. Now its major, minor is <8, 256>.
> > So /dev/ entires need to be re-created to match these, everytime
> > you reboot 2.4/2.5 etc. Greg KH udev might fix this for us.=20
> >
> > 2) Do we still need 16 majors for disks ?
> >
>
> No, we don't.  On the other hand, we really should change to 64
> partitions/disk, same as for non-SCSI disks.  16 really is too small.
>
> 	-hpa
>
> --
> <hpa@transmeta.com> at work, <hpa@zytor.com> in private!
> "Unix gives you enough rope to shoot yourself in the foot."
> Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 64-bit kdev_t - just for playing

[Lars Marowsky-Bree]

On 2003-04-03T02:09:48,
   David Lang <david.lang@digitalinsight.com> said:

> when you think of huge raid arrays that present themselves to the system
> as a single drive even 64 partitions can be limiting.

That's what logical volume management is for. Problem solved.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Wed, 2 Apr 2003, Badari Pulavarty wrote:

> (ii) We need to worry about backward compatibility. For example:
> 17th disk used to have <65, 0>. Now its major, minor is <8, 256>.
> So /dev/ entires need to be re-created to match these, everytime
> you reboot 2.4/2.5 etc. Greg KH udev might fix this for us. 

I expected something like this. As soon as we want to maintain 
compatibility it can get quite ugly.
I would prefer to leave the dev_t range below 0x10000 unchanged for 2.6 
(this means in the kernel the minor bits stay at 8) and new dev_t entries 
a created dynamically. Work which is currently still done in sd.c (e.g. 
numbering/naming) can be incrementally moved to the generic code and 
during 2.7 it can be dropped from sd.c. This way other drivers can 
automatically benefit from this and don't have to reproduce the hacks 
done in sd.c. Also if someone wants more partitions it would be a simple 
switch in sd.c and all dev_t numbers are allocated dynamically.

I am really interested why nobody wants to do now that small deciding step 
to use dynamic dev_t numbers. Everyone agrees that we want to use dynamic 
numbers, but almost everyone wants to hold on to static dev_t numbers. 
What will be the next step after enlarging dev_t? Playing around with the 
major/minor split will not help in the long term and only creates new 
problems. On the other hand at this point it's trivial now to maintain 
compability for old device numbers and create and use dynamic numbers.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Andries Brouwer]

On Thu, Apr 03, 2003 at 02:13:29PM +0200, Roman Zippel wrote:

> I am really interested why nobody wants to do now
> that small deciding step to use dynamic dev_t numbers.

Because we first want to have the numbers before using them?

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 3 Apr 2003, Andries Brouwer wrote:

> > I am really interested why nobody wants to do now
> > that small deciding step to use dynamic dev_t numbers.
> 
> Because we first want to have the numbers before using them?

Yes, I know this mantra now and that's not the problem (or will be fixed 
shortly).
This still doesn't answer what will come next. You must have _some_ idea, 
otherwise you wouldn't add a new interface, remove other infrastructure 
and provide a patch which modifies MKDEV & co. All of this only leads us 
away from the goal of dynamic device numbers. Why?

bye, Roman

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.44.0304031548090.12110-100000@serv>
By author:    Roman Zippel <zippel@linux-m68k.org>
In newsgroup: linux.dev.kernel
> 
> Yes, I know this mantra now and that's not the problem (or will be fixed 
> shortly).
> This still doesn't answer what will come next. You must have _some_ idea, 
> otherwise you wouldn't add a new interface, remove other infrastructure 
> and provide a patch which modifies MKDEV & co. All of this only leads us 
> away from the goal of dynamic device numbers. Why?
> 

I have an idea, why don't you read the archives of this mailing list
for the past eight years and learn, once again, why dynamic numbers
are broken for nearly all applications (disks and ptys being, perhaps,
the few case where they actually work.)

This has been hashed and rehashed on this list so many times it's not
even funny.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 7 Apr 2003, H. Peter Anvin wrote:

> > This still doesn't answer what will come next. You must have _some_ idea, 
> > otherwise you wouldn't add a new interface, remove other infrastructure 
> > and provide a patch which modifies MKDEV & co. All of this only leads us 
> > away from the goal of dynamic device numbers. Why?
> > 
> 
> I have an idea, why don't you read the archives of this mailing list
> for the past eight years and learn, once again, why dynamic numbers
> are broken for nearly all applications (disks and ptys being, perhaps,
> the few case where they actually work.)
> 
> This has been hashed and rehashed on this list so many times it's not
> even funny.

I ignore the flame bait in the hopes to get a reasonable answer this time, 
if you have a personal problem with me, you are free to tell me this 
directly, but flaming will help nobody.

The dynamic number discussion I remember were not only about numbering but 
also about naming. A specific device had to be addressable with a specific
number. We currently have a fixed mapping between numbers and names (as 
documented in devices.txt). Disks are an area where this doesn't work, the 
only exception is IDE, because most machines have a builtin controller, 
so it's easy to map to a fixed number. SCSI already has to use dynamic 
numbers, because it wouldn't otherwise fit into a 16bit dev_t and with SAN 
even 64bit will not be enough. This makes SCSI very annoying, when at the 
next boot everything has a new name, because a disk was added/removed.

Consequently it's impossible that the kernel guarantees a static name/ 
number for a specific device. devfs showed that the kernel is the wrong 
place for name policies. Is see no other possibility than to number 
devices dynamically and leave it to userspace to name them or did I miss 
an important point in previous discussions?
Maybe you could also answer me, what exactly you need a 64bit dev_t for?
If anything of this has been discussed previously, I'd also be happy about 
any URLs. Thanks.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 7 Apr 2003, Roman Zippel wrote:

> Consequently it's impossible that the kernel guarantees a static name/ 
> number for a specific device. devfs showed that the kernel is the wrong 
> place for name policies. Is see no other possibility than to number 
> devices dynamically and leave it to userspace to name them or did I miss 
> an important point in previous discussions?
> Maybe you could also answer me, what exactly you need a 64bit dev_t for?

Ok, Peter refuses to give me an answer to that, this either means I'm 
unworthy an answer or he doesn't know it either.
Is there really nobody who can answer me these questions?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Kevin P. Fleming]

Roman Zippel wrote:

>>Consequently it's impossible that the kernel guarantees a static name/ 
>>number for a specific device. devfs showed that the kernel is the wrong 
>>place for name policies. Is see no other possibility than to number 
>>devices dynamically and leave it to userspace to name them or did I miss 
>>an important point in previous discussions?
>>Maybe you could also answer me, what exactly you need a 64bit dev_t for?

Well, I'm not Peter, or Andries or anyone else for that matter... but if a 
userspace tool can be built that will enforce the user's (or distro's) desired 
naming policies for completely dynamic numbered devices, I'm all for that. It 
sure seems that it would greatly simplify much code in the kernel and completely 
eliminate the need to move beyond 32-bit dev_t at all. (I mean really, who's 
ever going to have more than 4G devices on a single Linux system image?)

However, there are significant hurdles to implementing this tool:

- defining an adequately powerful file format to define the desired naming policy
- getting the tool to run _both_ in early-userspace and real-userspace 
environments, using the _same_ policy file (or copies thereof)

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 7 Apr 2003, Kevin P. Fleming wrote:

> However, there are significant hurdles to implementing this tool:
> 
> - defining an adequately powerful file format to define the desired naming policy

The kernel only exports information, which you can get from sysfs. This 
will unlikely become a requirement for 2.6, so how this done can still 
change a bit.

> - getting the tool to run _both_ in early-userspace and real-userspace 
> environments, using the _same_ policy file (or copies thereof)

Early-userspace is not that much of a problem, as it only has to map 
the device information from sysfs to a device name. The required 
configuration file can be quite simple, every device has a list of 
properties, which closely resembles sysfs (one property == one value).

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 7 Apr 2003, H. Peter Anvin wrote:

> I have an idea, why don't you read the archives of this mailing list
> for the past eight years and learn, once again, why dynamic numbers
> are broken for nearly all applications (disks and ptys being, perhaps,
> the few case where they actually work.)
> 
> This has been hashed and rehashed on this list so many times it's not
> even funny.

Ok, I checked the archives and found some interesting mails:

http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1170.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1180.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1072.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1310.html
http://www.ussg.iu.edu/hypermail/linux/kernel/0105.1/1348.html

This is from the last dev_t discussion I was able to find and my apologies 
to Linus for dragging him into this, personally I prefer a discussion 
based on arguments, but you want to feel the whip of authority. (There are 
also some juicy mails from Al, but you can look for these yourself.)

Linus argues here for dynamic numbers and I was not able to find a single 
mail, where he completely changed his mind since then. If you know 
something I don't, I'd be really happy to hear about it (actually I found 
9 (nine!) year old mails, where he argues for a more dynamic system).

In above discussion, Alan was one of the few who actually came up with
reasonable arguments, some of his concerns were:
- he needed a solution for 2.4:
  we are at 2.5 now and the kernel is mostly ready for dynamic device 
  numbers
- compatibility:
  it's trivial to preserve dev numbers below 0x10000, new drivers start 
  above this
- hardcoded ioctl knowledge:
  this is partly a compatibility problem and in the meantime it's 
  generally accepted that they are a bad idea most of the time and
  drivers should use e.g. sysfs instead.

Am I now worthy of an answer, so you could please explain "why dynamic 
numbers are broken for nearly all applications"? What were I supposed to 
learn from the archives? Maybe you should read them yourself, because 
I didn't found a single discussion with a clear outcome.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Werner Almesberger]

Roman Zippel wrote:
> Date:   Mon, 7 Apr 2003 22:10:11 +0200 (CEST)
[...]
> Date: 	Mon, 7 Apr 2003 23:57:56 +0200 (CEST)
[...]
> Ok, Peter refuses to give me an answer to that,

That was a quick conclusion, after less than two hours :-)

Anyway, I agree with your general concern. It only seems good
engineering practice to also look at the numbering schemes that
are supposed to go with the device number enlargement.

Or, alternatively, to make sure that it's trivial to make further
enlargements (or shrinkages), if the need for them should arise.
I didn't look at the issue in detail, but perhaps the latter is
the case ?

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: 64-bit kdev_t - just for playing

[Andrew Morton]

"H. Peter Anvin" <hpa@zytor.com> wrote:
>
> dev_t is already 64 bits in glibc, and the glibc<->kernel interface
> needs to be fixed *anyway*.  We have to take the pain of migration, we
> might as well go all the way.

If I understand the words of Andries and Linus correctly, that is the plan.

Re: 64-bit kdev_t - just for playing

[David Lang]

the biggest problem I see with dynamic numbers is that it needs a
userspace devfs type solution for creating and maintaining the device
nodes that are then used. While this isn't rocket science it's also
somthing that is hard to get people to agree to (remember the devfs names
that everyone gripes about are not what richard started with it's what he
switched to to get things into the kernel, they changed many times during
that process)

I don't think many people will argue that dynamic assignments are evil,
but I think you will find a lot of people very nervous about switching to
them and the risk involved with doing so.

David Lang


 On Tue, 8 Apr 2003, Werner Almesberger wrote:

> Date: Tue, 8 Apr 2003 19:53:05 -0300
> From: Werner Almesberger <wa@almesberger.net>
> To: Roman Zippel <zippel@linux-m68k.org>
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 64-bit kdev_t - just for playing
>
> Roman Zippel wrote:
> > Date:   Mon, 7 Apr 2003 22:10:11 +0200 (CEST)
> [...]
> > Date: 	Mon, 7 Apr 2003 23:57:56 +0200 (CEST)
> [...]
> > Ok, Peter refuses to give me an answer to that,
>
> That was a quick conclusion, after less than two hours :-)
>
> Anyway, I agree with your general concern. It only seems good
> engineering practice to also look at the numbering schemes that
> are supposed to go with the device number enlargement.
>
> Or, alternatively, to make sure that it's trivial to make further
> enlargements (or shrinkages), if the need for them should arise.
> I didn't look at the issue in detail, but perhaps the latter is
> the case ?
>
> - Werner
>
> --
>   _________________________________________________________________________
>  / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
> /_http://www.almesberger.net/____________________________________________/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 64-bit kdev_t - just for playing

[Werner Almesberger]

David Lang wrote:
> the biggest problem I see with dynamic numbers is that it needs a
> userspace devfs type solution for creating and maintaining the device
> nodes that are then used.

Agreed, this is the difficult part. This probably means that the
kernel will have to come with a default initrd-like setup that is
built and attached by "make bzImage" and the like. I thought that
people were quite actively working towards something like this ?

> somthing that is hard to get people to agree to (remember the devfs names
> that everyone gripes about are not what richard started with it's what he
> switched to to get things into the kernel, they changed many times during
> that process)

Actually, here we may have an advantage now. The "old way" was to
write the driver, allocate a number, release the driver, and
suggest a device name. If somebody didn't like the name, they
could easily change it to something else. Nobody had to agree
with the author.

devfs was different. The one thing everybody has to agree on is
the kernel. And gratuitous differences in "vendor kernels" are
frowned upon, and are no less of a pain in individual kernels
either. So now everyone had to accept the name the author
assigned - or fight with the author until the name changed.

(Of course, one could have mounted devfs in some dark corner and
set up a link tree pointing to it, which wouldn't have been too
different from using sysfs. devfs was never really positioned to
take such a role, and seems to carry too much overhead trying to
be a "full" /dev replacement to be justifiable as an "almost"
/dev replacement.)

By completely removing such policy from the kernel, we return to
the status quo ante: user-visible names only need to be agreed on
during early system bringup. After that, the "device file
manager" takes over, and that one can just use a local device
name database.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.44.0304081607060.5766-100000@dlang.diginsite.com>
By author:    David Lang <david.lang@digitalinsight.com>
In newsgroup: linux.dev.kernel
>
> the biggest problem I see with dynamic numbers is that it needs a
> userspace devfs type solution for creating and maintaining the device
> nodes that are then used. While this isn't rocket science it's also
> somthing that is hard to get people to agree to (remember the devfs names
> that everyone gripes about are not what richard started with it's what he
> switched to to get things into the kernel, they changed many times during
> that process)
> 
> I don't think many people will argue that dynamic assignments are evil,
> but I think you will find a lot of people very nervous about switching to
> them and the risk involved with doing so.
> 

They aren't evil, they're just very, very hard to get right.

So far, *none* of the schemes used for dynamics have gotten it right.
They just ignore a fair number of the problems.  People keep focusing
on disks, and they are nearly uniformly the almost-trivial case in
comparison with especially character devices, where you don't have the
layer of indirection called /etc/fstab, persistent labels, etc.

It is also independent of the need to switch to a larger dev_t.
Claiming that we can squeeze more out of the existing device scheme if
we have an ideal-world dynamic scheme is unrealistic because:

a) There are, genuinely, systems with more than 65,536 devices or
anonymous mounts.  That rules out the current dev_t just by itself.

b) Despite the fact that people have tried since the mid-90's, we
still don't have a sane way to manage such dynamicity.

c)  We are now in what pretty much amounts to a crisis situation.  We
have needed to enlarge dev_t for well over half a decade.  Therefore,
it is too late to say "well, given X we wouldn't need it."  We need
something done in *this* kernel cycle.

Given that it has taken, literally, 8 years to get to this point, and
based on collective global experience with numberspaces, I'm arguing
for enlarging it far more than anyone can currently imagine being
necessary.

dev_t is already 64 bits in glibc, and the glibc<->kernel interface
needs to be fixed *anyway*.  We have to take the pain of migration, we
might as well go all the way.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[Kevin P. Fleming]

Werner Almesberger wrote:
> Agreed, this is the difficult part. This probably means that the
> kernel will have to come with a default initrd-like setup that is
> built and attached by "make bzImage" and the like. I thought that
> people were quite actively working towards something like this ?
> 

Yes, this is being worked on actively. This will use the 2.5.x initramfs 
infrastructure, and when it's up and going there will be early-userspace tools 
included in the kernel tarball to do the basic things that need doing 
(essentially responding to hotplug events and creating/removing devices nodes as 
needed).

> By completely removing such policy from the kernel, we return to
> the status quo ante: user-visible names only need to be agreed on
> during early system bringup. After that, the "device file
> manager" takes over, and that one can just use a local device
> name database.

Yes, this is true. Ideally, the early-userspace version and the normal-userspace 
version will be the same tool (possibly compiled differently, but otherwise 
identical). The early-userspace version should not have to create many nodes for 
a "normal" kernel boot, so most of the remaining ones can be created once the 
user's preferred naming policy can be located and used. Really, the 
early-userspace version only needs to create two nodes that I can think of to 
satisfy the "normal" kernel boot process: /dev/console and /dev/root. However, 
both of these need to be created with knowledge of parameters passed on the 
kernel command line (as the kernel does now), so that when the hotplug process 
sees the appropriate device "appear" it can create the proper device node.

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Tue, 8 Apr 2003, Werner Almesberger wrote:

> Roman Zippel wrote:
> > Date:   Mon, 7 Apr 2003 22:10:11 +0200 (CEST)
> [...]
> > Date: 	Mon, 7 Apr 2003 23:57:56 +0200 (CEST)
> [...]
> > Ok, Peter refuses to give me an answer to that,
> 
> That was a quick conclusion, after less than two hours :-)

I should have inserted a "privately" there, because there a few mails and 
even a small irc conversation inbetween.

> Anyway, I agree with your general concern. It only seems good
> engineering practice to also look at the numbering schemes that
> are supposed to go with the device number enlargement.
> 
> Or, alternatively, to make sure that it's trivial to make further
> enlargements (or shrinkages), if the need for them should arise.
> I didn't look at the issue in detail, but perhaps the latter is
> the case ?

It was really a mistake from Peter to point me to the archives, it's 
really amazing to how much crap from the static device number crew had to 
be objected by Linus and Al.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 8 Apr 2003, H. Peter Anvin wrote:

> So far, *none* of the schemes used for dynamics have gotten it right.
> They just ignore a fair number of the problems.  People keep focusing
> on disks, and they are nearly uniformly the almost-trivial case in
> comparison with especially character devices, where you don't have the
> layer of indirection called /etc/fstab, persistent labels, etc.

http://marc.theaimsgroup.com/?l=linux-kernel&m=95547434315472&w=2

> a) There are, genuinely, systems with more than 65,536 devices or
> anonymous mounts.  That rules out the current dev_t just by itself.

Absolutely nobody denies that we need a larger dev_t. It's really a poor 
argumentation that you have to come up with this.

> b) Despite the fact that people have tried since the mid-90's, we
> still don't have a sane way to manage such dynamicity.

Maybe you didn't notice, that only now the block device layer is clean 
enough to go dynamic. Maybe you didn't notice that scsi devices are 
already dynamically numbered and that there are already user space tools 
to translate them to constant device names.

> Given that it has taken, literally, 8 years to get to this point, and
> based on collective global experience with numberspaces, I'm arguing
> for enlarging it far more than anyone can currently imagine being
> necessary.

And since more than 8 years Linus is trying to tell you that static 
device numbers don't work. When do you start to listen?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Wed, Apr 09, 2003 at 02:40:24AM +0200, Roman Zippel wrote:
> > a) There are, genuinely, systems with more than 65,536 devices or
> > anonymous mounts.  That rules out the current dev_t just by itself.
> 
> Absolutely nobody denies that we need a larger dev_t. It's really a poor 
> argumentation that you have to come up with this.

Roman,
	There are a couple things being discussed here.  One is the size
of dev_t.  The other is dynamic numbers.  It would seem that most folks
agree with a larger dev_t and a more dynamic numbering system.  Let's
assume we want both for now (folks who don't, please keep out for a
second).  There are three courses of action that seem to be advocated.

1) Ship 2.6 with 16bit dev_t, work on a larger dev_t and perfect dynamic
   devices in 2.7.
2) Ship 2.6 with a (32|64)bit dev_t, work on a perfect dynamic scheme in
   2.7.
3) Hold 2.6 until it can ship with (32|64)bit dev_t and perfect dynamic
   devices.

	Many folks, Peter and myself included, are claiming that choice
(1) is absolutely untenable.  We need more device space today, not in 3
years when 2.7 becomes 2.8.
	If I understand you correctly (and here is why I mailed), you
feel that choice (2) is the worst of the choices.  You feel that we
should either choose course (1) or course (3).  I'm not sure which of
those you prefer.
	
Joel


-- 

Life's Little Instruction Book #335

	"Every so often, push your luck."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Tue, 8 Apr 2003, Joel Becker wrote:

> 1) Ship 2.6 with 16bit dev_t, work on a larger dev_t and perfect dynamic
>    devices in 2.7.
> 2) Ship 2.6 with a (32|64)bit dev_t, work on a perfect dynamic scheme in
>    2.7.
> 3) Hold 2.6 until it can ship with (32|64)bit dev_t and perfect dynamic
>    devices.
> 
> 	Many folks, Peter and myself included, are claiming that choice
> (1) is absolutely untenable.  We need more device space today, not in 3
> years when 2.7 becomes 2.8.
> 	If I understand you correctly (and here is why I mailed), you
> feel that choice (2) is the worst of the choices.  You feel that we
> should either choose course (1) or course (3).  I'm not sure which of
> those you prefer.

That misunderstanding is hopefully easy to resolve:

(4) Ship 2.6 with a (32|64)bit dev_t with an experimental dynamic scheme 
and keep the device numbers below 0x10000 as they are now.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 8 Apr 2003, H. Peter Anvin wrote:

> So far, *none* of the schemes used for dynamics have gotten it right.
> They just ignore a fair number of the problems.  People keep focusing
> on disks, and they are nearly uniformly the almost-trivial case in
> comparison with especially character devices, where you don't have the
> layer of indirection called /etc/fstab, persistent labels, etc.
> 
> [...]
> 
> Given that it has taken, literally, 8 years to get to this point, and
> based on collective global experience with numberspaces, I'm arguing
> for enlarging it far more than anyone can currently imagine being
> necessary.

Let's try it with a real example:
I have two onboard SCSI channels, the first one is for external devices 
and the second for internal devices. This means the internal drives are 
renumbered depending on whether I have my MO-drive turned on or off. Now I 
also want to access the MO-drive as normal user, but the device name 
can be /dev/sda or /dev/sdd.
So what options do I have here?
1. devfs: it would work in this situation, but it doesn't offer really 
stable device names.
2. scsidev: it's more flexible, but it is limited to scsi devices.

Another example:
Let's take a legacy free pc, where a serial port is only available via a 
usb port replicator. Is there really a good reason, why that port isn't 
available via /dev/ttyS0? Why has the user to find out, how that thing is 
called? Or why should a terminal program maintain a growing list of 
possible serial device names?

The information exported via sysfs offers now for the first time a more 
general mechanism to manage these problems. I'm trying very hard here to 
get an answer to the question "How are we going to manage a lot of 
devices?", instead I only get answers to the question "How do we get a lot 
of devices?". If we concentrate only on the latter, we neglect a major 
part of the problem.

I have to come back to the two questions I already asked earlier:
1. How do we want to manage devices in the future?
2. What compromises can we make for 2.6?

The first question requires a look at how we want to register devices, how 
is device information exported to the user and how the user can access the 
devices.
The second question requires a look at what we already have and what is 
still missing.

I think there is a general agreement that we need to manage more 
dynamically, so what do have already? Block devices are already the 
furthest and also need a larger dev_t the most. A lot of drivers which 
only use add_disk/register_blkdev (but not blk_register_region) care about 
the device number only during registration and don't need it otherwise. 
scsi already exports a lot of information via sysfs.
So if we just look at scsi we already have everything to manage devices 
more dynamically, userspace only has to make use of that information now. 
To manage now a lot of scsi devices, we only have to generate new device 
numbers for them, preferably in a way that a) other drivers can benefit 
from it and b) it doesn't break existing setups. The required kernel 
changes are minimal and certainly no reason to hold up any kernel release, 
but simply changing the kernel major/minor split breaks compatibility and 
should be avoided.

Character devices still need more work and that is mostly 2.7 material, 
but basically it's possible to give more control to user space similiar to 
block devices.

The only questions I still can't answer myself are:
1. What exactly do we need a 64bit dev_t for and what has "collective 
global experience with numberspaces" to do with the local device 
numberspace?
2. What advantage does a more static scheme have and what are the problems 
ignored by the dynamic schemes?
Above examples should help to demonstrate how to deal with the various 
problems.

Peter and Andries, I would really appreciate it, if you would stop 
ignoring me and start answering my questions or explain what is wrong in 
above explanation.
If anyone else can answer these question, you are of course welcome to do 
so, otherwise I hope you have more luck than me to get some answers here.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Pavel Machek]

Hi!

> so it's easy to map to a fixed number. SCSI already has to use dynamic 
> numbers, because it wouldn't otherwise fit into a 16bit dev_t and with SAN 
> even 64bit will not be enough. This makes SCSI very annoying, when at the 
> next boot everything has a new name, because a disk was added/removed.

Ugh? 64bits not enough? I don't think so.
You could do 20/44 bits split for example...
Or allocate 60000 majors just for disks.
But I do not think you have 2^32 disks
in any SAN.

-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 10 Apr 2003, Joel Becker wrote:

> 	No, the current patch DOES NOT BREAK compatibility.  Device
> numbers from 00:00 to FF:FF are not munged in any way.  As long as
> regular drivers don't expect anything else (eg, a new scsi driver that
> uses a new larger major), nothing is broken or changed.

Device number mapping _is_ changed, unless the driver is very careful. 
Look at the latest patch from Badari, now try to add more partition while 
maintaining compatibility and this has to be done for every driver!

> If you had read the patch, you would have seen this.

I did, but it's only half the story, how it will be used matters.

> > Producing a patch isn't that difficult, but I'd rather be interested, if 
> > there is even interest in such a patch? I already got not a single comment 
> > about the last patch.
> 
> 	Propose a dynamic system.  Show us your code.

Again, scsi _is_ already dynamic, there is not much to propose, it just 
needs to be generalized and I did already show some code.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Fri, Apr 11, 2003 at 01:53:07AM +0200, Roman Zippel wrote:
> So we do break compatibility! Wow, it really took quite some time until 
> someone confirmed this...

	No, the current patch DOES NOT BREAK compatibility.  Device
numbers from 00:00 to FF:FF are not munged in any way.  As long as
regular drivers don't expect anything else (eg, a new scsi driver that
uses a new larger major), nothing is broken or changed.

> The current numbering matches the naming, 0x0301 is /dev/hda1, 0x0802 is 
> /dev/sda2 and 0x4103 is /dev/sdq3, the proposed patch does change the 
> numbering _and_ naming policy.

	The proposed patch changes none of it.  Folks have proposed
other changes based upon it, but Andreas' patch as it stands (at least,
last time I looked at it) does not break this.

> This may be true in the kernel, but you move all the compatibility 
> problems into user space and so we just added another ugly emulation 
> layer.

	Naming is entirely a userspace problem.  It is also an extremely
hard problem.  Consider that the UK folks will want /dev/disc and the US
folks will want /dev/disk.  That's before we even approach persistent
naming.

> Oh, I do listen, you can be sure of that, but what am I supposed to so, 
> when it even takes weeks to get a clear answer about something simple as 
> compatibility? When I ask questions, I get evading answers, when I say 
> it's broken, I get silence, what else can I do?

	I will repeat:  The patch does not break compatibility, and any
discussion of new device spaces based on the larger number have been
theoretical.  If you had read the patch, you would have seen this.

> Producing a patch isn't that difficult, but I'd rather be interested, if 
> there is even interest in such a patch? I already got not a single comment 
> about the last patch.

	Propose a dynamic system.  Show us your code.  Until you do,
do not clutter the simple task of "expand the size of dev_t" with the
orthogonal task of "how do we use this new dev_t".

Joel

-- 

"You cannot bring about prosperity by discouraging thrift. You cannot
 strengthen the weak by weakening the strong. You cannot help the wage
 earner by pulling down the wage payer. You cannot further the
 brotherhood of man by encouraging class hatred. You cannot help the
 poor by destroying the rich. You cannot build character and courage by
 taking away a man's initiative and independence. You cannot help men
 permanently by doing for them what they could and should do for themselves."
	- Abraham Lincoln 


Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 10 Apr 2003, David Lang wrote:

> nobody has claimed that MAKEDEV and similar wouldn't need to be changed,
> just that changing them is not a major deal, and is far less work then
> implementing dynamic numbering.

Dynamic numbering is simple (scsi does it already), the big problem is to 
manage a large number of devices in a scalable way and keeping as much as 
possible dynamic actually helps here.

bye, Roman

Re: 64-bit kdev_t - just for playing

[David Lang]

but the compatability layer can be as simple as having the new kernel
recognise the old major/minor numbers and allocate the 'new' cleaned up
majors in the range that the old numbers don't use.

or it could be that the new major number gets stored in a different spot
on disk and the existing major/minor numbers become the new minor number
with a major number of 0

either way it doesn't break compatability until people choose to use the
new capibilities that they couldn't use otherwise, which is a reasonable
thing to do.

nobody has claimed that MAKEDEV and similar wouldn't need to be changed,
just that changing them is not a major deal, and is far less work then
implementing dynamic numbering.

David Lang

 On Fri, 11 Apr 2003,
Roman Zippel wrote:

> Date: Fri, 11 Apr 2003 01:53:07 +0200 (CEST)
> From: Roman Zippel <zippel@linux-m68k.org>
> To: James Bottomley <James.Bottomley@steeleye.com>
> Cc: Linux Kernel <linux-kernel@vger.kernel.org>
> Subject: Re: 64-bit kdev_t - just for playing
>
> Hi,
>
> On 10 Apr 2003, James Bottomley wrote:
>
> > > If you want to use only a single SCSI major, won't it change the device
> > > numbers?
> >
> > They will collapse down to a single major, yes.  No application should
> > be relying on knowing the device number, though (and if you are trying
> > to argue that they do, you must see that in that scenario dynamic
> > mapping is a total failure).  Once /dev is rebuilt, no-one will know.
> > Depending on whether people still dual boot 2.4 and 2.5, someone may
> > also come up with a compatibility layer, but one is not required.
>
> So we do break compatibility! Wow, it really took quite some time until
> someone confirmed this...
> The problem now is that unfortunately programs do know about the device
> number (e.g. MAKEDEV, scsidev), these programs will be broken and have to
> be "fixed" so that they now have to change their behaviour depending on
> the kernel version.
> This change will make it even more difficult for people to try a new
> kernel. I would say there are more people which use more than 16 disks
> than people who need more than 128 disks, but the former have to pay now
> for the latter?
> OTOH it is possible to keep all device numbers below 0x10000 constant and
> start allocating dynamic device numbers from above 0x10000. This won't
> change anything for the current users and only people who want to go
> beyond the current limits have to update their tools (what they have to do
> in either case).
>
> > > Changes in these area require kernel policy, exactly like dynamic device
> > > numbers. If you want to maintain compatibility, you likely require user
> > > space support, exactly like dynamic device numbers. So why don't we even
> > > consider dynamic device numbers, especially as they have the promise of
> > > the simpler long term solution. The required kernel changes are certainly
> > > not more complex than the patches Andries got merged and still wants to
> > > get merged. If you actually look at the character device patches I posted,
> > > they already allow simple cleanups for 2.6.
> >
> > Ah, OK, this may be the misunderstanding.  Kernel dev_t can be expanded
> > without any change to the current naming policy.  The only change is the
> > numbering in /dev, which is simple.
>
> The current numbering matches the naming, 0x0301 is /dev/hda1, 0x0802 is
> /dev/sda2 and 0x4103 is /dev/sdq3, the proposed patch does change the
> numbering _and_ naming policy.
>
> > > OTOH if you only need a
> > > solution for scsi now, it should be rather simple to expand scsidev.
> > > If you can live with dynamic device names (what scsi currently has), it's
> > > a simple shell script to generate the device nodes.
> >
> > But, as I've said before, not simpler than expanding kernel dev_t.
>
> This may be true in the kernel, but you move all the compatibility
> problems into user space and so we just added another ugly emulation
> layer.
>
> > > If you want to have thousands of disk, you need some kind of user support
> > > anyway, but I get surprisingly little answers, about how people want to
> > > actually manage that much devices? All I hear is "we want them and we
> > > want them now!", this makes it difficult to actually answer the question,
> > > how kernel should support this. In the kernel it's simpler to keep this
> > > dynamic, scsi does that already anyway. How will user space find these
> > > devices? Will it continue to scan thousands of nodes in /dev or will it
> > > just use the information already easily available via sysfs?
> >
> > :-)
> >
> > You take me back a decade to the "thirty two bits is enough for"
> > everybody and "why would an application want more than 640k of memory"
> >
> > Operating systems which impose arbitrary limits on their users, or
> > demand cast iron justification before adding features tend to be at a
> > competitive disadvantage.
>
> Splitting the device number into major and minor is such an arbitrary
> limit as well. If user space wants this it can do so, but it makes no
> sense in the kernel.
>
> > However, if you want an example of how it works today, consider
> > clustering tools:  They have to identify the same piece of shared
> > storage as it gets passed around a cluster.  The way we do it (for a few
> > hundred discs) is to get the SCSI WWN from all the discs (via sg, then
> > map sg<->sd).  Our application policy is to use the WWN to identify the
> > device and a number for the partition.  Other cluster tools use the
> > volume label, still others use the LVM UUID.
> >
> > The point is that none of these applications cares whether the kernel
> > uses dynamic or static, all they need to do is be able to identify all
> > the devices in the system.
>
> You cannot really compare the device number with other disk ids, because
> the device number is a purely local and dynamic number, while the other
> ids are global and static.
>
> > > I know I'm asking a lot of annoying questions,
> >
> > I wouldn't characterise the questions like that.  Not listening to the
> > answers now...
>
> Oh, I do listen, you can be sure of that, but what am I supposed to so,
> when it even takes weeks to get a clear answer about something simple as
> compatibility? When I ask questions, I get evading answers, when I say
> it's broken, I get silence, what else can I do?
>
> > As far as SCSI is concerned, the complete solution has already been
> > presented for the expansion of kernel dev_t, no such complete solution
> > has been presented for dynamic devices.  That constitutes a real
> > advantage to dev_t expansion in my book.
>
> Producing a patch isn't that difficult, but I'd rather be interested, if
> there is even interest in such a patch? I already got not a single comment
> about the last patch.
>
> bye, Roman
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 10 Apr 2003, James Bottomley wrote:

> > If you want to use only a single SCSI major, won't it change the device 
> > numbers?
> 
> They will collapse down to a single major, yes.  No application should
> be relying on knowing the device number, though (and if you are trying
> to argue that they do, you must see that in that scenario dynamic
> mapping is a total failure).  Once /dev is rebuilt, no-one will know. 
> Depending on whether people still dual boot 2.4 and 2.5, someone may
> also come up with a compatibility layer, but one is not required.

So we do break compatibility! Wow, it really took quite some time until 
someone confirmed this...
The problem now is that unfortunately programs do know about the device 
number (e.g. MAKEDEV, scsidev), these programs will be broken and have to 
be "fixed" so that they now have to change their behaviour depending on 
the kernel version.
This change will make it even more difficult for people to try a new 
kernel. I would say there are more people which use more than 16 disks 
than people who need more than 128 disks, but the former have to pay now 
for the latter?
OTOH it is possible to keep all device numbers below 0x10000 constant and 
start allocating dynamic device numbers from above 0x10000. This won't 
change anything for the current users and only people who want to go 
beyond the current limits have to update their tools (what they have to do 
in either case).

> > Changes in these area require kernel policy, exactly like dynamic device 
> > numbers. If you want to maintain compatibility, you likely require user 
> > space support, exactly like dynamic device numbers. So why don't we even 
> > consider dynamic device numbers, especially as they have the promise of 
> > the simpler long term solution. The required kernel changes are certainly 
> > not more complex than the patches Andries got merged and still wants to 
> > get merged. If you actually look at the character device patches I posted, 
> > they already allow simple cleanups for 2.6.
> 
> Ah, OK, this may be the misunderstanding.  Kernel dev_t can be expanded
> without any change to the current naming policy.  The only change is the
> numbering in /dev, which is simple.

The current numbering matches the naming, 0x0301 is /dev/hda1, 0x0802 is 
/dev/sda2 and 0x4103 is /dev/sdq3, the proposed patch does change the 
numbering _and_ naming policy.

> > OTOH if you only need a 
> > solution for scsi now, it should be rather simple to expand scsidev.
> > If you can live with dynamic device names (what scsi currently has), it's 
> > a simple shell script to generate the device nodes.
> 
> But, as I've said before, not simpler than expanding kernel dev_t.

This may be true in the kernel, but you move all the compatibility 
problems into user space and so we just added another ugly emulation 
layer.

> > If you want to have thousands of disk, you need some kind of user support 
> > anyway, but I get surprisingly little answers, about how people want to 
> > actually manage that much devices? All I hear is "we want them and we 
> > want them now!", this makes it difficult to actually answer the question, 
> > how kernel should support this. In the kernel it's simpler to keep this 
> > dynamic, scsi does that already anyway. How will user space find these 
> > devices? Will it continue to scan thousands of nodes in /dev or will it 
> > just use the information already easily available via sysfs?
> 
> :-)
> 
> You take me back a decade to the "thirty two bits is enough for"
> everybody and "why would an application want more than 640k of memory"
> 
> Operating systems which impose arbitrary limits on their users, or
> demand cast iron justification before adding features tend to be at a
> competitive disadvantage.

Splitting the device number into major and minor is such an arbitrary 
limit as well. If user space wants this it can do so, but it makes no 
sense in the kernel.

> However, if you want an example of how it works today, consider
> clustering tools:  They have to identify the same piece of shared
> storage as it gets passed around a cluster.  The way we do it (for a few
> hundred discs) is to get the SCSI WWN from all the discs (via sg, then
> map sg<->sd).  Our application policy is to use the WWN to identify the
> device and a number for the partition.  Other cluster tools use the
> volume label, still others use the LVM UUID.
> 
> The point is that none of these applications cares whether the kernel
> uses dynamic or static, all they need to do is be able to identify all
> the devices in the system.

You cannot really compare the device number with other disk ids, because 
the device number is a purely local and dynamic number, while the other 
ids are global and static.

> > I know I'm asking a lot of annoying questions,
> 
> I wouldn't characterise the questions like that.  Not listening to the
> answers now...

Oh, I do listen, you can be sure of that, but what am I supposed to so, 
when it even takes weeks to get a clear answer about something simple as 
compatibility? When I ask questions, I get evading answers, when I say 
it's broken, I get silence, what else can I do?

> As far as SCSI is concerned, the complete solution has already been
> presented for the expansion of kernel dev_t, no such complete solution
> has been presented for dynamic devices.  That constitutes a real
> advantage to dev_t expansion in my book.

Producing a patch isn't that difficult, but I'd rather be interested, if 
there is even interest in such a patch? I already got not a single comment 
about the last patch.

bye, Roman

Re: 64-bit kdev_t - just for playing

[James Bottomley]

On Thu, 2003-04-10 at 07:47, Roman Zippel wrote:
> If you want to use only a single SCSI major, won't it change the device 
> numbers?

They will collapse down to a single major, yes.  No application should
be relying on knowing the device number, though (and if you are trying
to argue that they do, you must see that in that scenario dynamic
mapping is a total failure).  Once /dev is rebuilt, no-one will know. 
Depending on whether people still dual boot 2.4 and 2.5, someone may
also come up with a compatibility layer, but one is not required.

> What will happen to setups, which already use more than 16 disks 
> (but are still way below the 128 disk limit)?
> Do you want to go to 64 partitions during 2.6? If yes, how? Won't that 
> change the device numbers too?

It won't matter to a user application whether /dev/sdaa7 is 65,161 or
257,1667.  As you should understand, since dynamic device numbers relies
on this property.

> Changes in these area require kernel policy, exactly like dynamic device 
> numbers. If you want to maintain compatibility, you likely require user 
> space support, exactly like dynamic device numbers. So why don't we even 
> consider dynamic device numbers, especially as they have the promise of 
> the simpler long term solution. The required kernel changes are certainly 
> not more complex than the patches Andries got merged and still wants to 
> get merged. If you actually look at the character device patches I posted, 
> they already allow simple cleanups for 2.6.

Ah, OK, this may be the misunderstanding.  Kernel dev_t can be expanded
without any change to the current naming policy.  The only change is the
numbering in /dev, which is simple.

> > I said in my last mail that "thanks to the work of Al Viro and others,
> > the problem of dynamic majors for block devices lies predominantly in
> > user space".  The piece that you label "trivial" is the missing
> > character device and user space components.  I don't happen to believe
> > it is at all trivial, but you're welcome to prove me wrong.
> 
> The only nontrivial missing piece is how we want to map the sysfs and 
> other information to static device names.

So we agree.

> OTOH if you only need a 
> solution for scsi now, it should be rather simple to expand scsidev.
> If you can live with dynamic device names (what scsi currently has), it's 
> a simple shell script to generate the device nodes.

But, as I've said before, not simpler than expanding kernel dev_t.

> If you want to have thousands of disk, you need some kind of user support 
> anyway, but I get surprisingly little answers, about how people want to 
> actually manage that much devices? All I hear is "we want them and we 
> want them now!", this makes it difficult to actually answer the question, 
> how kernel should support this. In the kernel it's simpler to keep this 
> dynamic, scsi does that already anyway. How will user space find these 
> devices? Will it continue to scan thousands of nodes in /dev or will it 
> just use the information already easily available via sysfs?

:-)

You take me back a decade to the "thirty two bits is enough for"
everybody and "why would an application want more than 640k of memory"

Operating systems which impose arbitrary limits on their users, or
demand cast iron justification before adding features tend to be at a
competitive disadvantage.

However, if you want an example of how it works today, consider
clustering tools:  They have to identify the same piece of shared
storage as it gets passed around a cluster.  The way we do it (for a few
hundred discs) is to get the SCSI WWN from all the discs (via sg, then
map sg<->sd).  Our application policy is to use the WWN to identify the
device and a number for the partition.  Other cluster tools use the
volume label, still others use the LVM UUID.

The point is that none of these applications cares whether the kernel
uses dynamic or static, all they need to do is be able to identify all
the devices in the system.

> I know I'm asking a lot of annoying questions, but I have the feeling, 
> that hardly anyone really thought all these problems through or someone 
> withholds some important information (e.g. Peter's vague hints about "a 
> fair number of the problems" for dynamic schemes or "collective global 
> experience with numberspaces" don't really help without further 
> explanation).

I wouldn't characterise the questions like that.  Not listening to the
answers now...

> Considering all the information I currently have, dynamic device numbering 
> is the better long term solution and is also usable for 2.6. I'm still 
> looking for the deciding reason, which would give static device numbers a 
> real advantage.

No, dynamic device numbering may turn out to be the better long term
solution.  To justify "is", we'd have to be able to see the solution.

As far as SCSI is concerned, the complete solution has already been
presented for the expansion of kernel dev_t, no such complete solution
has been presented for dynamic devices.  That constitutes a real
advantage to dev_t expansion in my book.

James

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 9 Apr 2003, James Bottomley wrote:

> > Why do we need majors at all? There is no perfect way to partition the 
> > device number, it will always be some compromise. This partitioning 
> > creates more problems than it solves.
> 
> Because without them we need a user space tool or kernel policy add on
> that doesn't yet exist.
> 
> [..]
> 
> > These "enterprise device demands" certainly shouldn't break existing 
> > setups? The patches I've seen from Andries so far do exactly this.
> > What I'm mostly trying to get out of this discussion is how this large 
> > dev_t will be used during 2.6, as this requires decisions now, I'd like 
> > to know and talk about the possible consequences.
> 
> I don't see how the current patches break anything, yet.  I've already
> said how I plan to use a larger kernel dev_t in SCSI.

If you want to use only a single SCSI major, won't it change the device 
numbers? What will happen to setups, which already use more than 16 disks 
(but are still way below the 128 disk limit)?
Do you want to go to 64 partitions during 2.6? If yes, how? Won't that 
change the device numbers too?
Changes in these area require kernel policy, exactly like dynamic device 
numbers. If you want to maintain compatibility, you likely require user 
space support, exactly like dynamic device numbers. So why don't we even 
consider dynamic device numbers, especially as they have the promise of 
the simpler long term solution. The required kernel changes are certainly 
not more complex than the patches Andries got merged and still wants to 
get merged. If you actually look at the character device patches I posted, 
they already allow simple cleanups for 2.6.

> > In this mail 
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=104928874409158&w=2
> > I demonstrated how new device numbers can be generated, without breaking 
> > backwards compatibility, it's quite trivial to complete this patch.
> 
> I said in my last mail that "thanks to the work of Al Viro and others,
> the problem of dynamic majors for block devices lies predominantly in
> user space".  The piece that you label "trivial" is the missing
> character device and user space components.  I don't happen to believe
> it is at all trivial, but you're welcome to prove me wrong.

The only nontrivial missing piece is how we want to map the sysfs and 
other information to static device names. OTOH if you only need a 
solution for scsi now, it should be rather simple to expand scsidev.
If you can live with dynamic device names (what scsi currently has), it's 
a simple shell script to generate the device nodes.
If you want to have thousands of disk, you need some kind of user support 
anyway, but I get surprisingly little answers, about how people want to 
actually manage that much devices? All I hear is "we want them and we 
want them now!", this makes it difficult to actually answer the question, 
how kernel should support this. In the kernel it's simpler to keep this 
dynamic, scsi does that already anyway. How will user space find these 
devices? Will it continue to scan thousands of nodes in /dev or will it 
just use the information already easily available via sysfs?

I know I'm asking a lot of annoying questions, but I have the feeling, 
that hardly anyone really thought all these problems through or someone 
withholds some important information (e.g. Peter's vague hints about "a 
fair number of the problems" for dynamic schemes or "collective global 
experience with numberspaces" don't really help without further 
explanation).
Considering all the information I currently have, dynamic device numbering 
is the better long term solution and is also usable for 2.6. I'm still 
looking for the deciding reason, which would give static device numbers a 
real advantage.

bye, Roman

Re: 64-bit kdev_t - just for playing

[James Bottomley]

On Wed, 2003-04-09 at 15:54, Roman Zippel wrote:
> Why do we need majors at all? There is no perfect way to partition the 
> device number, it will always be some compromise. This partitioning 
> creates more problems than it solves.

Because without them we need a user space tool or kernel policy add on
that doesn't yet exist.

> Simply start allocating from 0x10000 and you can have (2^32-2^16)/partnr 
> block devices.
> The sg nodes problem is also easy to solve, but it requires the character 
> device hash Andries removed.

But, isn't that, in part, the point of this discussion.  Sg with no code
changes will stand the expansion of the kernel dev_t.  There is only a
problem if you want the device numbers dynamically assigned.

> So let's "taste" a few ideas. I don't want any decision, I want to get a 
> discussion started, which explores some of the possibilities, so that we 
> have _some_ idea of what we need.

That discussion doesn't have much to do with the size of the kernel
dev_t, though.  For dynamic devices, it's just a number.

> > However, there is already consideration of this issue, see for example:
> > 
> > http://www.linuxsymposium.org/2003/view_abstract.php?talk=94
> 
> I'd love to see this implemented and I would certainly like to help, but 
> I'm mostly interested in the kernel side of this.
> I haven't found much information about this, so it's difficult to comment 
> on this.

Erm, I'm afraid I believe the idea to be based on leveraging the
existing infrastructure, so I'd be surprised if much kernel work were
required (well beyond what is already planned for hotplug, anyway).

> These "enterprise device demands" certainly shouldn't break existing 
> setups? The patches I've seen from Andries so far do exactly this.
> What I'm mostly trying to get out of this discussion is how this large 
> dev_t will be used during 2.6, as this requires decisions now, I'd like 
> to know and talk about the possible consequences.

I don't see how the current patches break anything, yet.  I've already
said how I plan to use a larger kernel dev_t in SCSI.

> In this mail 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=104928874409158&w=2
> I demonstrated how new device numbers can be generated, without breaking 
> backwards compatibility, it's quite trivial to complete this patch.

I said in my last mail that "thanks to the work of Al Viro and others,
the problem of dynamic majors for block devices lies predominantly in
user space".  The piece that you label "trivial" is the missing
character device and user space components.  I don't happen to believe
it is at all trivial, but you're welcome to prove me wrong.

James

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Wed, 9 Apr 2003 Andries.Brouwer@cwi.nl wrote:

> Your questions are about the meaning of this number,
> that is, about the third part. What I am doing is only
> removing certain restrictions on the size of the number.

If your patches would just removes these restriction, I wouldn't mind at 
all, but your patches do more than that. Why?

> Your letters carry the tone of "it is forbidden to work on the
> old scheme before you have shown how to solve all device naming
> problems". But I am not going to.
> 
> You have opinions and questions about future schemes.
> And so do I. But since time is limited I wrote you
> already a handful of times: "Later".
> 
> This number stuff is simple and straightforward, we know precisely
> what has to be done, but of course it needs to be done.

I don't want to forbid you anything, I want that you explain what you do, 
as your patches do more than simply enlarging dev_t. You still avoid any 
clear answer about this.

> Naming on the other hand is intricate, lots of complications.
> Device naming - but what is a device? Already that is complicated.
> These are good discussions, and maybe sysfs will provide the answer
> in certain cases, but these discussions are independent of dev_t.

They are not independent. You want to have a larger dev_t so it can be 
used for 2.6, but this also requires an answer to the question "How will 
it be used during 2.6?".

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 9 Apr 2003, James Bottomley wrote:

> > Let's try it with a real example:
> > I have two onboard SCSI channels, the first one is for external
> devices 
> > and the second for internal devices.
> 
> There seems to be some confusion here.  As I understand it, you're
> advocating a completely dynamic device space (both major and minor), but
> the concrete examples you post come from devices that do dynamic minors
> only.

These examples were intended to show problems of the current dynamic and 
static numbering. I use examples like these to evaluate how a new 
numbering scheme helps to manage devices.

> As far as SCSI goes, in the current 8/8 device scheme, we occupy 16
> different majors for sd already, but this only gives us room for 256
> discs and we had to compromise and only have 16 partitions on each (as
> opposed to the 64 that IDE has).  Even if you expand this (as there are
> patches to do), we get into trouble because we only have 256 sg nodes,
> etc.
> 
> Expanding the size of the kernel dev_t will allow us to occupy only one
> major,  for each SCSI device type, supply far more discs and still move
> to a 64 partition model.

Why do we need majors at all? There is no perfect way to partition the 
device number, it will always be some compromise. This partitioning 
creates more problems than it solves.
Simply start allocating from 0x10000 and you can have (2^32-2^16)/partnr 
block devices.
The sg nodes problem is also easy to solve, but it requires the character 
device hash Andries removed.

> > I have to come back to the two questions I already asked earlier:
> > 1. How do we want to manage devices in the future?
> 
> Well, it's a legitimate question to ask, but not one anyone is required
> to answer.  The whole "taste" thing in the kernel is about making
> correct decisions without necessarily seeing the ultimate end points. 
> Enabling rather than dictating.  Nothing about an expanded kernel dev_t
> precludes more dynamism in major number allocation.

So let's "taste" a few ideas. I don't want any decision, I want to get a 
discussion started, which explores some of the possibilities, so that we 
have _some_ idea of what we need.

> However, there is already consideration of this issue, see for example:
> 
> http://www.linuxsymposium.org/2003/view_abstract.php?talk=94

I'd love to see this implemented and I would certainly like to help, but 
I'm mostly interested in the kernel side of this.
I haven't found much information about this, so it's difficult to comment 
on this.

> > 2. What compromises can we make for 2.6?
> 
> I think that's expand the kernel's device type but keep the current
> static major/static or dynamic minor.  It seems to me, at this late
> stage in the game, that this will cause the minimum disruption and
> require the minimum of code changes, while still allowing us to satisfy
> the enterprise device demands.  Pragmatically as well, we already have
> the patches for this, we don't for dynamic majors.

These "enterprise device demands" certainly shouldn't break existing 
setups? The patches I've seen from Andries so far do exactly this.
What I'm mostly trying to get out of this discussion is how this large 
dev_t will be used during 2.6, as this requires decisions now, I'd like 
to know and talk about the possible consequences.

In this mail 
http://marc.theaimsgroup.com/?l=linux-kernel&m=104928874409158&w=2
I demonstrated how new device numbers can be generated, without breaking 
backwards compatibility, it's quite trivial to complete this patch.

Re: 64-bit kdev_t - just for playing

[James Bottomley]

> Let's try it with a real example:
> I have two onboard SCSI channels, the first one is for external
devices 
> and the second for internal devices.

There seems to be some confusion here.  As I understand it, you're
advocating a completely dynamic device space (both major and minor), but
the concrete examples you post come from devices that do dynamic minors
only.

Thanks to the work of Al Viro and others, the problem of dynamic majors
for block devices now lies predominantly in user space, but those
problems are still significant.

As far as SCSI goes, in the current 8/8 device scheme, we occupy 16
different majors for sd already, but this only gives us room for 256
discs and we had to compromise and only have 16 partitions on each (as
opposed to the 64 that IDE has).  Even if you expand this (as there are
patches to do), we get into trouble because we only have 256 sg nodes,
etc.

Expanding the size of the kernel dev_t will allow us to occupy only one
major,  for each SCSI device type, supply far more discs and still move
to a 64 partition model.

> I have to come back to the two questions I already asked earlier:
> 1. How do we want to manage devices in the future?

Well, it's a legitimate question to ask, but not one anyone is required
to answer.  The whole "taste" thing in the kernel is about making
correct decisions without necessarily seeing the ultimate end points. 
Enabling rather than dictating.  Nothing about an expanded kernel dev_t
precludes more dynamism in major number allocation.

However, there is already consideration of this issue, see for example:

http://www.linuxsymposium.org/2003/view_abstract.php?talk=94

If you have other contributions to make, I'm sure people will listen.

> 2. What compromises can we make for 2.6?

I think that's expand the kernel's device type but keep the current
static major/static or dynamic minor.  It seems to me, at this late
stage in the game, that this will cause the minimum disruption and
require the minimum of code changes, while still allowing us to satisfy
the enterprise device demands.  Pragmatically as well, we already have
the patches for this, we don't for dynamic majors.

James

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

> Peter and Andries, I would really appreciate it, if
> you would stop ignoring me

But Roman, I answered a dozen of your letters, and
it seems to me that further letters would only
lead to repetition.

Let me recapitulate.

(i) On dev_t
A dev_t is a number. There are three streams:
mknod sends such a number from user space to the file system,
stat sends such a number from the file system to user space,
open and mount send such a number from the file system to the kernel
where the kernel finds an associated device.

Your questions are about the meaning of this number,
that is, about the third part. What I am doing is only
removing certain restrictions on the size of the number.


(ii) On device naming.
One can handle device naming in the old-fashioned way,
as Unix always did, or one can invent one of many possible
schemes for the future.
This old-fashioned way has a myriad of problems, but it works,
people are used to it and know precisely what the problems are,
and our present software handles it.
These schemes of the future are not yet crystallized out very well,
there are several to choose from, we do not yet know very well
what the problems will be, most of the software to handle such
new schemes still has to be written.

[And note that stat is an important system call - even with
new naming schemes we may need numbers - possibly some hash of
whatever ID we have found.]

Clearly, there will be a long transition period where these
schemes will coexist.

Now this old fashioned scheme has run into some limits -
it ran into them already several years ago, witness the
introduction of several scsi disk majors. Removing these limits
is not very difficult, so we do that now.

Your letters carry the tone of "it is forbidden to work on the
old scheme before you have shown how to solve all device naming
problems". But I am not going to.

You have opinions and questions about future schemes.
And so do I. But since time is limited I wrote you
already a handful of times: "Later".

This number stuff is simple and straightforward, we know precisely
what has to be done, but of course it needs to be done.

Naming on the other hand is intricate, lots of complications.
Device naming - but what is a device? Already that is complicated.
These are good discussions, and maybe sysfs will provide the answer
in certain cases, but these discussions are independent of dev_t.

Andries

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Tue, 1 Apr 2003, Greg KH wrote:

> On Tue, Apr 01, 2003 at 03:42:14PM +0100, Alan Cox wrote:
> 
> > We need to default to 12:20 for char but where the 20 is actually
> > defaulting to 0000xx so we don't get extra minors for any device
> > that hasnt been audited for it

Why do we need to introduce new arbitrary limits?
I played a bit with the patch below, which adds dynamic device numbers and 
with a simple script like this the corresponding dev entries can be 
created:

mount -tsysfs none /sysfs
cd /sysfs/block
for d in *; do
        n=0x`cat $d/dev`
        ~/mn /dev/$d $n
        p=1
        while [ $p -lt `cat $d/range` ]; do
                ~/mn /dev/$d$p $[$n+$p]
                p=$[$p+1]
        done
done

We are sooo close to dynamic device numbers, so I really don't understand 
why people want to go back all the way back to static numbers.
The kernel is mostly ready, what is missing now is the userspace and a 
driver audit.

> I thought Andries's patch prevented the need for this, as it restricted
> the number of minors assigned to a device unless they converted to the
> new API.

This 'new API' is _huge_ step backwards, because it puts the burden of 
managing static device numbers on the drivers again.

bye, Roman

Index: drivers/block/genhd.c
===================================================================
RCS file: /home/other/cvs/linux/linux-2.5/drivers/block/genhd.c,v
retrieving revision 1.1.1.31
diff -u -p -r1.1.1.31 genhd.c
--- drivers/block/genhd.c	25 Mar 2003 09:21:17 -0000	1.1.1.31
+++ drivers/block/genhd.c	2 Apr 2003 11:07:16 -0000
@@ -253,6 +253,14 @@ static int exact_lock(dev_t dev, void *d
  */
 void add_disk(struct gendisk *disk)
 {
+	if (disk->flags & GENHD_FL_DYNAMIC) {
+		static dev_t next_dev = 0x10000;
+
+		disk->major = MAJOR(next_dev);
+		disk->first_minor = 0;
+		next_dev += 0x100;
+	}
+
 	disk->flags |= GENHD_FL_UP;
 	blk_register_region(MKDEV(disk->major, disk->first_minor),
 			    disk->minors, NULL, exact_match, exact_lock, disk);
Index: drivers/scsi/sd.c
===================================================================
RCS file: /home/other/cvs/linux/linux-2.5/drivers/scsi/sd.c,v
retrieving revision 1.1.1.38
diff -u -p -r1.1.1.38 sd.c
--- drivers/scsi/sd.c	18 Mar 2003 09:37:07 -0000	1.1.1.38
+++ drivers/scsi/sd.c	2 Apr 2003 12:41:34 -0000
@@ -56,7 +56,7 @@
  * Remaining dev_t-handling stuff
  */
 #define SD_MAJORS	16
-#define SD_DISKS	(SD_MAJORS << 4)
+#define SD_DISKS	4096
 
 /*
  * Time out in seconds for disks and Magneto-opticals (which are slower).
@@ -125,8 +125,7 @@ static int sd_major(int major_idx)
 	case 8 ... 15:
 		return SCSI_DISK8_MAJOR + major_idx;
 	default:
-		BUG();
-		return 0;	/* shut up gcc */
+		return 0;
 	}
 }
 
@@ -1344,6 +1343,10 @@ static int sd_attach(struct scsi_device 
 
 	gd->driverfs_dev = &sdp->sdev_driverfs_dev;
 	gd->flags = GENHD_FL_DRIVERFS | GENHD_FL_DEVFS;
+#if 0
+	if (!gd->major)
+#endif
+		gd->flags |= GENHD_FL_DYNAMIC;
 	if (sdp->removable)
 		gd->flags |= GENHD_FL_REMOVABLE;
 	gd->private_data = &sdkp->driver;
Index: include/asm-i386/posix_types.h
===================================================================
RCS file: /home/other/cvs/linux/linux-2.5/include/asm-i386/posix_types.h,v
retrieving revision 1.1.1.2
diff -u -p -r1.1.1.2 posix_types.h
--- include/asm-i386/posix_types.h	25 Feb 2003 11:50:47 -0000	1.1.1.2
+++ include/asm-i386/posix_types.h	2 Apr 2003 11:07:16 -0000
@@ -7,7 +7,7 @@
  * assume GCC is being used.
  */
 
-typedef unsigned short	__kernel_dev_t;
+typedef unsigned long	__kernel_dev_t;
 typedef unsigned long	__kernel_ino_t;
 typedef unsigned short	__kernel_mode_t;
 typedef unsigned short	__kernel_nlink_t;
Index: include/linux/genhd.h
===================================================================
RCS file: /home/other/cvs/linux/linux-2.5/include/linux/genhd.h,v
retrieving revision 1.1.1.18
diff -u -p -r1.1.1.18 genhd.h
--- include/linux/genhd.h	18 Mar 2003 09:39:54 -0000	1.1.1.18
+++ include/linux/genhd.h	2 Apr 2003 11:07:16 -0000
@@ -71,6 +71,7 @@ struct hd_struct {
 #define GENHD_FL_DEVFS	4
 #define GENHD_FL_CD	8
 #define GENHD_FL_UP	16
+#define GENHD_FL_DYNAMIC 32
 
 struct disk_stats {
 	unsigned read_sectors, write_sectors;
Index: include/linux/kdev_t.h
===================================================================
RCS file: /home/other/cvs/linux/linux-2.5/include/linux/kdev_t.h,v
retrieving revision 1.1.1.6
diff -u -p -r1.1.1.6 kdev_t.h
--- include/linux/kdev_t.h	5 Nov 2002 10:48:59 -0000	1.1.1.6
+++ include/linux/kdev_t.h	2 Apr 2003 11:07:16 -0000
@@ -70,7 +70,7 @@ aeb - 950811
  * static arrays, and they are sized for a 8-bit index.
  */
 typedef struct {
-	unsigned short value;
+	unsigned long value;
 } kdev_t;
 
 #define KDEV_MINOR_BITS		8
@@ -112,7 +112,7 @@ static inline int kdev_same(kdev_t dev1,
 
 /* Mask off the high bits for now.. */
 #define minor(dev)	((dev).value & 0xff)
-#define major(dev)	(((dev).value >> KDEV_MINOR_BITS) & 0xff)
+#define major(dev)	((dev).value >> KDEV_MINOR_BITS)
 
 /* These are for user-level "dev_t" */
 #define MINORBITS	8

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Christoph Hellwig wrote:
> 
> For block devices this is no problem, they register regions not majors.
> For character devices that sounds the way to go, too.  I'll look into it
> once I've finished by devfs API sanitizing, but if viro magically reappeared
> and could post his existing patches for it this would make it even easier...
 >

Personally I think it's needless complexity (workaround for a too-small 
address space which is better fixed by making the address space larger), 
but it does allow for more flexibility and assuming it doesn't come with 
unacceptable overhead I guess it doesn't hurt.

	-hpa

Re: 64-bit kdev_t - just for playing

[Christoph Hellwig]

On Tue, Apr 01, 2003 at 01:59:23PM -0800, H. Peter Anvin wrote:
> Or 32:32... if we have a hope for that.  Given that the bulk of the
> overhead is *already* taken in userspace I still think we should go
> all the way on this one.

Yes, having the same represantation as glibc sounds like a good idea
in general.  And the more we use struct block_device * / struct char_device *
in the kernel the less the big size actually matters.

> But yes, we need to make sure old-api
> devices get -ENXIO on open anything beyond the 8-bit minor space.

For block devices this is no problem, they register regions not majors.
For character devices that sounds the way to go, too.  I'll look into it
once I've finished by devfs API sanitizing, but if viro magically reappeared
and could post his existing patches for it this would make it even easier...

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <1049208134.19703.12.camel@dhcp22.swansea.linux.org.uk>
By author:    Alan Cox <alan@lxorguk.ukuu.org.uk>
In newsgroup: linux.dev.kernel
> 
> We need to default to 12:20 for char but where the 20 is actually
> defaulting to 0000xx so we don't get extra minors for any device
> that hasnt been audited for it
> 

Or 32:32... if we have a hope for that.  Given that the bulk of the
overhead is *already* taken in userspace I still think we should go
all the way on this one.  But yes, we need to make sure old-api
devices get -ENXIO on open anything beyond the 8-bit minor space.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

> Why do we need a split at all?

Inside the kernel? No need at all.
But the Unix API is in terms of major,minor.

The NFS specification talks about major,minor.
The ISO 9660 (RockRidge) standard talks about major,minor.
Etc.

Inside the kernel we can do whatever we want, and no split
is required. In userspace such a split definitely exists.

See also mknod(1) and ls(1).

Andries

Re: 64-bit kdev_t - just for playing

[Christoph Hellwig]

On Tue, Apr 01, 2003 at 03:42:24PM +0100, Alan Cox wrote:
> We need to default to 12:20 for char but where the 20 is actually
> defaulting to 0000xx so we don't get extra minors for any device
> that hasnt been audited for it

Why do we need a split at all?  Just make it reserve a 8k range which
is the same as the old major.  No problem with wasting space then.

Re: 64-bit kdev_t - just for playing

[Greg KH]

On Tue, Apr 01, 2003 at 03:42:14PM +0100, Alan Cox wrote:
> On Tue, 2003-04-01 at 00:42, Roman Zippel wrote:
> > Hi,
> > 
> > On 31 Mar 2003, Alan Cox wrote:
> > 
> > > > 2. What compromises can we make for 2.6?
> > > 
> > > Defaulting char devices to 256 minors and a lot of space so stuff doesnt
> > > break. Viro has done the block stuff and we have the scope to do sane
> > > stuff like /dev/disk/.. for all disks now.
> > 
> > What do you mean with "a lot of space so stuff doesnt break"?
> 
> We need to default to 12:20 for char but where the 20 is actually
> defaulting to 0000xx so we don't get extra minors for any device
> that hasnt been audited for it

I thought Andries's patch prevented the need for this, as it restricted
the number of minors assigned to a device unless they converted to the
new API.

thanks,

greg k-h

Re: 64-bit kdev_t - just for playing

[Alan Cox]

On Tue, 2003-04-01 at 00:42, Roman Zippel wrote:
> Hi,
> 
> On 31 Mar 2003, Alan Cox wrote:
> 
> > > 2. What compromises can we make for 2.6?
> > 
> > Defaulting char devices to 256 minors and a lot of space so stuff doesnt
> > break. Viro has done the block stuff and we have the scope to do sane
> > stuff like /dev/disk/.. for all disks now.
> 
> What do you mean with "a lot of space so stuff doesnt break"?

We need to default to 12:20 for char but where the 20 is actually
defaulting to 0000xx so we don't get extra minors for any device
that hasnt been audited for it


> > Glibc already has a bigger dev_t
> 
> and a broken mknod implementation...

Easy enough to deal with

Re: 64-bit kdev_t - just for playing

[Alan Cox]

On Tue, 2003-04-01 at 00:42, Roman Zippel wrote:
> Hi,
> 
> On 31 Mar 2003, Alan Cox wrote:
> 
> > > 2. What compromises can we make for 2.6?
> > 
> > Defaulting char devices to 256 minors and a lot of space so stuff doesnt
> > break. Viro has done the block stuff and we have the scope to do sane
> > stuff like /dev/disk/.. for all disks now.
> 
> What do you mean with "a lot of space so stuff doesnt break"?

We need to default to 12:20 for char but where the 20 is actually
defaulting to 0000xx so we don't get extra minors for any device
that hasnt been audited for it


> > Glibc already has a bigger dev_t
> 
> and a broken mknod implementation...

Easy enough to deal with

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Mon, Mar 31, 2003 at 11:18:54PM +0100, Alan Cox wrote:
> Glibc already has a bigger dev_t

	Yes, but they hand-map 8:8 in functions like xmknod().  They
should really be using macros.

Joel

-- 

Life's Little Instruction Book #314

	"Never underestimate the power of forgiveness."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 31 Mar 2003, Alan Cox wrote:

> > 2. What compromises can we make for 2.6?
> 
> Defaulting char devices to 256 minors and a lot of space so stuff doesnt
> break. Viro has done the block stuff and we have the scope to do sane
> stuff like /dev/disk/.. for all disks now.

What do you mean with "a lot of space so stuff doesnt break"?

> > Without answering these questions now, we risk to pay heavily for it 
> > later. The ones who ask now for a larger dev_t the loudest are likely the 
> > first to demand later not change anything for "compability", because they 
> > hardcoded certain assumptions about dev_t into their applications.
> 
> Glibc already has a bigger dev_t

and a broken mknod implementation...

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 31 Mar 2003, Joel Becker wrote:

> 	I'm right here campaigning loudly for a larger dev_t.  I intend
> to never, ever make assumptions about dev_t.  In fact, I'd rather not
> deal with dev_t.  But I do need a way to map 4k or 8k or 16k disks.
> now.

Fine, so write the software to do this, but what exactly is there still 
for the kernel to do?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Mon, Mar 31, 2003 at 11:32:55PM +0200, Roman Zippel wrote:
> later. The ones who ask now for a larger dev_t the loudest are likely the 
> first to demand later not change anything for "compability", because they 
> hardcoded certain assumptions about dev_t into their applications.

	I'm right here campaigning loudly for a larger dev_t.  I intend
to never, ever make assumptions about dev_t.  In fact, I'd rather not
deal with dev_t.  But I do need a way to map 4k or 8k or 16k disks.
now.

Joel

-- 

"Up and down that road in our worn out shoes,
 Talking bout good things and singing the blues."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Alan Cox]

On Mon, 2003-03-31 at 22:32, Roman Zippel wrote:
> 1. How do we want to manage dev_t numbers in the future?

Mostly dynamically it seems

> 2. What compromises can we make for 2.6?

Defaulting char devices to 256 minors and a lot of space so stuff doesnt
break. Viro has done the block stuff and we have the scope to do sane
stuff like /dev/disk/.. for all disks now.

> Without answering these questions now, we risk to pay heavily for it 
> later. The ones who ask now for a larger dev_t the loudest are likely the 
> first to demand later not change anything for "compability", because they 
> hardcoded certain assumptions about dev_t into their applications.

Glibc already has a bigger dev_t

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 31 Mar 2003, Joel Becker wrote:

> > > Can you envision solutions based on 16 bit kdev_t infrastructure? 
> > 
> > I know that 16bit is getting small (but with dynamic assignment even 
> > that is still enough for most people), but OTOH I don't understand the 
> > obsession for 64bit. 32bit is more than enough on a 32bit system.
> 
> 	There are big companies out there that require thousands of
> disks.  Many don't want to use hardware raid, they just want JBOD.
> There are installations today with 2k-4k disks covering the gamut from
> massive databases to HPC to research facilities.  Today.
> 	A 32bit dev_t with the 12/20 Linus split provides 64k minors.
> That's 16k disks with our current 15-partitions-per limit.  But if
> someone wants 35 partitions (I've seen that somewhere) you're down to
> 8k.  And the places using 2-4k disks today will be getting to 8k disks
> soon after 2.6 becomes usable, if not before.  They will likely hit 16k
> disks before 2.6 becomes an afterthought.

Umm, I must be missing something, I get here 1024k minors, 64k disks with 
15 partitions and 16k disks with 63 partitions.
Anyway, the sooner userspace accepts that dev_t number is just a number 
the better. The major/minor split is useless information in userspace and 
it becomes less and less important in the kernel.

> > Somehow it sounds that we just have to introduce a huge dev_t and all our 
> > problems are magically solved and I doubt that. If people want to encode 
> 
> 	64bit dev_t is not a panacea.  However, if we have to do this
> change, we want to do it once.  This only solves the space issue.  All
> of the other issues, such as naming, are orthogonal to this change.
> Holding one change until everything else has been written is silly.

SCSI already enumerates the devices sequentially, without userspace tools
already now it's very painful to manage thousands of disk, but all the
kernel can do is to give you all the information about the disks it has
and the number you can use to access the disk. How you call that thing is
a userspace issue not a kernel issue.
A lot of information is already exported via sysfs, missing information 
can be added. Changing the kernel to sequentially assign dev_t numbers 
starting 0x10000, when they are registered via add_disk, is a rather 
simple change. Then you have can have 16M disks with 256 partitions, that 
should be enough for a while?
What mostly is missing now is the userspace code. I haven't seen a 
request, that someone has simple daemon and needs now this and that 
information from the kernel to map a disk reliably to a name.

> 	There is no conspiracy.  Everyone agrees we need more dev_t
> space.  Peter, Andries, and others see it like I do; we only want to do
> this once, and we already can see a day when 20bits of minors isn't
> enough.

If you only want to do it once, then do it right. There are 2 basic 
questions, which have to be answered:
1. How do we want to manage dev_t numbers in the future?
2. What compromises can we make for 2.6?
Without answering these questions now, we risk to pay heavily for it 
later. The ones who ask now for a larger dev_t the loudest are likely the 
first to demand later not change anything for "compability", because they 
hardcoded certain assumptions about dev_t into their applications.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Mon, Mar 31, 2003 at 10:52:53AM +0200, Roman Zippel wrote:
> > Can you envision solutions based on 16 bit kdev_t infrastructure? 
> 
> I know that 16bit is getting small (but with dynamic assignment even 
> that is still enough for most people), but OTOH I don't understand the 
> obsession for 64bit. 32bit is more than enough on a 32bit system.

	There are big companies out there that require thousands of
disks.  Many don't want to use hardware raid, they just want JBOD.
There are installations today with 2k-4k disks covering the gamut from
massive databases to HPC to research facilities.  Today.
	A 32bit dev_t with the 12/20 Linus split provides 64k minors.
That's 16k disks with our current 15-partitions-per limit.  But if
someone wants 35 partitions (I've seen that somewhere) you're down to
8k.  And the places using 2-4k disks today will be getting to 8k disks
soon after 2.6 becomes usable, if not before.  They will likely hit 16k
disks before 2.6 becomes an afterthought.

> Somehow it sounds that we just have to introduce a huge dev_t and all our 
> problems are magically solved and I doubt that. If people want to encode 

	64bit dev_t is not a panacea.  However, if we have to do this
change, we want to do it once.  This only solves the space issue.  All
of the other issues, such as naming, are orthogonal to this change.
Holding one change until everything else has been written is silly.

> (Slowly I'm getting the feeling that there is some sort of 64bit dev_t 
> conspiracy going on here, with the amount of answers I'm (not) getting 
> here.)

	There is no conspiracy.  Everyone agrees we need more dev_t
space.  Peter, Andries, and others see it like I do; we only want to do
this once, and we already can see a day when 20bits of minors isn't
enough.

Joel

-- 

Life's Little Instruction Book #207

	"Swing for the fence."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Mon, 31 Mar 2003, bert hubert wrote:

> > If Andries would actually explain, what he wants to do with the larger 
> > dev_t, it would be a lot easier to help him, so that we can at least avoid 
> > the biggest mistakes.
> 
> Can you envision solutions based on 16 bit kdev_t infrastructure? 

I know that 16bit is getting small (but with dynamic assignment even 
that is still enough for most people), but OTOH I don't understand the 
obsession for 64bit. 32bit is more than enough on a 32bit system.
Somehow it sounds that we just have to introduce a huge dev_t and all our 
problems are magically solved and I doubt that. If people want to encode 
random information into dev_t, then even 64bit will be soon not enough 
anymore, so I want to know how people actually want to use that large 
dev_t number. Is that really too much to ask?
(Slowly I'm getting the feeling that there is some sort of 64bit dev_t 
conspiracy going on here, with the amount of answers I'm (not) getting 
here.)

bye, Roman

Re: 64-bit kdev_t - just for playing

[bert hubert]

On Fri, Mar 28, 2003 at 07:48:13PM +0100, Roman Zippel wrote:

> If Andries would actually explain, what he wants to do with the larger 
> dev_t, it would be a lot easier to help him, so that we can at least avoid 
> the biggest mistakes.

Can you envision solutions based on 16 bit kdev_t infrastructure? 

Regards,

bert

-- 
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 30 Mar 2003, H. Peter Anvin wrote:

> > The size of dev_t doesn't matter at all,
> 
> Bullshit.

Huh?
(Do I love all these detailed explanations... *sigh*)

bye, Roman

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <UTC200303270109.h2R19ME28410.aeb@smtp.cwi.nl>
By author:    Andries.Brouwer@cwi.nl
In newsgroup: linux.dev.kernel
>
> >> Maybe I should send another patch tonight, just for playing.
> 
> > Please, I'd like that.
> 
> Below a random version of kdev_t.h.
> (The file is smaller than the patch.)
> 
> kdev_t is the kernel-internal representation
> dev_t is the kernel idea of the user space representation
> (of course glibc uses 64 bits, split up as 8+8 :-)
> 
> kdev_t can be equal to dev_t.
> 
> The file below completely randomly makes kdev_t
> 64 bits, split up 32+32, and dev_t 32 bits, split up 12+20.
> 

I have a few brief questions:

a) Along all of these you have assumed that it's more efficient to
have a separate type (kdev_t) for kernel-internal "decoded" device number
handling, as opposed to "encoded" device number handling.  At some
point, however, it ends up being a struct char_dev * or struct
block_dev *.  How big is this gap and does it really make sense
introducing a special type for it?

b) If we do have such a type, would it make more sense to have cdev_t
and bdev_t, and have per-type distinction of block- versus charness?

c) If we do have such a type, any reason to have it be a "unsigned
long long" (really should be u64), instead of "u32 major; u32 minor;"?

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.44.0303281219350.5042-100000@serv>
By author:    Roman Zippel <zippel@linux-m68k.org>
In newsgroup: linux.dev.kernel
> 
> The size of dev_t doesn't matter at all,
> 

Bullshit.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Fri, 28 Mar 2003, Joel Becker wrote:

> 	I represent the users which need this most, and I an tell you we
> will be 100x happier pointing and guessing at enough dev_t space.  If we
> were to have to stick with the ancient, serously outdated limits of the
> current space, we will be terribly unhappy.
> 	Not having the perfect solution all at once doesn't mean you do
> nothing.  The size of dev_t is orthogonal to device naming.  Once this
> is in, the current device naming (however poor it is) can handle the
> number of devices we need.  Future device naming strategies (like the
> one Greg is working on) will work with a large dev_t just fine.

I don't mind temporary solutions, but I prefer the ones, which don't have 
to be thrown away completely (e.g. like Andries char device changes).
If Andries would actually explain, what he wants to do with the larger 
dev_t, it would be a lot easier to help him, so that we can at least avoid 
the biggest mistakes.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Joel Becker]

On Fri, Mar 28, 2003 at 10:47:14AM +0100, Roman Zippel wrote:
> On Thu, 27 Mar 2003, Greg KH wrote:
> > They point and guess, just like they do today :)
> 
> I think the users which need this most won't be particular happy.

	I represent the users which need this most, and I an tell you we
will be 100x happier pointing and guessing at enough dev_t space.  If we
were to have to stick with the ancient, serously outdated limits of the
current space, we will be terribly unhappy.
	Not having the perfect solution all at once doesn't mean you do
nothing.  The size of dev_t is orthogonal to device naming.  Once this
is in, the current device naming (however poor it is) can handle the
number of devices we need.  Future device naming strategies (like the
one Greg is working on) will work with a large dev_t just fine.

Joel

-- 

"Vote early and vote often." 
        - Al Capone

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Fri, 28 Mar 2003 Andries.Brouwer@cwi.nl wrote:

> I would prefer if you waited a bit. This little detail,
> changing the size of dev_t, requires an audit of the
> kernel source. That takes some time.
> I would much prefer postponing discussion about device
> handling until after number handling is in good shape.

You already changed the device handling and you don't want to discuss it? 
My patch does the same without the questionable device handling changes.

> Generally it is a bad idea when two people simultaneously
> change the same code.

Ignoring other people's comments and questions is generally a bad idea as 
well.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

    On Fri, 28 Mar 2003 Andries.Brouwer@cwi.nl wrote:

    > > the actual size of this number is only a small detail
    > 
    > Yes. It is that detail I am concerned with.

    If you don't want to or can't answer my question, it means I can revert 
    your character device changes?

I am not Linus. You can send him whatever you think best
and see whether he applies it.

I would prefer if you waited a bit. This little detail,
changing the size of dev_t, requires an audit of the
kernel source. That takes some time.
I would much prefer postponing discussion about device
handling until after number handling is in good shape.

Generally it is a bad idea when two people simultaneously
change the same code.

Andries

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Fri, 28 Mar 2003 Andries.Brouwer@cwi.nl wrote:

> > the actual size of this number is only a small detail
> 
> Yes. It is that detail I am concerned with.

If you don't want to or can't answer my question, it means I can revert 
your character device changes?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

> the actual size of this number is only a small detail

Yes. It is that detail I am concerned with.

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Fri, 28 Mar 2003 Andries.Brouwer@cwi.nl wrote:

> Roman, Your questions are misguided.

Thanks for your trust. :-(

> A larger dev_t is infrastructure.
> A sand road that is turned into an asphalt road.
> 
> Nobody has to use this improved infrastructure.
> But many uses are conceivable.

The size of dev_t doesn't matter at all, what matters is how this number 
is managed and used. The kernel has somehow to generate a number for a 
device and tell the user about it, so that he can use it to access the 
device. This requires infrastructure and the actual size of this number is 
only a small detail in the whole picture. I want to know how the whole 
picture looks like, so could you please stop talking bullshit and answer 
my questions?

> I can imagine that there will be people wanting
> to take part of the available space for a universal
> hash of disk serial number or partition label or
> I don't know what, so that devices are addressable
> by content instead of path.

This won't happen, dev_t is the wrong place to encode such information.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

Roman, Your questions are misguided.
A larger dev_t is infrastructure.
A sand road that is turned into an asphalt road.

Nobody has to use this improved infrastructure.
But many uses are conceivable.

Long ago I reserved 2^40 values for dynamically
assigned anonymous devices. Convenient, a very
small fraction of the available space.

I can imagine that there will be people wanting
to take part of the available space for a universal
hash of disk serial number or partition label or
I don't know what, so that devices are addressable
by content instead of path.

A lot of room can be put to many uses.

Andries

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 27 Mar 2003, Greg KH wrote:

> > > Devices.txt or dynamic assignment
> > 
> > The first case means a /dev directory with millions of dev entries.
> > How does the user find out about the number of partitions in the second 
> > case?
> 
> They point and guess, just like they do today :)

I think the users which need this most won't be particular happy.

> > > > Who creates these device entries (user or daemon)?
> > > 
> > > Who cares 8)  Thats just the devfs argument all over again 8)
> > 
> > Why? I specifically didn't mention the kernel.
> > Anyone has to care, somehow this large number space must be managed.
> 
> Yes, some of us are working on this.  But this has nothing to do with
> the kernel, or Andries's patches.  It's a userspace issue.

Somehow the kernel and the userspace have to work together. What I want to 
know is whether we just create another crutch, barely usable for the 
desperate or if we create a solution which has a small chance to still 
work in the future.

bye, Roman

Re: 64-bit kdev_t - just for playing

[Greg KH]

On Fri, Mar 28, 2003 at 12:19:25AM +0100, Roman Zippel wrote:
> > > How will the user know about these numbers?
> > 
> > Devices.txt or dynamic assignment
> 
> The first case means a /dev directory with millions of dev entries.
> How does the user find out about the number of partitions in the second 
> case?

They point and guess, just like they do today :)

> > > Who creates these device entries (user or daemon)?
> > 
> > Who cares 8)  Thats just the devfs argument all over again 8)
> 
> Why? I specifically didn't mention the kernel.
> Anyone has to care, somehow this large number space must be managed.

Yes, some of us are working on this.  But this has nothing to do with
the kernel, or Andries's patches.  It's a userspace issue.

thanks,

greg k-h

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On 27 Mar 2003, Alan Cox wrote:

> > How are these disks registered and how will the dev_t number look like?
> 
> Al Viro's work so far makes those issues you can defer nicely. 

I know his work, I'm just trying to find out, whether Andries understands 
it too, or if he maybe knows something I don't.

> > How will the user know about these numbers?
> 
> Devices.txt or dynamic assignment

The first case means a /dev directory with millions of dev entries.
How does the user find out about the number of partitions in the second 
case?

> > Who creates these device entries (user or daemon)?
> 
> Who cares 8)  Thats just the devfs argument all over again 8)

Why? I specifically didn't mention the kernel.
Anyone has to care, somehow this large number space must be managed.

> > SCSI has multiple majors, disks 0-15 are at major 8, disks 16-31 are at 
> > 65, ...., disks 112-127 are at major 71. Will this stay the same? Where 
> > are the disk 128-xxx?
> > Can I have now more than 15 partitions?
> 
> It becomes possible, more importantly we can begin to support
> partitioned CD-ROM both for multisession and for real partition
> tables on CD (eg Macintrash)

How exactly does this become possible?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 27 Mar 2003 Andries.Brouwer@cwi.nl wrote:

> But, as I answered you several times already,
> right now my topic is dev_t, not devices or partitions.
> Just the number.

Well, what can I do with that number? Your patches must provide some sort 
of benefit when we have that number. I'm currently trying to find out, 
what happens after we have this number, so I would be really grateful, if 
you would answer my questions:

How are these disks registered and how will the dev_t number look like?
How will the user know about these numbers?
Who creates these device entries (user or daemon)?
SCSI has multiple majors, disks 0-15 are at major 8, disks 16-31 are at
65, ...., disks 112-127 are at major 71. Will this stay the same? Where
are the disk 128-xxx?
Can I have now more than 15 partitions?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Alan Cox]

On Thu, 2003-03-27 at 22:12, Roman Zippel wrote:
> How are these disks registered and how will the dev_t number look like?

Al Viro's work so far makes those issues you can defer nicely. 

> How will the user know about these numbers?

Devices.txt or dynamic assignment

> Who creates these device entries (user or daemon)?

Who cares 8)  Thats just the devfs argument all over again 8)

> SCSI has multiple majors, disks 0-15 are at major 8, disks 16-31 are at 
> 65, ...., disks 112-127 are at major 71. Will this stay the same? Where 
> are the disk 128-xxx?
> Can I have now more than 15 partitions?

It becomes possible, more importantly we can begin to support
partitioned CD-ROM both for multisession and for real partition
tables on CD (eg Macintrash)

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

> Can I have now more than 15 partitions?

Two years ago I amused myself creating a few hundred partitions
on a SCSI disk. Yes, no doubt the availability of numbers will
remove the current limits on the number of partitions of a disk.

But, as I answered you several times already,
right now my topic is dev_t, not devices or partitions.
Just the number.

Andries

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 27 Mar 2003 Andries.Brouwer@cwi.nl wrote:

You must have overlooked some of my questions:

How are these disks registered and how will the dev_t number look like?
How will the user know about these numbers?
Who creates these device entries (user or daemon)?

> > How is backward compatibility done, so that I can still boot a 2.4 kernel?
> 
> Old device numbers remain valid, so all changes are completely
> transparent.

SCSI has multiple majors, disks 0-15 are at major 8, disks 16-31 are at 
65, ...., disks 112-127 are at major 71. Will this stay the same? Where 
are the disk 128-xxx?
Can I have now more than 15 partitions?

bye, Roman

Re: 64-bit kdev_t - just for playing

[Andries.Brouwer]

> It would really help, if you would explain how a larger dev_t
> will work during 2.6.

> How is backward compatibility done, so that I can still boot a 2.4 kernel?

Old device numbers remain valid, so all changes are completely
transparent.

> How have user space utilities to be changed, which know about
> dev_t (e.g. ls or fdisk)?

If you do not use mknod to create device nodes with large device numbers,
then no new utilities are needed. If you really want to use large
device numbers, you need a new glibc; some utilities will require
recompilation because of the use of sysmacros.h.

Andries

Re: 64-bit kdev_t - just for playing

[Roman Zippel]

Hi,

On Thu, 27 Mar 2003 Andries.Brouwer@cwi.nl wrote:

> kdev_t is the kernel-internal representation
> dev_t is the kernel idea of the user space representation
> (of course glibc uses 64 bits, split up as 8+8 :-)
> 
> kdev_t can be equal to dev_t.
> 
> The file below completely randomly makes kdev_t
> 64 bits, split up 32+32, and dev_t 32 bits, split up 12+20.

It would really help, if you would explain how a larger dev_t will work 
during 2.6.
Let's take an example, the first user is likely the scsi layer, so how 
can I manage thousands of disks under 2.6?
How are these disks registered and how will the dev_t number look like?
How will the user know about these numbers?
Who creates these device entries (user or daemon)?
How have user space utilities to be changed, which know about dev_t (e.g. 
ls or fdisk)?
How is backward compatibility done, so that I can still boot a 2.4 kernel?

bye, Roman

64-bit kdev_t - just for playing

[Andries.Brouwer]

>> Maybe I should send another patch tonight, just for playing.

> Please, I'd like that.

Below a random version of kdev_t.h.
(The file is smaller than the patch.)

kdev_t is the kernel-internal representation
dev_t is the kernel idea of the user space representation
(of course glibc uses 64 bits, split up as 8+8 :-)

kdev_t can be equal to dev_t.

The file below completely randomly makes kdev_t
64 bits, split up 32+32, and dev_t 32 bits, split up 12+20.

Andries

------------------------------------------------------------
#ifndef _LINUX_KDEV_T_H
#define _LINUX_KDEV_T_H
#ifdef __KERNEL__

typedef struct {
	unsigned long long value;
} kdev_t;

#define KDEV_MINOR_BITS		32
#define KDEV_MAJOR_BITS		32
#define KDEV_MINOR_MASK		((1ULL << KDEV_MINOR_BITS) - 1)

#define __mkdev(major, minor)	(((unsigned long long)(major) << KDEV_MINOR_BITS) + (minor))

#define mk_kdev(major, minor)	((kdev_t) { __mkdev(major,minor) } )

/*
 * The "values" are just _cookies_, usable for 
 * internal equality comparisons and for things
 * like NFS filehandle conversion.
 */
static inline unsigned long long kdev_val(kdev_t dev)
{
	return dev.value;
}

static inline kdev_t val_to_kdev(unsigned long long val)
{
	kdev_t dev;
	dev.value = val;
	return dev;
}

#define HASHDEV(dev)	(kdev_val(dev))
#define NODEV		(mk_kdev(0,0))

extern const char * kdevname(kdev_t);	/* note: returns pointer to static data! */

static inline int kdev_same(kdev_t dev1, kdev_t dev2)
{
	return dev1.value == dev2.value;
}

#define kdev_none(d1)	(!kdev_val(d1))

#define minor(dev)	(unsigned int)((dev).value & KDEV_MINOR_MASK)
#define major(dev)	(unsigned int)((dev).value >> KDEV_MINOR_BITS)

/* These are for user-level "dev_t" */
#define MINORBITS	8
#define MINORMASK	((1U << MINORBITS) - 1)
#define DEV_MINOR_BITS	20
#define	DEV_MAJOR_BITS	12
#define	DEV_MINOR_MASK	((1U << DEV_MINOR_BITS) - 1)
#define DEV_MAJOR_MASK	((1U << DEV_MAJOR_BITS) - 1)

#include <linux/types.h>	/* dev_t */

#define MAJOR(dev)	((unsigned int)(((dev) & 0xffff0000) ? ((dev) >> DEV_MINOR_BITS) & DEV_MAJOR_MASK : ((dev) >> 8) & 0xff))
#define MINOR(dev)	((unsigned int)(((dev) & 0xffff0000) ? ((dev) & DEV_MINOR_MASK) : ((dev) & 0xff)))
#define MKDEV(ma,mi)	((dev_t)((((ma) & ~0xff) == 0 && ((mi) & ~0xff) == 0) ? (((ma) << 8) | (mi)) : (((ma) << DEV_MINOR_BITS) | (mi))))

/*
 * Conversion functions
 */

static inline int kdev_t_to_nr(kdev_t dev)
{
	unsigned int ma = major(dev);
	unsigned int mi = minor(dev);
	return MKDEV(ma, mi);
}

static inline kdev_t to_kdev_t(dev_t dev)
{
	unsigned int ma = MAJOR(dev);
	unsigned int mi = MINOR(dev);
	return mk_kdev(ma, mi);
}

#else /* __KERNEL__ */

/*
Some programs want their definitions of MAJOR and MINOR and MKDEV
from the kernel sources. These must be the externally visible ones.
Of course such programs should be updated.
*/
#define MAJOR(dev)	((dev)>>8)
#define MINOR(dev)	((dev) & 0xff)
#define MKDEV(ma,mi)	((ma)<<8 | (mi))
#endif /* __KERNEL__ */
#endif