About File systems magic numbers

About File systems magic numbers

[inventsekar]

Hi All, i have been reading
Unix filesystem - evolution, design and implementation by Steve D. Pate

and it provides a simple file system source code for learning and playing
with.
On the header file, it defines the fs magic number as
#define UX_MAGIC 0x58494e55

1. which means, will it assign this magic number to ALL files created on
this sample fs? That can't be so. Otherwise, why we need to define a magic
number?
2. I assume, magic number will be assigned to files by the tools which we
use to create the files(like vi, cat, etc). Is this correct?
3. It seems, windows files also have the magic numbers. let's say I copy
some files from windows to Linux. We should have the common magic numbers
between different Operating systems, so there won't be confusions. I mean,
is there any standards common for all os's files magic numbers?!?


Best regards
Sekar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180117/ade99725/attachment-0001.html>

About File systems magic numbers

[Manish Katiyar]

On Wed, Jan 17, 2018 at 8:05 AM, inventsekar <inventsekar@gmail.com> wrote:

> Hi All, i have been reading
> Unix filesystem - evolution, design and implementation by Steve D. Pate
>
> and it provides a simple file system source code for learning and playing
> with.
> On the header file, it defines the fs magic number as
> #define UX_MAGIC 0x58494e55
>
>

You need to understand the reason for having the magic number first. Why is
it needed and then most of the below questions can be answered.


> 1. which means, will it assign this magic number to ALL files created on
> this sample fs? That can't be so. Otherwise, why we need to define a magic
> number?
>

There is no rule for it and it totally depends on the will of the
filesystem developer. If he wants, he can put a magic number in every
file/block etc. even though it might be stupid to do so. Magic numbers are
just a way to identify something you already know. They are used during
filesystem recovery, to verify that things are ok before mounting etc. etc.
If you can't find your known pattern at the place you wrote or where you
expect then something is wrong.


> 2. I assume, magic number will be assigned to files by the tools which we
> use to create the files(like vi, cat, etc). Is this correct?
>

Do not assume :-). Look at the code. Tools don't decide what or where to
put the magic number. It is the FS code which decides irrespective of tool.



> 3. It seems, windows files also have the magic numbers. let's say I copy
> some files from windows to Linux. We should have the common magic numbers
> between different Operating systems, so there won't be confusions. I mean,
> is there any standards common for all os's files magic numbers?!?
>
>
Nope. Magic numbers are a personal (somewhat) choice. They aren't preserved
across filesystems. It would actually be wrong if you can read the magic
number of the FS while doing 'cat' on a file.

Thanks -
Manish




>
> Best regards
> Sekar
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180117/85e3f202/attachment.html>

About File systems magic numbers

[Jim Davis]

n Wed, Jan 17, 2018 at 9:05 AM, inventsekar <inventsekar@gmail.com> wrote:
> Hi All, i have been reading
> Unix filesystem - evolution, design and implementation by Steve D. Pate
>
> and it provides a simple file system source code for learning and playing
> with.
> On the header file, it defines the fs magic number as
> #define UX_MAGIC 0x58494e55
>
> 1. which means, will it assign this magic number to ALL files created on
> this sample fs? That can't be so. Otherwise, why we need to define a magic
> number?

There can be many different types of filesystems on a Linux computer;
the magic number can be used to distinguish which type a particular
filesystem is. If you look at usr/include/linux/magic.h you'll see,
for example,

#define EXT2_SUPER_MAGIC    0xEF53

and as an example of using that particular magic number, check out
arch/blackfin/kernel/setup.c or fs/ext2/super.c.

-- 
Jim

About File systems magic numbers

[valdis.kletnieks at vt.edu]

On Wed, 17 Jan 2018 21:35:13 +0530, inventsekar said:

> On the header file, it defines the fs magic number as
> #define UX_MAGIC 0x58494e55
>
> 1. which means, will it assign this magic number to ALL files created on
> this sample fs? That can't be so. Otherwise, why we need to define a magic
> number?

That's not a file magic number, that's a filesystem superblock magic number.

That appears just once in the filesystem, usually in/near the superblock. It's only
reason for existence is so things like fsck can read in the superblock, and do a:

	if (superblock.magic != UX_MAGIC) {
		printf("I see no UX filesystem here!\n);
		exit(1);
	}

> 2. I assume, magic number will be assigned to files by the tools which we
> use to create the files(like vi, cat, etc). Is this correct?

No, it's scribbled into the superblock by mkfs

> 3. It seems, windows files also have the magic numbers. let's say I copy
> some files from windows to Linux. We should have the common magic numbers
> between different Operating systems, so there won't be confusions. I mean,
> is there any standards common for all os's files magic numbers?!?

Well, since file magic numbers are different from filesystem magic numbers,
and a Windows box is probably never going to be able to mount an ext4
filesystem(footnote *), it's probably mostly irrelevant.  And things like tar's magic number,
only matter to /usr/bin/tar and /usr/bin/file...

See /usr/share/misc/magic (or wherever your system's /bin/file command stashes
the file - that's the list of magic numbers that 'file' uses to do stuff like this:

% file /bin/bash ./bmap-1.0.20.tar.gz
/bin/bash:            ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=44318cacd823951e3359ae4b5dba06b5383df1f4, stripped
./bmap-1.0.20.tar.gz: gzip compressed data, last modified: Tue Jan 23 17:40:23 2007, from Unix

(blast from the past entry in there:

##------------------------------------------------------------------------------
## $File: alliant,v 1.7 2009/09/19 16:28:07 christos Exp $
## alliant:  file(1) magic for Alliant FX series a.out files
##
## If the FX series is the one that had a processor with a 68K-derived
## instruction set, the "short" should probably become "beshort" and the
## "long" should probably become "belong".
## If it's the i860-based one, they should probably become either the
## big-endian or little-endian versions, depending on the mode they ran
## the 860 in....
##
0	short		0420		0420 Alliant virtual executable
>2	short		&0x0020		common library
>16	long		>0		not stripped
0	short		0421		0421 Alliant compact executable
>2	short		&0x0020		common library
>16	long		>0		not stripped

The Alliant was a machine that I worked on at my previous job.  I'm coming
up on 29 years in my current job, it's unclear whether there are any Alliant
a.out files to be found anyplace..

Though the fact that it also knows about gzip files created on TOPS-20 is
pretty trippy too....

(footnote *) It's been attempted, and the filesystem gods did heavily smite the programmer
for their arrogance. :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180117/d4cd0d51/attachment.sig>

About File systems magic numbers

[inventsekar]

Thanks Manish, Jim..

Thanks Valdis, .. learnt good deal. I will check further regarding the
file&tar example and revert back if further queries. Thanks again.


Best regards
Sekar

On Jan 18, 2018 2:38 AM, <valdis.kletnieks@vt.edu> wrote:

> On Wed, 17 Jan 2018 21:35:13 +0530, inventsekar said:
>
> > On the header file, it defines the fs magic number as
> > #define UX_MAGIC 0x58494e55
> >
> > 1. which means, will it assign this magic number to ALL files created on
> > this sample fs? That can't be so. Otherwise, why we need to define a
> magic
> > number?
>
> That's not a file magic number, that's a filesystem superblock magic
> number.
>
> That appears just once in the filesystem, usually in/near the superblock.
> It's only
> reason for existence is so things like fsck can read in the superblock,
> and do a:
>
>         if (superblock.magic != UX_MAGIC) {
>                 printf("I see no UX filesystem here!\n);
>                 exit(1);
>         }
>
> > 2. I assume, magic number will be assigned to files by the tools which we
> > use to create the files(like vi, cat, etc). Is this correct?
>
> No, it's scribbled into the superblock by mkfs
>
> > 3. It seems, windows files also have the magic numbers. let's say I copy
> > some files from windows to Linux. We should have the common magic numbers
> > between different Operating systems, so there won't be confusions. I
> mean,
> > is there any standards common for all os's files magic numbers?!?
>
> Well, since file magic numbers are different from filesystem magic numbers,
> and a Windows box is probably never going to be able to mount an ext4
> filesystem(footnote *), it's probably mostly irrelevant.  And things like
> tar's magic number,
> only matter to /usr/bin/tar and /usr/bin/file...
>
> See /usr/share/misc/magic (or wherever your system's /bin/file command
> stashes
> the file - that's the list of magic numbers that 'file' uses to do stuff
> like this:
>
> % file /bin/bash ./bmap-1.0.20.tar.gz
> /bin/bash:            ELF 64-bit LSB shared object, x86-64, version 1
> (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for
> GNU/Linux 3.2.0, BuildID[sha1]=44318cacd823951e3359ae4b5dba06b5383df1f4,
> stripped
> ./bmap-1.0.20.tar.gz: gzip compressed data, last modified: Tue Jan 23
> 17:40:23 2007, from Unix
>
> (blast from the past entry in there:
>
> ##----------------------------------------------------------
> --------------------
> ## $File: alliant,v 1.7 2009/09/19 16:28:07 christos Exp $
> ## alliant:  file(1) magic for Alliant FX series a.out files
> ##
> ## If the FX series is the one that had a processor with a 68K-derived
> ## instruction set, the "short" should probably become "beshort" and the
> ## "long" should probably become "belong".
> ## If it's the i860-based one, they should probably become either the
> ## big-endian or little-endian versions, depending on the mode they ran
> ## the 860 in....
> ##
> 0       short           0420            0420 Alliant virtual executable
> >2      short           &0x0020         common library
> >16     long            >0              not stripped
> 0       short           0421            0421 Alliant compact executable
> >2      short           &0x0020         common library
> >16     long            >0              not stripped
>
> The Alliant was a machine that I worked on at my previous job.  I'm coming
> up on 29 years in my current job, it's unclear whether there are any
> Alliant
> a.out files to be found anyplace..
>
> Though the fact that it also knows about gzip files created on TOPS-20 is
> pretty trippy too....
>
> (footnote *) It's been attempted, and the filesystem gods did heavily
> smite the programmer
> for their arrogance. :)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180119/e70469a5/attachment.html>