Intel fakeraid working?

Intel fakeraid working?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I was trying to use mdadm to activate an intel fakeraid last night and mdadm insisted on activating it as read-only.  I could not get it to switch to read-wrte.  Today I came across this bug:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/972960

It seems that the kernel OOPSed here:

void md_write_start(struct mddev *mddev, struct bio *bi)
{
	int did_change = 0;
	if (bio_data_dir(bi) != WRITE)
		return;

	BUG_ON(mddev->ro == 1);

It appears this is because it wanted to update the metadata, but the array was read-only, so I thought this might be related to what I was seeing, and there is a bug that is preventing the array from correctly going read-write.

Is the Intel fakeraid support actually working yet, or still in development?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPfO3QAAoJEJrBOlT6nu75IS4H/0nerXBIW6wnYrmZPbWZH+sr
kVTVPDAqLCvbUATicLdwsJULWwqJmsyWnXTwnm7Ssug2phmcnK8cDFzNiLY/Q3ez
A0YdMmVog74viMSfZTLVfeiPYnccuTYLeH/U3glJVJ2ISyoFQi0mH6Kd/1eEO153
4XazWr7pj/QFeBsPnYuVlojvlsHR6QwdjU3aAsObdCxWdITrnJLLRYC9S4cgqS9m
Mdh1A4dbzeyj5SrNDAX1McQD+SQDnrPR6r17Tftr9eMcLCp6mzaxYkUu+GE1r8jr
9DRqGqe0sAdNjOn4cFMRZVOaHxMjGUXxxn8tuDNS2/p99P6kdCvQO280PFHByog=
=H98J
-----END PGP SIGNATURE-----

Re: Intel fakeraid working?

[NeilBrown]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 04 Apr 2012 20:56:48 -0400 Phillip Susi <psusi@ubuntu.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I was trying to use mdadm to activate an intel fakeraid last night and mdadm insisted on activating it as read-only.  I could not get it to switch to read-wrte.  Today I came across this bug:
> 
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/972960

That bug is fixed by 

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=c744a65c1e2d59acc54333ce80a5b0702a98010b

and is not specific to Intel IMSM RAID.

(I personally don't like the term 'fakeraid' - there is nothing fake about
it.  Firmware RAID is a suitable term).

I believe Intel RAID works and it certainly works in my tests, but they are
fairly limited.  Intel do much broader testing and they seem to be reasonably
happy with it.

If the array was activated read-only, that suggests that 'mdmon' didn't run.
mdmon is a daemon which monitors the array and updates the metadata as
needed.  until it is running it is not safe for the array to be writable.
However it should be started automatically by mdadm.

How are you attempting to start the array?

NeilBrown


> 
> It seems that the kernel OOPSed here:
> 
> void md_write_start(struct mddev *mddev, struct bio *bi)
> {
> 	int did_change = 0;
> 	if (bio_data_dir(bi) != WRITE)
> 		return;
> 
> 	BUG_ON(mddev->ro == 1);
> 
> It appears this is because it wanted to update the metadata, but the array was read-only, so I thought this might be related to what I was seeing, and there is a bug that is preventing the array from correctly going read-write.
> 
> Is the Intel fakeraid support actually working yet, or still in development?
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQEcBAEBAgAGBQJPfO3QAAoJEJrBOlT6nu75IS4H/0nerXBIW6wnYrmZPbWZH+sr
> kVTVPDAqLCvbUATicLdwsJULWwqJmsyWnXTwnm7Ssug2phmcnK8cDFzNiLY/Q3ez
> A0YdMmVog74viMSfZTLVfeiPYnccuTYLeH/U3glJVJ2ISyoFQi0mH6Kd/1eEO153
> 4XazWr7pj/QFeBsPnYuVlojvlsHR6QwdjU3aAsObdCxWdITrnJLLRYC9S4cgqS9m
> Mdh1A4dbzeyj5SrNDAX1McQD+SQDnrPR6r17Tftr9eMcLCp6mzaxYkUu+GE1r8jr
> 9DRqGqe0sAdNjOn4cFMRZVOaHxMjGUXxxn8tuDNS2/p99P6kdCvQO280PFHByog=
> =H98J
> -----END PGP SIGNATURE-----
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iQIVAwUBT3zxrTnsnt1WYoG5AQJpCRAAuqqH17YGVY9VzLRoa2oQfEJNZ2XP4GFl
a+ICYLGtLHMGEkMJJcPKPMjmzkyU4F6hFUDWK7npASVeVKqWqlnmFV+Hs0JifEJv
aIeS47yLgjzQ5iluBaFkRXP8Dyb03RTZlBM9FVI+w/oXbmgU4h1Z8sBlOJnh3Pb1
4ET/wGioy0yQyZ3vi9VVj16ugKnQHYjnMShZoHvEd0zEySvFfy+5NSZY/kJF6ElM
+ZHQQc1xB83zSQ1sZ0P6meCIS3zbwNnRl5yUckgXI5yMXxBjP/QFbmu1Gd9M8Q25
omPtNeFxDCxk/u5fakP24jotoFZcA+5+iAdySzXAiVErNxIXfBRLp2KVHjboEMLz
pbZTTFK7by1xQ59Dm+s1As9AgA3NLXhcPocFkJq9UpI9INddqqC2SMi7k1HsuOyf
VkTY8Kh3UkHWj0aqeHGaEHh3EavKe8jdm8ER/g0QXlAAcZpUm9GUnkTvEraTrBnN
Wjki3cnx5tfRbqQ+1jY7W1L9zdlOTpdwIGMH8PfPeptrYKD1OhrupTsTGP0SuT6P
X2Lsaeeh4etk9m7nSo5UG7cCMDNxVOIElwXDw9O38tPRNeuk4+UuFmKL0IzgtnX9
MBzNNPSVvQGoYF8Mo+rMaeTDcYSfEhTuA4JsnBL9WcScLsQqYfhQoEjKtnI/l/fR
LQev2W+HASQ=
=N8QX
-----END PGP SIGNATURE-----

Re: Intel fakeraid working?

[Phillip Susi]

On 4/4/2012 9:13 PM, NeilBrown wrote:
> If the array was activated read-only, that suggests that 'mdmon' didn't run.
> mdmon is a daemon which monitors the array and updates the metadata as
> needed.  until it is running it is not safe for the array to be writable.
> However it should be started automatically by mdadm.

That explains it... mdmon does not appear to have been added to the 
Ubuntu package.

> How are you attempting to start the array?

mdadm --activate --scan

Re: Intel fakeraid working?

[Phillip Susi]

On 4/4/2012 9:13 PM, NeilBrown wrote:
>
> That bug is fixed by
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=c744a65c1e2d59acc54333ce80a5b0702a98010b
>
> and is not specific to Intel IMSM RAID.

That commit appears to have to do with shutdown, but this did not happen 
during shutdown.  I suppose it happened as a result of the array being 
stuck in read only mode due to the missing mdmon.

Re: Intel fakeraid working?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Thu, 05 Apr 2012 14:23:16 -0400 Phillip Susi <psusi@ubuntu.com> wrote:

> On 4/4/2012 9:13 PM, NeilBrown wrote:
> >
> > That bug is fixed by
> >
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=c744a65c1e2d59acc54333ce80a5b0702a98010b
> >
> > and is not specific to Intel IMSM RAID.
> 
> That commit appears to have to do with shutdown, but this did not happen 
> during shutdown.  I suppose it happened as a result of the array being 
> stuck in read only mode due to the missing mdmon.
> 

Yes, you are right - I read too hastily.

Something wrote to the md device despite it being marked 'read-only'.
Some filesystems do that to replay their journal.  totally inexcusable
behaviour, but what can we do....

I'm surprised that would happen during installation though.
But certainly the array is marked read-only, and certainly something is
writing to it, and that is the real bug.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Intel fakeraid working?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

On Thu, 05 Apr 2012 14:20:53 -0400 Phillip Susi <psusi@ubuntu.com> wrote:

> On 4/4/2012 9:13 PM, NeilBrown wrote:
> > If the array was activated read-only, that suggests that 'mdmon' didn't run.
> > mdmon is a daemon which monitors the array and updates the metadata as
> > needed.  until it is running it is not safe for the array to be writable.
> > However it should be started automatically by mdadm.
> 
> That explains it... mdmon does not appear to have been added to the 
> Ubuntu package.

oops...  :-(

> 
> > How are you attempting to start the array?
> 
> mdadm --activate --scan
> 

I suspect you mean "mdadm --assemble --scan", but yes: that should work.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Intel fakeraid working?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/05/2012 07:24 PM, NeilBrown wrote:
> Something wrote to the md device despite it being marked 'read-only'.
> Some filesystems do that to replay their journal.  totally inexcusable
> behaviour, but what can we do....

I agree... read-only means read ONLY.

> I'm surprised that would happen during installation though.
> But certainly the array is marked read-only, and certainly something is
> writing to it, and that is the real bug.

If the block device is flagged as read only, then shouldn't it reject write attempts even if the fs or userspace issue them?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPfkJpAAoJEJrBOlT6nu75GUMH/jcgx+iMzZKICCKmwnU3IA9K
LTjC95EmOrq7Twt5e77D5rePTZqWZKzDyNEkiZ8hwOHvg+ohyKA7HA42DPIBpfT2
Fb5urHbhtVwLHNeusQPOIOX5psByEFjhlz/CgjuAaZjlsOu5VY6CXtdkPOdO65GC
XnOIlJpUAvUhvv9x9bFT1d4TjmcvDTt4hrErBUhHPMPXJGSY7/24G1wjgWLAUx3w
cR8YHChLAgJ9PAYHLCNINyTVQGdET9sLHqTSMjxrp2tryDh3IDBNGxUSRMzrPaJd
EprVL+3BmH0J9bHLuUfkr/q/zZrHxGF8F4ePi+lvXsNqPIKuXdN68KJgU9KeTL4=
=JG+f
-----END PGP SIGNATURE-----

Re: Intel fakeraid working?

[Jan Ceuleers]

On 05/04/12 20:20, Phillip Susi wrote:
> On 4/4/2012 9:13 PM, NeilBrown wrote:
>> If the array was activated read-only, that suggests that 'mdmon'
>> didn't run.
>> mdmon is a daemon which monitors the array and updates the metadata as
>> needed. until it is running it is not safe for the array to be writable.
>> However it should be started automatically by mdadm.
>
> That explains it... mdmon does not appear to have been added to the
> Ubuntu package.

https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/957494

Re: Intel fakeraid working?

[NeilBrown]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, 05 Apr 2012 21:10:07 -0400 Phillip Susi <psusi@ubuntu.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 04/05/2012 07:24 PM, NeilBrown wrote:
> > Something wrote to the md device despite it being marked 'read-only'.
> > Some filesystems do that to replay their journal.  totally inexcusable
> > behaviour, but what can we do....
> 
> I agree... read-only means read ONLY.
> 
> > I'm surprised that would happen during installation though.
> > But certainly the array is marked read-only, and certainly something is
> > writing to it, and that is the real bug.
> 
> If the block device is flagged as read only, then shouldn't it reject write attempts even if the fs or userspace issue them?
>

It will reject writes from user-space, and it will reject attempts to mount a
filesystem unless the filesystem is mounted "read-only".
But if a read-only mounted filesystem decides to write anyway (XFS, ext3,
ext4...) then the block layer doesn't stop it.

NeilBrown

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iQIVAwUBT4N0Ijnsnt1WYoG5AQI4DQ/9HF2EJMx2RLV8pAb6pOCY4YRcjGPomRpe
86MxPF0GtlFI+ZPZP7fIHmGxkDYg6Y3zW/sNDAXS7xHzt3gyHa9w91adv54BK79T
xQghdL+ZXS3NxdHWi04Ya20R44VGnVKuoyNdP8mgddZi4ABoy3eaQSUdM9uoJ/bn
79JKPlUfbwTSSx0ub20ZINZU/KjIBxxyyp5bT3RxNqw795G6XIx6jpHMgQNomMSL
TpywEky2gGjnC/6T8mM80EGKv+Fx8IMteWb7StIuMu0xNPyfrwkUGVKHAFs7PbAT
CTkV/vqkNO/DHeDiXSmepH5tQAV0BsNUFRo4DzFW5id0ANaSFCj0wPXQjPAkfkOl
XVBxVFm8riVQtXYYAjFqj4aAUkkovp3bpCwPgy5HajmxBkTOAt9dRyy1Uzp5VRNh
y942F4xi6uiTG0H3NmZaY7nvDVYkLVAn6wMrzm4UAbgxmvIeIgKU773FEt0A1HRy
oXE+gVLT/V2pF1+S0X5s3lp6cDpUbd5M8hCsN9rhmWovPV6mEwfWk857+eVvmiRX
lUxgC56HumMf0vvTsmJOS5GOgnHxADV7ugDYIdau4wpulghL/pgt6wOrCYBtzbA1
moqaD0BSlQBBwXRL/j8qdH3dm5ResNxnQpPURYU7aZX5T0mQ6bQ1stjXZ9qOmw3A
zhSC8abbZpk=
=HgL3
-----END PGP SIGNATURE-----

Re: Intel fakeraid working?

[Phillip Susi]

On 4/9/2012 7:43 PM, NeilBrown wrote:
> It will reject writes from user-space, and it will reject attempts to mount a
> filesystem unless the filesystem is mounted "read-only".
> But if a read-only mounted filesystem decides to write anyway (XFS, ext3,
> ext4...) then the block layer doesn't stop it.

How does that work?  How does the block layer know or care what mount 
flags are used?  My understanding is that setting the block layer 
read-only flag with blockdev --setro actually causes the write bios to 
be rejected, thus preventing ext[34] from playing back the journal. 
Ubuntu has been using this to prevent accidental damage by read-only 
mounts since this abhorrent behavior was discovered.  I would think that 
md should work the same way.

Re: Intel fakeraid working?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

On Tue, 10 Apr 2012 09:14:58 -0400 Phillip Susi <psusi@ubuntu.com> wrote:

> On 4/9/2012 7:43 PM, NeilBrown wrote:
> > It will reject writes from user-space, and it will reject attempts to mount a
> > filesystem unless the filesystem is mounted "read-only".
> > But if a read-only mounted filesystem decides to write anyway (XFS, ext3,
> > ext4...) then the block layer doesn't stop it.
> 
> How does that work?  How does the block layer know or care what mount 
> flags are used?  My understanding is that setting the block layer 
> read-only flag with blockdev --setro actually causes the write bios to 
> be rejected, thus preventing ext[34] from playing back the journal. 
> Ubuntu has been using this to prevent accidental damage by read-only 
> mounts since this abhorrent behavior was discovered.  I would think that 
> md should work the same way.
> 

It's actually a long time since I looked at this in detail, so I've had
another hunt around the code to see what the current state really is.
I've only looked at ext3, not other filesystems, but it should be fairly
representative.

ext3 does appear to take care not to write anything if the device is marked
readonly - so maybe I've been doing it a dis-service there.  However there
have been bugs.  The most recent was fixed about 18 months ago:

commit 31d710a7bd42f0d89e30d53bdaad427c5f191d0d
Author: Maciej 305273enczykowski <zenczykowski@gmail.com>
Date:   Sun Sep 26 12:38:28 2010 +0000

    ext3: don't update sb journal_devnum when RO dev
    
    An ext3 filesystem on a read-only device, with an external journal
    which is at a different device number then recorded in the superblock
    will fail to honor the read-only setting of the device and trigger
    a superblock update (write).
    
    For example:
      - ext3 on a software raid which is in read-only mode
      - external journal on a read-write device which has changed device num
      - attempt to mount with -o journal_dev=<new_number>
      - hits BUG_ON(mddev->ro = 1) in md.c
    
    Cc: Theodore Ts'o <tytso@mit.edu>
    Signed-off-by: Maciej 305273enczykowski <zenczykowski@gmail.com>
    Signed-off-by: Jan Kara <jack@suse.cz>

This fix was in 2.6.38.  I don't know if was backported at all.

The read-only flag does not cause the block-layer to do, or not do, anything.
It is merely information that is made available to the filesystem (or the
"block_dev" pseudo-filesystem that presents /dev/sda or whatever and maps
read/write requests directly to bios which get sent down).

It is up the the filesystem to honour the read-only setting.  Maybe this is
poor design: I haven't thought about it much.  But that is the way it is.

When ext3 sees that it needs to replay the journal it will check if the
device is read-only. If it is, it will failed with EROFS.  So you cannot
mount an inconsistent filesystem from a read-only device.

md does set the same flag that "blockdev --setro" sets.  If some filesystem
writes anyway, then it is definitely a bug and should be fixed.

So it looks like everybody is doing the "right" thing (allowing for
occasional bugs here and there), and your real problem is just that Ubuntu
didn't package mdmon.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Intel fakeraid working?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/22/2012 10:25 PM, NeilBrown wrote:
> md does set the same flag that "blockdev --setro" sets.  If some filesystem
> writes anyway, then it is definitely a bug and should be fixed.

It sounds like that's what happened in this case.

> So it looks like everybody is doing the "right" thing (allowing for
> occasional bugs here and there), and your real problem is just that Ubuntu
> didn't package mdmon.

The lack of mdmon certainly is a problem, but triggering that BUG_ON when it is missing seems to point to a bug somewhere in the kernel.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPlNVJAAoJEJrBOlT6nu75VmQIAKTl4VPgadyeV1xpzJzFmPKP
YX9MZf8cZgcrmy1qRZrJLGa9Yfd6A8L0sp9Y4kzUM7pvPsmxG6uDr+iBsxx6MkJZ
cBbpIxuIiB1zwOIR6YcvyCG2htZxqS11ZFzZL+fIFTI9OXMgdBbHJoMRm1S6XdE4
erKhYhrAD31tKcHV1RIOFSAf8GMUsn5iUzdO6YpBefVjmCodJMMeaUyxIoMLrIC1
jWzS8Iu30L0fn2kyOWLaLvkupVTe1TG4hThWrdHkb/sUCEmcuSp2YJ7uMgL7Jwn+
A4hr+yOXnKZmTnYlJcL4fg4xvavcaKqirZYYmcdv5OmIHwcfYORdkHAP2m5G4pQ=
=w81A
-----END PGP SIGNATURE-----