Probable bug in md with rdev->new_data_offset

Probable bug in md with rdev->new_data_offset

[Étienne Buira]

Hi all,

Please apologise if i hit the wrong list.

I searched a bit, but could not find bug report or commits that seemed
related, please apologise if i'm wrong here.

I was going to grow a raid6 array (that contained a spare), using this
command:
# mdadm --grow -n 7 /dev/mdx

But when doing so, i got a PAX message saying that a size overflow was
detected in super_1_sync on the decl new_offset. The array was then in
unusable state (presumably because some locks were held).

After printking the values for rdev->new_data_offset and
rdev->data_offset in the
if (rdev->new_data_offset != rdev->data_offset) { ...
block of super_1_sync, i found that new_data_offset (252928 in my case)
where smaller than data_offset (258048), thus, the substraction to
compute sb->new_data_offset yielded an insanely high value.

For all partitions this array is made of, mdadm -E /dev/sdxy reports a
data offset of 258048 sectors (the value of rdev->data_offset).

IMHO, it seems a good idea to put a BUG_ON or similar if
rdev->new_data_offset is smaller than rdev->data offset at this place,
but that would not address the real issue.

I could solve my problem by setting mdadm's backup-file= option.

Kernel version was Gentoo hardened v4.4.2.

Full PAX size overflow detection line:
size overflow detected in function super_1_sync drivers/md/md.c:1683
cicus.1522_314 min, count: 158, decl: new_offset; num: 0; context:
mdp_superblock_1

Call stack (without addresses):
dump_stack
report_size_overflow
super_1_sync
? sched_clock_cpu
md_update_sb
? account_entity_dequeue
? dequeue_task_fair
? mutex_lock
? bitmap_daemon_work
md_check_recovery
raid5d
? try_to_del_timer_sync
? del_timer_sync
md_thread
? wait_woken
? find_pers
kthread
? kthread_create_on_node
ret_from_fork
? kthread_create_on_node

I am not familiar with kernel coding, so i won't create a patch, but i'm
willing to give more information if needed to track this issue.

Regards.

Re: Probable bug in md with rdev->new_data_offset

[Phil Turmel]

On 03/28/2016 06:31 AM, Étienne Buira wrote:
> Hi all,
> 
> Please apologise if i hit the wrong list.

This is the right list. :-)

> I searched a bit, but could not find bug report or commits that seemed
> related, please apologise if i'm wrong here.
> 
> I was going to grow a raid6 array (that contained a spare), using this
> command:
> # mdadm --grow -n 7 /dev/mdx
> 
> But when doing so, i got a PAX message saying that a size overflow was
> detected in super_1_sync on the decl new_offset. The array was then in
> unusable state (presumably because some locks were held).
> 
> After printking the values for rdev->new_data_offset and
> rdev->data_offset in the
> if (rdev->new_data_offset != rdev->data_offset) { ...
> block of super_1_sync, i found that new_data_offset (252928 in my case)
> where smaller than data_offset (258048), thus, the substraction to
> compute sb->new_data_offset yielded an insanely high value.

Modern mdadm and kernels avoid the use of backup files by adjusting the
data offset.  The lowered offset you see is normal.

I suspect the grsecurity kernels haven't kept up with this.  If you can
reproduce a problem with a vanilla kernel, please report back here.
Otherwise you'll have to report to your kernel provider.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Probable bug in md with rdev->new_data_offset

[Étienne Buira]

Sorry, forgot to reply to list as well, resending for completeness.

On Mon, Mar 28, 2016 at 08:19:12AM -0400, Phil Turmel wrote:
> On 03/28/2016 06:31 AM, Étienne Buira wrote:

../..

> > After printking the values for rdev->new_data_offset and
> > rdev->data_offset in the
> > if (rdev->new_data_offset != rdev->data_offset) { ...
> > block of super_1_sync, i found that new_data_offset (252928 in my case)
> > where smaller than data_offset (258048), thus, the substraction to
> > compute sb->new_data_offset yielded an insanely high value.
> 
> Modern mdadm and kernels avoid the use of backup files by adjusting the
> data offset.  The lowered offset you see is normal.
> 
> I suspect the grsecurity kernels haven't kept up with this.  If you can
> reproduce a problem with a vanilla kernel, please report back here.
> Otherwise you'll have to report to your kernel provider.
> 
> Phil

Hi,

Thank you for the answer.

I tried to reproduce the case with vanilla 4.4.6, but couldn't enter the
above said 'if', so i'm giving up on this topic.

However, i'm still surprised that sb->new_offset gets assigned a
'negative' (well, high, because it is computed unsigned) value.

Regards.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Probable bug in md with rdev->new_data_offset

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On Tue, Mar 29 2016, Étienne Buira wrote:

> Sorry, forgot to reply to list as well, resending for completeness.
>
> On Mon, Mar 28, 2016 at 08:19:12AM -0400, Phil Turmel wrote:
>> On 03/28/2016 06:31 AM, Étienne Buira wrote:
>
> ../..
>
>> > After printking the values for rdev->new_data_offset and
>> > rdev->data_offset in the
>> > if (rdev->new_data_offset != rdev->data_offset) { ...
>> > block of super_1_sync, i found that new_data_offset (252928 in my case)
>> > where smaller than data_offset (258048), thus, the substraction to
>> > compute sb->new_data_offset yielded an insanely high value.
>> 
>> Modern mdadm and kernels avoid the use of backup files by adjusting the
>> data offset.  The lowered offset you see is normal.
>> 
>> I suspect the grsecurity kernels haven't kept up with this.  If you can
>> reproduce a problem with a vanilla kernel, please report back here.
>> Otherwise you'll have to report to your kernel provider.
>> 
>> Phil
>
> Hi,
>
> Thank you for the answer.
>
> I tried to reproduce the case with vanilla 4.4.6, but couldn't enter the
> above said 'if', so i'm giving up on this topic.
>
> However, i'm still surprised that sb->new_offset gets assigned a
> 'negative' (well, high, because it is computed unsigned) value.
>

I guess sb->new_offset should be called sb->delta_offset.  If you look
in md_p.h you will see:

	__le32  new_offset;	/* signed number to add to data_offset in new
				 * layout.  0 == no-change.  This can be
				 * different on each device in the array.
				 */

which goes some way to explaining the situation.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Probable bug in md with rdev->new_data_offset

[Étienne Buira]

On Fri, Apr 01, 2016 at 04:31:28PM +1100, NeilBrown wrote:
> On Tue, Mar 29 2016, Étienne Buira wrote:

../..

> 
> I guess sb->new_offset should be called sb->delta_offset.  If you look
> in md_p.h you will see:
> 
> 	__le32  new_offset;	/* signed number to add to data_offset in new
> 				 * layout.  0 == no-change.  This can be
> 				 * different on each device in the array.
> 				 */
> 
> which goes some way to explaining the situation.
> 
> NeilBrown

Hi Neil,

Thank you for the explanation, i have also looked at the sb loading
counterpart, and indeed, PAX did catch a false positive here.

For the record, the bug have been reported on Gentoo bugtracker [1].

[1] https://bugs.gentoo.org/show_bug.cgi?id=578502

Regards.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html