ARM imx.6 SATA speed regression

ARM imx.6 SATA speed regression

[Dan MacDonald]

Hi LKML

I recently got a SABRE Lite, iMX.6q based SBC. I bought this board
over the RPi3, for example, because I wanted SATA and open source
graphics drivers via etnaviv / armada.

Under Arch Linux ARM with Linux 4.9.0, hdparm said I was getting an
average of ~120 MB/s buffered disk read speed. After upgrading to
Linux 4.9.8 buffered read speed is just under 90 MB/s. That's a
notable hit! I both cases I was running the stock ALARM armv7h kernel
package.

Might any of you Linux ARM devs know what has changed since 4.9.0 that
could cause such a big decrease in SATA IO?

Thanks

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

Hi all

I'm still running the same 4.9.8 Arch kernel I was a few days ago and
I haven't done any updates but I decided to run hdparm again, a few
times actually, and now I'm getting 120/125 MB/s again on my SABRE
lite, as I was under 4.9.0.

When I last ran hdparm I ran it a few times and all the results were
all around 90 MB/s. I have no idea what caused those results to be so
far below the norm.

If I discover what the cause was I'll let you know but otherwise I'm
sorry about the noise!

Dan

On Wed, Feb 8, 2017 at 6:38 PM, Dan MacDonald <allcoms@gmail.com> wrote:
> Hi Gary
>
> I'll hopefully get to try some different kernels tonight or tomorrow
> night, otherwise it may have to wait until the weekend.
>
> As for bisecting and cross-compiling (which I won't have time to try
> again until this weekend), maybe not having cross-compilation working
> won't be such an obstacle provided I dont have to rebuild every kernel
> source file each time. I'm guessing I'll have to try approx 7 to 10
> kernel builds before the commit to blame is found, right? I've not
> built a kernel on my SABRE Lite yet but I recall building the generic
> armv7h kernel under Arch took approx 12 hours on my Banana Pi, which
> isn't that much slower than that the SL and has the same amount of RAM
> so at least the first build will take a smiliar amount of time.
>
> I'll update you all as soon as I have at least tried the various Arch
> kernel packages to narrow it down more.
>
> Thanks
>
> On Wed, Feb 8, 2017 at 5:48 PM, Gary Bisson
> <gary.bisson@boundarydevices.com> wrote:
>> Hi Dan,
>>
>> On Tue, Feb 07, 2017 at 05:02:40PM +0000, Dan MacDonald wrote:
>>> Thankfully there is an archive of older Arch ARM packages which should
>>> reduce the amount amount of bisection / compiling I may need to do and
>>> they have packages for most stable kernels between 4.9.0 and 4.9.8
>>>
>>> http://tardis.tiny-vps.com/aarm/packages/l/linux-armv7/
>>
>> How did that go?
>>
>> I've tried 'hdparm -Tt' on several kernel version and couldn't confirm
>> your obervations:
>>
>> 4.9.0:
>> Timing buffer-cache reads:   366 MB in 0.51 seconds = 733382 kB/s
>> Timing buffered disk reads:  221 MB in 3.00 seconds = 75411 kB/s
>>
>> 4.9.8:
>> Timing buffer-cache reads:   358 MB in 0.51 seconds = 717720 kB/s
>> Timing buffered disk reads:  221 MB in 3.00 seconds = 75244 kB/s
>>
>> 4.10-rc7:
>> Timing buffer-cache reads:   360 MB in 0.50 seconds = 723576 kB/s
>> Timing buffered disk reads:  244 MB in 3.00 seconds = 83024 kB/s
>>
>> Are you sure rolling back to 4.9.0 improves the perf in your case?
>>
>> Regards,
>> Gary

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

Hi Gary

I'll hopefully get to try some different kernels tonight or tomorrow
night, otherwise it may have to wait until the weekend.

As for bisecting and cross-compiling (which I won't have time to try
again until this weekend), maybe not having cross-compilation working
won't be such an obstacle provided I dont have to rebuild every kernel
source file each time. I'm guessing I'll have to try approx 7 to 10
kernel builds before the commit to blame is found, right? I've not
built a kernel on my SABRE Lite yet but I recall building the generic
armv7h kernel under Arch took approx 12 hours on my Banana Pi, which
isn't that much slower than that the SL and has the same amount of RAM
so at least the first build will take a smiliar amount of time.

I'll update you all as soon as I have at least tried the various Arch
kernel packages to narrow it down more.

Thanks

On Wed, Feb 8, 2017 at 5:48 PM, Gary Bisson
<gary.bisson@boundarydevices.com> wrote:
> Hi Dan,
>
> On Tue, Feb 07, 2017 at 05:02:40PM +0000, Dan MacDonald wrote:
>> Thankfully there is an archive of older Arch ARM packages which should
>> reduce the amount amount of bisection / compiling I may need to do and
>> they have packages for most stable kernels between 4.9.0 and 4.9.8
>>
>> http://tardis.tiny-vps.com/aarm/packages/l/linux-armv7/
>
> How did that go?
>
> I've tried 'hdparm -Tt' on several kernel version and couldn't confirm
> your obervations:
>
> 4.9.0:
> Timing buffer-cache reads:   366 MB in 0.51 seconds = 733382 kB/s
> Timing buffered disk reads:  221 MB in 3.00 seconds = 75411 kB/s
>
> 4.9.8:
> Timing buffer-cache reads:   358 MB in 0.51 seconds = 717720 kB/s
> Timing buffered disk reads:  221 MB in 3.00 seconds = 75244 kB/s
>
> 4.10-rc7:
> Timing buffer-cache reads:   360 MB in 0.50 seconds = 723576 kB/s
> Timing buffered disk reads:  244 MB in 3.00 seconds = 83024 kB/s
>
> Are you sure rolling back to 4.9.0 improves the perf in your case?
>
> Regards,
> Gary

Re: ARM imx.6 SATA speed regression

[Gary Bisson]

Hi Dan,

On Tue, Feb 07, 2017 at 05:02:40PM +0000, Dan MacDonald wrote:
> Thankfully there is an archive of older Arch ARM packages which should
> reduce the amount amount of bisection / compiling I may need to do and
> they have packages for most stable kernels between 4.9.0 and 4.9.8
> 
> http://tardis.tiny-vps.com/aarm/packages/l/linux-armv7/

How did that go?

I've tried 'hdparm -Tt' on several kernel version and couldn't confirm
your obervations:

4.9.0:
Timing buffer-cache reads:   366 MB in 0.51 seconds = 733382 kB/s
Timing buffered disk reads:  221 MB in 3.00 seconds = 75411 kB/s

4.9.8:
Timing buffer-cache reads:   358 MB in 0.51 seconds = 717720 kB/s
Timing buffered disk reads:  221 MB in 3.00 seconds = 75244 kB/s

4.10-rc7:
Timing buffer-cache reads:   360 MB in 0.50 seconds = 723576 kB/s
Timing buffered disk reads:  244 MB in 3.00 seconds = 83024 kB/s

Are you sure rolling back to 4.9.0 improves the perf in your case?

Regards,
Gary

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

Thankfully there is an archive of older Arch ARM packages which should
reduce the amount amount of bisection / compiling I may need to do and
they have packages for most stable kernels between 4.9.0 and 4.9.8

http://tardis.tiny-vps.com/aarm/packages/l/linux-armv7/

On Tue, Feb 7, 2017 at 3:45 PM, Dan MacDonald <allcoms@gmail.com> wrote:
> This is one of the cross-compilation guides I tried and failed to get to work.
>
> https://archlinuxarm.org/wiki/Distcc_Cross-Compiling
>
> It's been a year or two since I last tried to get distcc working. I
> followed all those instructions but packages would only ever compile
> locally. I can't remember what other guides I tried but that one was
> the most directly relevant and most likely to work as I use Arch on
> all my machines, x86 and ARM.
>
> On Tue, Feb 7, 2017 at 3:18 PM, Dan MacDonald <allcoms@gmail.com> wrote:
>> Hi Fabio
>>
>> I've never used bisect before and I have thusfar failed to get ARM
>> cross-compilation to work so I'm a bit wary of having to spend several
>> days rebuilding Linux several times on my board.
>>
>> I think the first thing I need to do is to see what other archived
>> Arch kernel packages I can get - hopefully I can find packages for
>> 4.9.1, 4.9.2 etc. I would imagine that if I can find the stable
>> release post 4.9.0 where imx6 SATA speed took a nosedive, that will
>> make bisecting much faster.
>>
>> If anyone has an easy to follow, known working guide to setting up
>> cross-compilation from amd64 -> armv7h, that would be massively
>> helpful as none of the ones I have tried in the past have worked.
>>
>> Thanks
>>
>> On Tue, Feb 7, 2017 at 2:58 PM, Fabio Estevam <fabio.estevam@nxp.com> wrote:
>>> [Sorry for the top-post]
>>>
>>>
>>> Hi Dan,
>>>
>>>
>>> Could you run 'git bisect' between kernel 4.9.0 and 4.9.8 to understand
>>> where this performance regression come from?
>>>
>>> ________________________________
>>> From: Dan MacDonald <allcoms@gmail.com>
>>> Sent: Tuesday, February 7, 2017 12:31:10 PM
>>> To: linux-kernel@vger.kernel.org
>>> Cc: Gary Bisson; Fabio Estevam; shawnguo@kernel.org; tj@kernel.org
>>> Subject: Re: ARM imx.6 SATA speed regression
>>>
>>> (Fake) Reply to CC in relevant people.

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

This is one of the cross-compilation guides I tried and failed to get to work.

https://archlinuxarm.org/wiki/Distcc_Cross-Compiling

It's been a year or two since I last tried to get distcc working. I
followed all those instructions but packages would only ever compile
locally. I can't remember what other guides I tried but that one was
the most directly relevant and most likely to work as I use Arch on
all my machines, x86 and ARM.

On Tue, Feb 7, 2017 at 3:18 PM, Dan MacDonald <allcoms@gmail.com> wrote:
> Hi Fabio
>
> I've never used bisect before and I have thusfar failed to get ARM
> cross-compilation to work so I'm a bit wary of having to spend several
> days rebuilding Linux several times on my board.
>
> I think the first thing I need to do is to see what other archived
> Arch kernel packages I can get - hopefully I can find packages for
> 4.9.1, 4.9.2 etc. I would imagine that if I can find the stable
> release post 4.9.0 where imx6 SATA speed took a nosedive, that will
> make bisecting much faster.
>
> If anyone has an easy to follow, known working guide to setting up
> cross-compilation from amd64 -> armv7h, that would be massively
> helpful as none of the ones I have tried in the past have worked.
>
> Thanks
>
> On Tue, Feb 7, 2017 at 2:58 PM, Fabio Estevam <fabio.estevam@nxp.com> wrote:
>> [Sorry for the top-post]
>>
>>
>> Hi Dan,
>>
>>
>> Could you run 'git bisect' between kernel 4.9.0 and 4.9.8 to understand
>> where this performance regression come from?
>>
>> ________________________________
>> From: Dan MacDonald <allcoms@gmail.com>
>> Sent: Tuesday, February 7, 2017 12:31:10 PM
>> To: linux-kernel@vger.kernel.org
>> Cc: Gary Bisson; Fabio Estevam; shawnguo@kernel.org; tj@kernel.org
>> Subject: Re: ARM imx.6 SATA speed regression
>>
>> (Fake) Reply to CC in relevant people.

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

Hi Fabio

I've never used bisect before and I have thusfar failed to get ARM
cross-compilation to work so I'm a bit wary of having to spend several
days rebuilding Linux several times on my board.

I think the first thing I need to do is to see what other archived
Arch kernel packages I can get - hopefully I can find packages for
4.9.1, 4.9.2 etc. I would imagine that if I can find the stable
release post 4.9.0 where imx6 SATA speed took a nosedive, that will
make bisecting much faster.

If anyone has an easy to follow, known working guide to setting up
cross-compilation from amd64 -> armv7h, that would be massively
helpful as none of the ones I have tried in the past have worked.

Thanks

On Tue, Feb 7, 2017 at 2:58 PM, Fabio Estevam <fabio.estevam@nxp.com> wrote:
> [Sorry for the top-post]
>
>
> Hi Dan,
>
>
> Could you run 'git bisect' between kernel 4.9.0 and 4.9.8 to understand
> where this performance regression come from?
>
> ________________________________
> From: Dan MacDonald <allcoms@gmail.com>
> Sent: Tuesday, February 7, 2017 12:31:10 PM
> To: linux-kernel@vger.kernel.org
> Cc: Gary Bisson; Fabio Estevam; shawnguo@kernel.org; tj@kernel.org
> Subject: Re: ARM imx.6 SATA speed regression
>
> (Fake) Reply to CC in relevant people.

Re: ARM imx.6 SATA speed regression

[Dan MacDonald]

(Fake) Reply to CC in relevant people.