mmc: core: complete/wait_for_completion performance

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hi,

I started the discussion on this mailing list in another thread [1],
but I'd like to move it to a new thread, because it might be mmc
specific.

The issue is that I am noticed low wifi network throughput on an i.MX28
board with the mainline kernel (v4.7.10, about 6 Mbps) compared to the
vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi chip is
attached using the SDIO interface.

I started investigation where the bottleneck in the mainline
kernel?might come from. Therefore I checked that the configs and
settings for the interfaces and drivers are the same. They are.

For comparing both kernel version I measured the latency for the
different parts in the code flow by adding some GPIOs to toggle and
useing an osci.

It turns out, that the time needed do pass a mmc request from the wifi
driver to the dma and waiting for the response of the dma is almost the
same for both kernel versions. So the path from the wifi driver to the
dma driver is not the issue.

However, the major difference was the time needed to signal the
completion of the request back to the mmc core driver.

For mainline, the time elapsed?from calling complete() in
mmc_wait_done() [2] in the mmc core driver until actually receiving the
completion signal in mmc_wait_for_req_done() [3] is about 120 us. 

Whereas, for the vendor kernel,?the elapsed time for this notifcation
is about 100 us less!

I wonder why the notification takes so much longer in the mainline
kernel than in the vendor kernel? Where should I start looking at to
eliminate this bottleneck? Is it a scheduling issue?

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-October/
461137.html
[2] http://lxr.free-electrons.com/source/drivers/mmc/core/core.c#L386
[3] http://lxr.free-electrons.com/source/drivers/mmc/core/core.c#L492

Best regards,
J?rg Krause

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016 um 13:27
> geschrieben:
> 
> 
> Hi,
> 
> I started the discussion on this mailing list in another thread [1],
> but I'd like to move it to a new thread, because it might be mmc
> specific.
> 
> The issue is that I am noticed low wifi network throughput on an i.MX28
> board with the mainline kernel (v4.7.10, about 6 Mbps) compared to the
> vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi chip is
> attached using the SDIO interface.
> 
> I started investigation where the bottleneck in the mainline
> kernel?might come from. Therefore I checked that the configs and
> settings for the interfaces and drivers are the same. They are.

so you're not using the mxs_defconfig settings anymore? Better provide your
settings because there are differences between the vendor defconfig and mainline
defconfig.

Examples:

Vendor updater defconfig (2.6.35) - [1]

CONFIG_PREEMPTY_NONE=y
CONFIG_DEFAULT_IOSCHED="noop"

Vendor defconfig (2.6.35) - [2]

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT=y
CONFIG_DEFAULT_IOSCHED="cfq"

Mainline defconfig (4.8) - [3]

CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_DEFAULT_IOSCHED="noop"

[1] -
http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/tree/arch/arm/configs/imx28evk_updater_defconfig?h=imx_2.6.35_maintain
[2] -
http://git.freescale.com/git/cgit.cgi/imx/linux-2.6-imx.git/tree/arch/arm/configs/imx28evk_defconfig?h=imx_2.6.35_maintain
[3] - http://lxr.free-electrons.com/source/arch/arm/configs/mxs_defconfig?v=4.8

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hi Stefan,

On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> Hi J?rg,
> 
> > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > um 13:27
> > geschrieben:
> > 
> > 
> > Hi,
> > 
> > I started the discussion on this mailing list in another thread
> > [1],
> > but I'd like to move it to a new thread, because it might be mmc
> > specific.
> > 
> > The issue is that I am noticed low wifi network throughput on an
> > i.MX28
> > board with the mainline kernel (v4.7.10, about 6 Mbps) compared to
> > the
> > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi chip
> > is
> > attached using the SDIO interface.
> > 
> > I started investigation where the bottleneck in the mainline
> > kernel?might come from. Therefore I checked that the configs and
> > settings for the interfaces and drivers are the same. They are.
> 
> so you're not using the mxs_defconfig settings anymore?

No, I changed the settings.

> Better provide your settings because there are differences between
> the vendor defconfig and mainline defconfig.

The configs I have for the vendor and mainline kernel are pretty much
the same for the parts you showed as an example:

Vendor kernel config:

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT=y
CONFIG_HZ=100
CONFIG_DEFAULT_IOSCHED="cfq"

Mainline kernel config:

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_IDLE=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_PREEMPT=y
CONFIG_HZ=100
CONFIG_DEFAULT_IOSCHED="cfq"

J?rg

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

> J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016 um 15:42
> geschrieben:
> 
> 
> Hi Stefan,
> 
> On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > 
> > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > > um 13:27
> > > geschrieben:
> > > 
> > > 
> > > Hi,
> > > 
> > > I started the discussion on this mailing list in another thread
> > > [1],
> > > but I'd like to move it to a new thread, because it might be mmc
> > > specific.
> > > 
> > > The issue is that I am noticed low wifi network throughput on an
> > > i.MX28
> > > board with the mainline kernel (v4.7.10, about 6 Mbps) compared to
> > > the
> > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi chip
> > > is
> > > attached using the SDIO interface.
> > > 
> > > I started investigation where the bottleneck in the mainline
> > > kernel?might come from. Therefore I checked that the configs and
> > > settings for the interfaces and drivers are the same. They are.
> > 
> > so you're not using the mxs_defconfig settings anymore?
> 
> No, I changed the settings.
> 

What happens to performance to if you change the following settings to the same
like in mxs_defconfig?

CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_DEFAULT_IOSCHED="noop"

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > um 15:42
> > geschrieben:
> > 
> > 
> > Hi Stefan,
> > 
> > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > Hi J?rg,
> > > 
> > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > 2016
> > > > um 13:27
> > > > geschrieben:
> > > > 
> > > > 
> > > > Hi,
> > > > 
> > > > I started the discussion on this mailing list in another thread
> > > > [1],
> > > > but I'd like to move it to a new thread, because it might be
> > > > mmc
> > > > specific.
> > > > 
> > > > The issue is that I am noticed low wifi network throughput on
> > > > an
> > > > i.MX28
> > > > board with the mainline kernel (v4.7.10, about 6 Mbps) compared
> > > > to
> > > > the
> > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi
> > > > chip
> > > > is
> > > > attached using the SDIO interface.
> > > > 
> > > > I started investigation where the bottleneck in the mainline
> > > > kernel?might come from. Therefore I checked that the configs
> > > > and
> > > > settings for the interfaces and drivers are the same. They are.
> > > 
> > > so you're not using the mxs_defconfig settings anymore?
> > 
> > No, I changed the settings.
> > 
> 
> What happens to performance to if you change the following settings
> to the same
> like in mxs_defconfig?
> 
> CONFIG_PREEMPT_VOLUNTARY=y
> CONFIG_DEFAULT_IOSCHED="noop"

No much change at all. The time difference between complete() and
wait_for_complete() decreases in best case to 110 us, but also varies
to above 130 us.

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016 um 20:10
> geschrieben:
> 
> 
> On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > > um 15:42
> > > geschrieben:
> > > 
> > > 
> > > Hi Stefan,
> > > 
> > > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > > Hi J?rg,
> > > > 
> > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > > 2016
> > > > > um 13:27
> > > > > geschrieben:
> > > > > 
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > I started the discussion on this mailing list in another thread
> > > > > [1],
> > > > > but I'd like to move it to a new thread, because it might be
> > > > > mmc
> > > > > specific.
> > > > > 
> > > > > The issue is that I am noticed low wifi network throughput on
> > > > > an
> > > > > i.MX28
> > > > > board with the mainline kernel (v4.7.10, about 6 Mbps) compared
> > > > > to
> > > > > the
> > > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The wifi
> > > > > chip
> > > > > is
> > > > > attached using the SDIO interface.
> > > > > 
> > > > > I started investigation where the bottleneck in the mainline
> > > > > kernel?might come from. Therefore I checked that the configs
> > > > > and
> > > > > settings for the interfaces and drivers are the same. They are.
> > > > 
> > > > so you're not using the mxs_defconfig settings anymore?
> > > 
> > > No, I changed the settings.
> > > 
> > 
> > What happens to performance to if you change the following settings
> > to the same
> > like in mxs_defconfig?
> > 
> > CONFIG_PREEMPT_VOLUNTARY=y
> > CONFIG_DEFAULT_IOSCHED="noop"
> 
> No much change at all. The time difference between complete() and
> wait_for_complete() decreases in best case to 110 us, but also varies
> to above 130 us.

just a weird idea. Did you tried to add MMC_CAP_CMD23 into the caps [1]?

[1] - http://lxr.free-electrons.com/source/drivers/mmc/host/mxs-mmc.c?v=4.8#L642

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hit Stefan,

On Sat, 2016-11-26 at 20:10 +0100, Stefan Wahren wrote:
> Hi J?rg,
> 
> > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > um 20:10
> > geschrieben:
> > 
> > 
> > On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > 2016
> > > > um 15:42
> > > > geschrieben:
> > > > 
> > > > 
> > > > Hi Stefan,
> > > > 
> > > > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > > > Hi J?rg,
> > > > > 
> > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > November
> > > > > > 2016
> > > > > > um 13:27
> > > > > > geschrieben:
> > > > > > 
> > > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > I started the discussion on this mailing list in another
> > > > > > thread
> > > > > > [1],
> > > > > > but I'd like to move it to a new thread, because it might
> > > > > > be
> > > > > > mmc
> > > > > > specific.
> > > > > > 
> > > > > > The issue is that I am noticed low wifi network throughput
> > > > > > on
> > > > > > an
> > > > > > i.MX28
> > > > > > board with the mainline kernel (v4.7.10, about 6 Mbps)
> > > > > > compared
> > > > > > to
> > > > > > the
> > > > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The
> > > > > > wifi
> > > > > > chip
> > > > > > is
> > > > > > attached using the SDIO interface.
> > > > > > 
> > > > > > I started investigation where the bottleneck in the
> > > > > > mainline
> > > > > > kernel?might come from. Therefore I checked that the
> > > > > > configs
> > > > > > and
> > > > > > settings for the interfaces and drivers are the same. They
> > > > > > are.
> > > > > 
> > > > > so you're not using the mxs_defconfig settings anymore?
> > > > 
> > > > No, I changed the settings.
> > > > 
> > > 
> > > What happens to performance to if you change the following
> > > settings
> > > to the same
> > > like in mxs_defconfig?
> > > 
> > > CONFIG_PREEMPT_VOLUNTARY=y
> > > CONFIG_DEFAULT_IOSCHED="noop"
> > 
> > No much change at all. The time difference between complete() and
> > wait_for_complete() decreases in best case to 110 us, but also
> > varies
> > to above 130 us.
> 
> just a weird idea. Did you tried to add MMC_CAP_CMD23 into the caps
> [1]?
> 
> [1] - http://lxr.free-electrons.com/source/drivers/mmc/host/mxs-mmc.c
> ?v=4.8#L642

I tried, but it did not improved the timing or throughput. However,
many thanks for the input.

J?rg

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 7. Dezember 2016 um 08:32
> geschrieben:
> 
> 
> Hit Stefan,
> 
> On Sat, 2016-11-26 at 20:10 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > 
> > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November 2016
> > > um 20:10
> > > geschrieben:
> > > 
> > > 
> > > On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > > 2016
> > > > > um 15:42
> > > > > geschrieben:
> > > > > 
> > > > > 
> > > > > Hi Stefan,
> > > > > 
> > > > > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > > > > Hi J?rg,
> > > > > > 
> > > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > > November
> > > > > > > 2016
> > > > > > > um 13:27
> > > > > > > geschrieben:
> > > > > > > 
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I started the discussion on this mailing list in another
> > > > > > > thread
> > > > > > > [1],
> > > > > > > but I'd like to move it to a new thread, because it might
> > > > > > > be
> > > > > > > mmc
> > > > > > > specific.
> > > > > > > 
> > > > > > > The issue is that I am noticed low wifi network throughput
> > > > > > > on
> > > > > > > an
> > > > > > > i.MX28
> > > > > > > board with the mainline kernel (v4.7.10, about 6 Mbps)
> > > > > > > compared
> > > > > > > to
> > > > > > > the
> > > > > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The
> > > > > > > wifi
> > > > > > > chip
> > > > > > > is
> > > > > > > attached using the SDIO interface.
> > > > > > > 
> > > > > > > I started investigation where the bottleneck in the
> > > > > > > mainline
> > > > > > > kernel?might come from. Therefore I checked that the
> > > > > > > configs
> > > > > > > and
> > > > > > > settings for the interfaces and drivers are the same. They
> > > > > > > are.
> > > > > > 
> > > > > > so you're not using the mxs_defconfig settings anymore?
> > > > > 
> > > > > No, I changed the settings.
> > > > > 
> > > > 
> > > > What happens to performance to if you change the following
> > > > settings
> > > > to the same
> > > > like in mxs_defconfig?
> > > > 
> > > > CONFIG_PREEMPT_VOLUNTARY=y
> > > > CONFIG_DEFAULT_IOSCHED="noop"
> > > 
> > > No much change at all. The time difference between complete() and
> > > wait_for_complete() decreases in best case to 110 us, but also
> > > varies
> > > to above 130 us.
> > 
> > just a weird idea. Did you tried to add MMC_CAP_CMD23 into the caps
> > [1]?
> > 
> > [1] - http://lxr.free-electrons.com/source/drivers/mmc/host/mxs-mmc.c
> > ?v=4.8#L642
> 
> I tried, but it did not improved the timing or throughput. However,
> many thanks for the input.
> 
> J?rg

did you try cyclictest [1]?

Beside the time for a request the amount of requests for the complete iperf test
would we interesting. Maybe there are retries.

I'm still interested in your PIO mode patches for mxs-mmc even without clean up.

[1] - https://git.kernel.org/cgit/utils/rt-tests/rt-tests.git/

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hi Stefan,

On Wed, 2016-12-07 at 20:23 +0100, Stefan Wahren wrote:
> Hi J?rg,
> 
> > J?rg Krause <joerg.krause@embedded.rocks> hat am 7. Dezember 2016
> > um 08:32
> > geschrieben:
> > 
> > 
> > Hit Stefan,
> > 
> > On Sat, 2016-11-26 at 20:10 +0100, Stefan Wahren wrote:
> > > Hi J?rg,
> > > 
> > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > 2016
> > > > um 20:10
> > > > geschrieben:
> > > > 
> > > > 
> > > > On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > November
> > > > > > 2016
> > > > > > um 15:42
> > > > > > geschrieben:
> > > > > > 
> > > > > > 
> > > > > > Hi Stefan,
> > > > > > 
> > > > > > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > > > > > Hi J?rg,
> > > > > > > 
> > > > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > > > November
> > > > > > > > 2016
> > > > > > > > um 13:27
> > > > > > > > geschrieben:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > I started the discussion on this mailing list in
> > > > > > > > another
> > > > > > > > thread
> > > > > > > > [1],
> > > > > > > > but I'd like to move it to a new thread, because it
> > > > > > > > might
> > > > > > > > be
> > > > > > > > mmc
> > > > > > > > specific.
> > > > > > > > 
> > > > > > > > The issue is that I am noticed low wifi network
> > > > > > > > throughput
> > > > > > > > on
> > > > > > > > an
> > > > > > > > i.MX28
> > > > > > > > board with the mainline kernel (v4.7.10, about 6 Mbps)
> > > > > > > > compared
> > > > > > > > to
> > > > > > > > the
> > > > > > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The
> > > > > > > > wifi
> > > > > > > > chip
> > > > > > > > is
> > > > > > > > attached using the SDIO interface.
> > > > > > > > 
> > > > > > > > I started investigation where the bottleneck in the
> > > > > > > > mainline
> > > > > > > > kernel?might come from. Therefore I checked that the
> > > > > > > > configs
> > > > > > > > and
> > > > > > > > settings for the interfaces and drivers are the same.
> > > > > > > > They
> > > > > > > > are.
> > > > > > > 
> > > > > > > so you're not using the mxs_defconfig settings anymore?
> > > > > > 
> > > > > > No, I changed the settings.
> > > > > > 
> > > > > 
> > > > > What happens to performance to if you change the following
> > > > > settings
> > > > > to the same
> > > > > like in mxs_defconfig?
> > > > > 
> > > > > CONFIG_PREEMPT_VOLUNTARY=y
> > > > > CONFIG_DEFAULT_IOSCHED="noop"
> > > > 
> > > > No much change at all. The time difference between complete()
> > > > and
> > > > wait_for_complete() decreases in best case to 110 us, but also
> > > > varies
> > > > to above 130 us.
> > > 
> > > just a weird idea. Did you tried to add MMC_CAP_CMD23 into the
> > > caps
> > > [1]?
> > > 
> > > [1] - http://lxr.free-electrons.com/source/drivers/mmc/host/mxs-m
> > > mc.c
> > > ?v=4.8#L642
> > 
> > I tried, but it did not improved the timing or throughput. However,
> > many thanks for the input.
> > 
> > J?rg
> 
> did you try cyclictest [1]?

Not yet. Not sure what to measure and which values to compare here.

> 
> Beside the time for a request the amount of requests for the complete
> iperf test
> would we interesting. Maybe there are retries.
> 
> I'm still interested in your PIO mode patches for mxs-mmc even
> without clean up.

Actually, the patch does not implement a PIO mode, but drops DMA and
uses polling instead. I've attached the patch.

> [1] - https://git.kernel.org/cgit/utils/rt-tests/rt-tests.git/

Best regards
J?rg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-mmc-mxs-mmc-drop-DMA-and-use-polling-mode.patch
Type: text/x-patch
Size: 7570 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20161214/c2c74830/attachment-0001.bin>

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 14. Dezember 2016 um 10:03 geschrieben:
> 
> 
> Hi Stefan,
> 
> On Wed, 2016-12-07 at 20:23 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > 
> > > J?rg Krause <joerg.krause@embedded.rocks> hat am 7. Dezember 2016
> > > um 08:32
> > > geschrieben:
> > > 
> > > 
> > > Hit Stefan,
> > > 
> > > On Sat, 2016-11-26 at 20:10 +0100, Stefan Wahren wrote:
> > > > Hi J?rg,
> > > > 
> > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20. November
> > > > > 2016
> > > > > um 20:10
> > > > > geschrieben:
> > > > > 
> > > > > 
> > > > > On Sun, 2016-11-20 at 16:44 +0100, Stefan Wahren wrote:
> > > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > > November
> > > > > > > 2016
> > > > > > > um 15:42
> > > > > > > geschrieben:
> > > > > > > 
> > > > > > > 
> > > > > > > Hi Stefan,
> > > > > > > 
> > > > > > > On Sun, 2016-11-20 at 14:28 +0100, Stefan Wahren wrote:
> > > > > > > > Hi J?rg,
> > > > > > > > 
> > > > > > > > > J?rg Krause <joerg.krause@embedded.rocks> hat am 20.
> > > > > > > > > November
> > > > > > > > > 2016
> > > > > > > > > um 13:27
> > > > > > > > > geschrieben:
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > I started the discussion on this mailing list in
> > > > > > > > > another
> > > > > > > > > thread
> > > > > > > > > [1],
> > > > > > > > > but I'd like to move it to a new thread, because it
> > > > > > > > > might
> > > > > > > > > be
> > > > > > > > > mmc
> > > > > > > > > specific.
> > > > > > > > > 
> > > > > > > > > The issue is that I am noticed low wifi network
> > > > > > > > > throughput
> > > > > > > > > on
> > > > > > > > > an
> > > > > > > > > i.MX28
> > > > > > > > > board with the mainline kernel (v4.7.10, about 6 Mbps)
> > > > > > > > > compared
> > > > > > > > > to
> > > > > > > > > the
> > > > > > > > > vendor kernel (Freescale v2.6.35.3, about 20 Mbps). The
> > > > > > > > > wifi
> > > > > > > > > chip
> > > > > > > > > is
> > > > > > > > > attached using the SDIO interface.
> > > > > > > > > 
> > > > > > > > > I started investigation where the bottleneck in the
> > > > > > > > > mainline
> > > > > > > > > kernel might come from. Therefore I checked that the
> > > > > > > > > configs
> > > > > > > > > and
> > > > > > > > > settings for the interfaces and drivers are the same.
> > > > > > > > > They
> > > > > > > > > are.
> > > > > > > > 
> > > > > > > > so you're not using the mxs_defconfig settings anymore?
> > > > > > > 
> > > > > > > No, I changed the settings.
> > > > > > > 
> > > > > > 
> > > > > > What happens to performance to if you change the following
> > > > > > settings
> > > > > > to the same
> > > > > > like in mxs_defconfig?
> > > > > > 
> > > > > > CONFIG_PREEMPT_VOLUNTARY=y
> > > > > > CONFIG_DEFAULT_IOSCHED="noop"
> > > > > 
> > > > > No much change at all. The time difference between complete()
> > > > > and
> > > > > wait_for_complete() decreases in best case to 110 us, but also
> > > > > varies
> > > > > to above 130 us.
> > > > 
> > > > just a weird idea. Did you tried to add MMC_CAP_CMD23 into the
> > > > caps
> > > > [1]?
> > > > 
> > > > [1] - http://lxr.free-electrons.com/source/drivers/mmc/host/mxs-m
> > > > mc.c
> > > > ?v=4.8#L642
> > > 
> > > I tried, but it did not improved the timing or throughput. However,
> > > many thanks for the input.
> > > 
> > > J?rg
> > 
> > did you try cyclictest [1]?
> 
> Not yet. Not sure what to measure and which values to compare here.

i tought you have the vendor kernel and the mainline kernel available for your platform.

So you could compare the both kernels.

> 
> > 
> > Beside the time for a request the amount of requests for the complete
> > iperf test
> > would we interesting. Maybe there are retries.
> > 
> > I'm still interested in your PIO mode patches for mxs-mmc even
> > without clean up.
> 
> Actually, the patch does not implement a PIO mode, but drops DMA and
> uses polling instead. I've attached the patch.

Thanks. I applied it, but unfortunately this breaks SD card support for my Duckbill and the kernel isn't able to mount the rootfs:

[    2.267073] mxs-mmc 80010000.ssp: initialized
[    2.272624] mxs-mmc 80010000.ssp: AC command error 0xffffff92

> 
> > [1] - https://git.kernel.org/cgit/utils/rt-tests/rt-tests.git/
> 
> Best regards
> J?rg

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hi Stefan,

On Wed, 2016-12-14 at 19:57 +0100, Stefan Wahren wrote:
> Hi J?rg,
> 

[snip]

> > > 
> > > did you try cyclictest [1]?
> > 
> > Not yet. Not sure what to measure and which values to compare here.
> 
> i tought you have the vendor kernel and the mainline kernel available
> for your platform.
> 
> So you could compare the both kernels.

Yes, that's right. I will have a look at this tool.

> > 
> > > 
> > > Beside the time for a request the amount of requests for the
> > > complete
> > > iperf test
> > > would we interesting. Maybe there are retries.
> > > 
> > > I'm still interested in your PIO mode patches for mxs-mmc even
> > > without clean up.
> > 
> > Actually, the patch does not implement a PIO mode, but drops DMA
> > and
> > uses polling instead. I've attached the patch.
> 
> Thanks. I applied it, but unfortunately this breaks SD card support
> for my Duckbill and the kernel isn't able to mount the rootfs:
> 
> [????2.267073] mxs-mmc 80010000.ssp: initialized
> [????2.272624] mxs-mmc 80010000.ssp: AC command error 0xffffff92

Sorry, I messed up the branches. I attached the correct patch which is
working for me on Linux v4.9.

J?rg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0200-mmc-mxs-mmc-use-PIO-mode.patch
Type: text/x-patch
Size: 15450 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20161215/b5c33807/attachment-0001.bin>

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 15. Dezember 2016 um 14:50 geschrieben:
> 
> 
> Hi Stefan,
> 
> On Wed, 2016-12-14 at 19:57 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > 
> 
> [snip]
> 
> > > > 
> > > > did you try cyclictest [1]?
> > > 
> > > Not yet. Not sure what to measure and which values to compare here.
> > 
> > i tought you have the vendor kernel and the mainline kernel available
> > for your platform.
> > 
> > So you could compare the both kernels.
> 
> Yes, that's right. I will have a look at this tool.
> 
> > > 
> > > > 
> > > > Beside the time for a request the amount of requests for the
> > > > complete
> > > > iperf test
> > > > would we interesting. Maybe there are retries.
> > > > 
> > > > I'm still interested in your PIO mode patches for mxs-mmc even
> > > > without clean up.
> > > 
> > > Actually, the patch does not implement a PIO mode, but drops DMA
> > > and
> > > uses polling instead. I've attached the patch.
> > 
> > Thanks. I applied it, but unfortunately this breaks SD card support
> > for my Duckbill and the kernel isn't able to mount the rootfs:
> > 
> > [    2.267073] mxs-mmc 80010000.ssp: initialized
> > [    2.272624] mxs-mmc 80010000.ssp: AC command error 0xffffff92
> 
> Sorry, I messed up the branches. I attached the correct patch which is
> working for me on Linux v4.9.

i tested the second version but there isn't any performance gain with the patch.

Duckbill with class 10 SD card
Linux 4.8 without patch

dd if=/dev/zero of=test bs=1k count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB) copied, 2.68934 s, 3.8 MB/s

dd if=/dev/zero of=test bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 8.24305 s, 9.9 MB/s


Duckbill with class 10 SD card
Linux 4.8 with patch

dd if=/dev/zero of=test bs=1k count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB) copied, 3.41193 s, 3.0 MB/s

dd if=/dev/zero of=test bs=8k count=10000
10000+0 records in
10000+0 records out
81920000 bytes (82 MB) copied, 14.4564 s, 5.7 MB/s

Additionally i get these warning during boot:

[    2.278445] mxs-mmc 80010000.ssp: initialized
[    2.283996] mxs-mmc 80010000.ssp: AC command error -110
[    2.305158] mxs-mmc 80010000.ssp: AC command error -110
[    2.322975] mxs-mmc 80010000.ssp: AC command error -110
[    2.338660] mxs-mmc 80010000.ssp: AC command error -110
[    2.344289] mxs-mmc 80010000.ssp: AC command error -110
[    2.365653] mxs-mmc 80010000.ssp: AC command error -110

Regards
Stefan

> 
> J?rg

mmc: core: complete/wait_for_completion performance

[Jörg Krause]

Hi Stefan,

On Thu, 2016-12-15 at 19:51 +0100, Stefan Wahren wrote:
> Hi J?rg,
> 
> > J?rg Krause <joerg.krause@embedded.rocks> hat am 15. Dezember 2016
> > um 14:50 geschrieben:
> > 
> > 
> > Hi Stefan,
> > 
> > On Wed, 2016-12-14 at 19:57 +0100, Stefan Wahren wrote:
> > > Hi J?rg,
> > > 
> > 
> > [snip]
> > 
> > > > > 
> > > > > did you try cyclictest [1]?
> > > > 
> > > > Not yet. Not sure what to measure and which values to compare
> > > > here.
> > > 
> > > i tought you have the vendor kernel and the mainline kernel
> > > available
> > > for your platform.
> > > 
> > > So you could compare the both kernels.
> > 
> > Yes, that's right. I will have a look at this tool.
> > 
> > > > 
> > > > > 
> > > > > Beside the time for a request the amount of requests for the
> > > > > complete
> > > > > iperf test
> > > > > would we interesting. Maybe there are retries.
> > > > > 
> > > > > I'm still interested in your PIO mode patches for mxs-mmc
> > > > > even
> > > > > without clean up.
> > > > 
> > > > Actually, the patch does not implement a PIO mode, but drops
> > > > DMA
> > > > and
> > > > uses polling instead. I've attached the patch.
> > > 
> > > Thanks. I applied it, but unfortunately this breaks SD card
> > > support
> > > for my Duckbill and the kernel isn't able to mount the rootfs:
> > > 
> > > [????2.267073] mxs-mmc 80010000.ssp: initialized
> > > [????2.272624] mxs-mmc 80010000.ssp: AC command error 0xffffff92
> > 
> > Sorry, I messed up the branches. I attached the correct patch which
> > is
> > working for me on Linux v4.9.
> 
> i tested the second version but there isn't any performance gain with
> the patch.

In the vendor kernel the polling is used only for small chunks of <=
1024 bytes to save the context switches when using DMA. This patch does
not use DMA at all, but only polling.

As I said before, I guess the limitation in the mxs-mmc driver is the
time needed to return the mmc request to the mmc core driver.

I have a Cubietruck with the same wifi chipset as on my i.MX28 target
where I get ~20Mbps throughput. Furthermore, I've found a benchmark on
a NXP thread [1] measuring about 30Mbps for an i.MX6 target and a
similiar wifi chip.

Looking at the sunxi-mmc driver shows that it calls mmc_request_done()
in an interrupt context and does not use the dmaengine driver@all.

For now, I would drop the polling mode and look how to optimize the
control flow between the DMA controller and the MMC host.
Unfortunately, this will need some time...

> Duckbill with class 10 SD card
> Linux 4.8 without patch
> 
> dd if=/dev/zero of=test bs=1k count=10000
> 10000+0 records in
> 10000+0 records out
> 10240000 bytes (10 MB) copied, 2.68934 s, 3.8 MB/s
> 
> dd if=/dev/zero of=test bs=8k count=10000
> 10000+0 records in
> 10000+0 records out
> 81920000 bytes (82 MB) copied, 8.24305 s, 9.9 MB/s
> 
> 
> Duckbill with class 10 SD card
> Linux 4.8 with patch
> 
> dd if=/dev/zero of=test bs=1k count=10000
> 10000+0 records in
> 10000+0 records out
> 10240000 bytes (10 MB) copied, 3.41193 s, 3.0 MB/s
> 
> dd if=/dev/zero of=test bs=8k count=10000
> 10000+0 records in
> 10000+0 records out
> 81920000 bytes (82 MB) copied, 14.4564 s, 5.7 MB/s
> 
> Additionally i get these warning during boot:
> 
> [????2.278445] mxs-mmc 80010000.ssp: initialized
> [????2.283996] mxs-mmc 80010000.ssp: AC command error -110
> [????2.305158] mxs-mmc 80010000.ssp: AC command error -110
> [????2.322975] mxs-mmc 80010000.ssp: AC command error -110
> [????2.338660] mxs-mmc 80010000.ssp: AC command error -110
> [????2.344289] mxs-mmc 80010000.ssp: AC command error -110
> [????2.365653] mxs-mmc 80010000.ssp: AC command error -110

I get this errors, too. The MMC host is sending some commands and the
MMC client is not (yet) responding to those commands. I haven't looked
any closer at this.

[1] https://community.nxp.com/thread/317396

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 16. Dezember 2016 um 11:06 geschrieben:
> 
> 
> Hi Stefan,
> 
> On Thu, 2016-12-15 at 19:51 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > 
> > > J?rg Krause <joerg.krause@embedded.rocks> hat am 15. Dezember 2016
> > > um 14:50 geschrieben:
> > > 
> > > 
> > > Hi Stefan,
> > > 
> > > On Wed, 2016-12-14 at 19:57 +0100, Stefan Wahren wrote:
> > > > Hi J?rg,
> > > > 
> > > 
> > > [snip]
> > > 
> > > > > > 
> > > > > > did you try cyclictest [1]?
> > > > > 
> > > > > Not yet. Not sure what to measure and which values to compare
> > > > > here.
> > > > 
> > > > i tought you have the vendor kernel and the mainline kernel
> > > > available
> > > > for your platform.
> > > > 
> > > > So you could compare the both kernels.
> > > 
> > > Yes, that's right. I will have a look at this tool.
> > > 
> > > > > 
> > > > > > 
> > > > > > Beside the time for a request the amount of requests for the
> > > > > > complete
> > > > > > iperf test
> > > > > > would we interesting. Maybe there are retries.
> > > > > > 
> > > > > > I'm still interested in your PIO mode patches for mxs-mmc
> > > > > > even
> > > > > > without clean up.
> > > > > 
> > > > > Actually, the patch does not implement a PIO mode, but drops
> > > > > DMA
> > > > > and
> > > > > uses polling instead. I've attached the patch.
> > > > 
> > > > Thanks. I applied it, but unfortunately this breaks SD card
> > > > support
> > > > for my Duckbill and the kernel isn't able to mount the rootfs:
> > > > 
> > > > [    2.267073] mxs-mmc 80010000.ssp: initialized
> > > > [    2.272624] mxs-mmc 80010000.ssp: AC command error 0xffffff92
> > > 
> > > Sorry, I messed up the branches. I attached the correct patch which
> > > is
> > > working for me on Linux v4.9.
> > 
> > i tested the second version but there isn't any performance gain with
> > the patch.
> 
> In the vendor kernel the polling is used only for small chunks of <=
> 1024 bytes to save the context switches when using DMA. This patch does
> not use DMA at all, but only polling.

also the vendor kernel uses polling for AC and BC commands. I tried this approach (use polling for AC/BC/BCR commands and DMA for all ADTC commands) [1] on Duckbill with SD card but the resulting read and write performance stays the same. Maybe you want to give it a try with Wifi over SDIO.

Here are some read performance values with Duckbill (Kernel 4.8, class 10 microSD card):

dd if=/dev/mmcblk0p2 of=/dev/null
64260+0 records in
64260+0 records out
32901120 bytes (33 MB) copied, 1.68618 s, 19.5 MB/s

> 
> As I said before, I guess the limitation in the mxs-mmc driver is the
> time needed to return the mmc request to the mmc core driver.

I don't think this is the problem. I added some GPIO handling into mxs-mmc driver and i couldn't see any big delay between the mmc requests with a logic analyzer.

> 
> I have a Cubietruck with the same wifi chipset as on my i.MX28 target
> where I get ~20Mbps throughput. Furthermore, I've found a benchmark on
> a NXP thread [1] measuring about 30Mbps for an i.MX6 target and a
> similiar wifi chip.
> 
> Looking at the sunxi-mmc driver shows that it calls mmc_request_done()
> in an interrupt context and does not use the dmaengine driver at all.
> 
> For now, I would drop the polling mode and look how to optimize the
> control flow between the DMA controller and the MMC host.
> Unfortunately, this will need some time...

I also rebased an old patch from Shawn Guo [2] with pre_req and post_req support, tried to call the DMA channel callback from the interrupt context instead of scheduling the tasklet within the DMA engine driver and implement CMD23 support [3]. But none of them show any measurable performance improvement.

Btw here are some really performance critical kernel config parameter which really needs to be disabled:

# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_PROVE_RCU is not set

[1] - https://github.com/lategoodbye/linux-mxs-power/commit/beb341ed948ae9b8afe7378cff6b9d50144fd0b9
[2] - https://github.com/lategoodbye/linux-mxs-power/commit/e96b28e8730ccfcfecb7ec286102bc6969aa1ee0
[3] - https://github.com/lategoodbye/linux-mxs-power/commit/e53a3c9169a63eb61f9e67ff88724972acf312a9

mmc: core: complete/wait_for_completion performance

[Stefan Wahren]

Hi J?rg,

> J?rg Krause <joerg.krause@embedded.rocks> hat am 15. Januar 2017 um 21:42 geschrieben:
> 
> 
> Hi Stefan,
> 
> On Tue, 2016-12-27 at 00:03 +0100, Stefan Wahren wrote:
> > Hi J?rg,
> > ...
> > I also rebased an old patch from Shawn Guo [2] with pre_req and
> > post_req support, tried to call the DMA channel callback from the
> > interrupt context instead of scheduling the tasklet within the DMA
> > engine driver and implement CMD23 support [3]. But none of them show
> > any measurable performance improvement.
> 
> I tested the three patches after disabling any debugging options in the
> config. There is no performance gain, but the timings have changed.
> Please have a look at the attached graphs. The time between complete()
> and wait_for_completion() is reduced to 15us whereas the time from
> return from wait_for_completion() to return to sdio_readsb() increases
> to 23us (from maybe 2us before).

that confirms my results. I think you are searching on the wrong layer. You better step "back" and take a look at the whole transfer instead of single blocks. Maybe there are some bigger delays.

Stefan

> 
> J?rg