[SPDK] Re: ioat: "could not start channel" after abnormal exit

[SPDK] Re: ioat: "could not start channel" after abnormal exit

[lijh2015]

[-- Attachment #1: Type: text/plain, Size: 1549 bytes --]

Hi Paul,

Thanks for your advice. But that didn't seem to work. The output of /scripts/setup.sh is same as normal
In fact, I found out how such error happens.  It's because I accidentally launch ioat copy
on system malloc-ed memory space, I've already fixed that in my code. But there are still some
errors around. After reboot, and run scripts/setup.sh again, there are still 4 channels (1 channel
from each process) cannot be attached:

E0716 19:02:24.496325 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
 error = 0
E0716 19:02:24.496567 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
EAL: Requested device 0000:00:04.4 cannot be used
E0716 19:02:24.527472 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
 error = 0
E0716 19:02:24.527948 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
EAL: Requested device 0000:40:04.4 cannot be used
E0716 19:02:24.558609 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
 error = 0
E0716 19:02:24.559036 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
EAL: Requested device 0000:80:04.4 cannot be used
E0716 19:02:24.589241 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
 error = 0
E0716 19:02:24.589682 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
EAL: Requested device 0000:c0:04.4 cannot be used

The other channels are working fine now, but these errors are strange.

Thanks,
Jiahao Li

[SPDK] Re: ioat: "could not start channel" after abnormal exit

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 1345 bytes --]

You bet!  Here's the main announcement https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator and there are several links at the bottom, including the spec.  Right now I've done all the development using a simulator that is not publicly available but as soon as this launches the support will be there in SPDK.  Here's the bottom of the current patch chain that finishes off all planned capabilities to add to SPDK https://review.spdk.io/gerrit/c/spdk/spdk/+/3319/1 and currently there is basic support for copy and already introduced last release.  This release adds quite a bit of functionality.

Next release we'll start working on using the accel_fw within various SPDK modules where it makes sense (CRC32C, compare, fill, copy, etc)

-Paul

On 7/16/20, 11:20 AM, "lijh2015(a)mail.ustc.edu.cn" <lijh2015(a)mail.ustc.edu.cn> wrote:

    Hi Paul,

    A power cycle solved my problem. Thanks again for your help!

    Being new to IOAT technology, I am interested to learn how powerful the next gen IOAT technology will be, and what performance gain does the new accel framework have. Please tell me more about them.

    Thanks,
    Jiahao Li
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: ioat: "could not start channel" after abnormal exit

[lijh2015]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Hi Paul,

A power cycle solved my problem. Thanks again for your help!

Being new to IOAT technology, I am interested to learn how powerful the next gen IOAT technology will be, and what performance gain does the new accel framework have. Please tell me more about them.

Thanks,
Jiahao Li

[SPDK] Re: ioat: "could not start channel" after abnormal exit

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2950 bytes --]

Hi Jiahao,

Well that’s good news.  I did try ctrl-c exit from both ioat_perf and the new accel_perf that I'm working on now and they both worked fine.  BTW, accel_perf is being developed as the perf tool for the new (renamed and expanded) acceleration framework that you might be interested in.  It's not quite feature complete yet but for the upcoming release in just a few weeks the accel framework will include support for IOAT, DSA (next gen IOAT not yet available in Si but once it is it will be a huge performance boost, plus new features) as well as sw implementations for all the features so that the framework API can be used in any environment.  Let me know if you'd like more info on any of that.

Now, wrt your IOAT issue, are you saying that following simply using a bogus address in a command that you're in a state where powering down the machine and powering it that some channels are still bad? Please confirm, that sounds "odd" for sure. Also please confirm that you power cycled, not just rebooted.

Thx
Paul

On 7/16/20, 4:23 AM, "lijh2015(a)mail.ustc.edu.cn" <lijh2015(a)mail.ustc.edu.cn> wrote:

    Hi Paul,

    Thanks for your advice. But that didn't seem to work. The output of /scripts/setup.sh is same as normal
    In fact, I found out how such error happens.  It's because I accidentally launch ioat copy
    on system malloc-ed memory space, I've already fixed that in my code. But there are still some
    errors around. After reboot, and run scripts/setup.sh again, there are still 4 channels (1 channel
    from each process) cannot be attached:

    E0716 19:02:24.496325 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
     error = 0
    E0716 19:02:24.496567 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
    EAL: Requested device 0000:00:04.4 cannot be used
    E0716 19:02:24.527472 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
     error = 0
    E0716 19:02:24.527948 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
    EAL: Requested device 0000:40:04.4 cannot be used
    E0716 19:02:24.558609 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
     error = 0
    E0716 19:02:24.559036 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
    EAL: Requested device 0000:80:04.4 cannot be used
    E0716 19:02:24.589241 189532 ioat.c:480] [SPDK/ioat_channel_start] could not start channel: status = 0x3
     error = 0
    E0716 19:02:24.589682 189532 ioat.c:547] [SPDK/ioat_enum_cb] ioat_attach() failed
    EAL: Requested device 0000:c0:04.4 cannot be used

    The other channels are working fine now, but these errors are strange.

    Thanks,
    Jiahao Li
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: ioat: "could not start channel" after abnormal exit

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

Hi Jiahao,

I don't know off the top of my head but I can try and repro with one of the SPDK tools and see what I can see.  In the meantime you can try "/scripts/setup.sh reset" followed by setup.sh again to see if that gives you a temp workaround

Thx
Paul 

On 7/14/20, 7:49 PM, "lijh2015(a)mail.ustc.edu.cn" <lijh2015(a)mail.ustc.edu.cn> wrote:

    Hi SPDK team,

    I'm using SPDK (v20-01-pre) to evaluate I/OAT performance. There are some bugs in my benchmark program, 
    but after I use ctrl+c to quit my program and restart it again, some error messages appeared:

    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:00:04.0 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:00:04.1 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:00:04.2 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:00:04.3 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:40:04.0 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0
    ioat.c: 547:ioat_enum_cb: *ERROR*: ioat_attach() failed
    EAL: Requested device 0000:40:04.1 cannot be used
    ioat.c: 480:ioat_channel_start: *ERROR*: could not start channel: status = 0x3
     error = 0

    I checked the code in lib/ioat/ioat.c, it seems is_ioat_idle always returns negative results.
    What should I do to deal with this problem without rebooting my server? Or, at least,
    how to prevent it?

    Thanks,
    Jiahao Li
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org