From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: Date: Thu, 10 Feb 2022 07:29:25 +0100 MIME-Version: 1.0 Subject: Re: DMA with EVL-attached Thread Content-Language: en-US References: From: Jan Kiszka In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit List-Id: Discussions about the Xenomai project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Russell Johnson , "xenomai@xenomai.org" On 10.02.22 00:00, Russell Johnson via Xenomai wrote: > Hello, > > I currently have a DMA driver written based on the Linux DMA API found here: https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt. I have multiple realtime threads in my userspace app that have to make constant DMA transfers. I have not seen any issue with DMA throughput using the Linux API. The question is, do I need to re-architect the dma driver to add in the oob_ioctl() FD so that a userspace EVL attached thread can use it? If the answer is yes, I have experimented a bit with doing that and ran into an issue. I have created a userspace test app with a thread attached to EVL and attempted to allocate, write, read, and free a dma buffer from the driver I modified with the oob_ioctl() FD. Alloc, write, and read all appear to work fine. However, calling free (which boils down to the kernel call "dma_free_coherent" is throwing a warning in the console: > > [75787.696762] ------------[ cut here ]------------ > [75787.696763] WARNING: CPU: 9 PID: 25074 at kernel/dma/mapping.c:532 dma_free_attrs+0x40/0x90 > [75787.696772] Modules linked in: dma_buf_mgr(OE) xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm ixgbe ahci libahci igb libata mdio crc32c_intel ptp pps_core i2c_algo_bit dca [last unloaded: dma_buf_mgr] > [75787.696793] CPU: 9 PID: 25074 Comm: test_thread:250 Tainted: G W OE 5.15.0evl+ #6 > [75787.696796] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018 > [75787.696797] IRQ stage: EVL > [75787.696799] RIP: 0010:dma_free_attrs+0x40/0x90 > [75787.696802] Code: 89 fc 53 48 83 ec 08 48 8b 9f 38 02 00 00 4c 89 45 d0 48 85 db 48 0f 44 1d 4d ce 8b 01 e8 88 23 80 00 85 c0 4c 8b 45 d0 74 02 <0f> 0b 4d 85 ed 74 16 48 85 db 75 20 4c 89 f9 4c 89 ea 4c 89 f6 4c > [75787.696804] RSP: 0018:ffffb3fd40747d48 EFLAGS: 00010202 > [75787.696807] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00000000798fa000 > [75787.696809] RDX: ffff8dda798fa000 RSI: 0000000000000400 RDI: ffff8ddb01b2f0d0 > [75787.696810] RBP: ffffb3fd40747d78 R08: 0000000000000000 R09: 00080000000000ff > [75787.696811] R10: 00000000807eb2a2 R11: 0000000000000033 R12: ffff8ddb01b2f0d0 > [75787.696813] R13: ffff8dda798fa000 R14: 0000000000000400 R15: 00000000798fa000 > [75787.696814] FS: 00007f355a0d7700(0000) GS:ffff8dde6d880000(0000) knlGS:0000000000000000 > [75787.696816] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [75787.696818] CR2: 0000000000000000 CR3: 00000001b5448001 CR4: 00000000003706e0 > [75787.696819] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [75787.696820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [75787.696822] Call Trace: > [75787.696826] free_dma_buf+0x5d/0xb0 [dma_buf_mgr] > [75787.696831] dma_buf_mgr_oob_ioctl+0x159/0x396 [dma_buf_mgr] > [75787.696835] EVL_ioctl+0x46/0xa0 > [75787.696841] handle_pipelined_syscall+0x181/0x2d0 > [75787.696846] __pipeline_syscall+0xbc/0x230 > [75787.696851] ? sched_clock+0x9/0x10 > [75787.696856] pipeline_syscall+0x34/0x100 > [75787.696860] syscall_enter_from_user_mode+0x39/0xf0 > [75787.696865] do_syscall_64+0x15/0xa0 > [75787.696867] entry_SYSCALL_64_after_hwframe+0x44/0xae > [75787.696872] RIP: 0033:0x7f355b0f2f9b > [75787.696874] Code: Unable to access opcode bytes at RIP 0x7f355b0f2f71. > [75787.696875] RSP: 002b:00007f355a0d6860 EFLAGS: 00000246 ORIG_RAX: 000000000000009d > [75787.696877] RAX: ffffffffffffffda RBX: 00007f355a0d6900 RCX: 00007f355b0f2f9b > [75787.696878] RDX: 0000000040042501 RSI: 0000000000000006 RDI: 0000000010000002 > [75787.696880] RBP: 0000000040042501 R08: 0000000000000000 R09: 0000000000020750 > [75787.696881] R10: 00007f355a0d6900 R11: 0000000000000246 R12: 0000000000000006 > [75787.696882] R13: 0000000000801000 R14: 0000000000000000 R15: 00007f355a0d7700 > [75787.696884] ---[ end trace 9678e85a266ff468 ]--- > > This warning does not exist when running the same functionality from the normal ioctl() FD from a standard thread in Linux. Is what I am trying to do possible? If so, any clue on what I am seeing here? Yes, classic mistake: You are calling a Linux, in-band-only function (free_dma_buf) from an RT, out-of-band context. That does not work. Linux resource management has to happen in in-band context (standard ioctl handler). Jan -- Siemens AG, Technology Competence Center Embedded Linux