Propper use of pci_map_sg

[malte.vesper@postgrad.manchester.ac.uk (Malte Vesper)]

The problem was that I did not know to call `sg_init_table`, it is not 
sufficient to just init the seperate entries with sg_set_page.

On 02/02/15 19:49, Malte Vesper wrote:
> Since my last mail went unanswered I did some further investigation. I 
> compiled a kernel with the DMA-API debug fs support and got the dmesg 
> output below. Note how I run the program 3 times and it only reports 
> on the third run that the page mapping count is off. The thing 
> confusing me the most about this, is that I can't see where I go 
> wrong. Since I call pci_map_sg and pci_unmap_sg directly after each 
> other there is no room for parameter modification. (I have only one 
> device, and the interrupt handler has output when invoked). So what is 
> wrong with:
>
>                 mapped =*pci_map_sg*(pcidev, scatterlist, pinned, 
> DMA_BIDIRECTIONAL);
> *pci_unmap_sg*(pcidev, scatterlist, pinned, DMA_BIDIRECTIONAL);
>
> Do I have to recreate the vanilla scatterlist for pci_unmap_sg?
>
> Any hints/help is highly appreciated.
>
> Ted
>
>
>
> Dmesg output:
>
>     [ 2087.171247] SCAS FPGA 0000:01:00.0: [fpga+]: Removing device 0
>     [ 2087.171771] [fpga+]: SCAS MAJOR UNREGISTERED: 100
>     [ 2087.180070] [fpga+]: fpga+ registered with major number 100
>     [ 2087.180096] SCAS FPGA 0000:01:00.0: Probing
>     [ 2087.180163] SCAS FPGA 0000:01:00.0: irq 66 for MSI/MSI-X
>     [ 2087.180182] SCAS FPGA 0000:01:00.0: [fpga+]: Board tied to slot 0
>     [ 2092.359203] [fpga+]: IOCTL 2148040970
>     [ 2092.359206] [fpga+]: MINOR: 0
>     [ 2092.359207] [fpga+]: ARG: 140736686734672
>     [ 2092.359210] SCAS FPGA 0000:01:00.0: [fpga+]: IOCTL:
>     IOCTL_FPGA_PIN_PAGE
>     [ 2092.359211] Testers hell
>     [ 2092.359215] pinned 2
>     [ 2092.359216] setting page: 0
>     [ 2092.359217] setting page: 1
>     [ 2092.359223] Mapped: 2
>     [ 2092.359227] Done
>     [ 2117.846740] [fpga+]: IOCTL 2148040970
>     [ 2117.846742] [fpga+]: MINOR: 0
>     [ 2117.846743] [fpga+]: ARG: 140735106116944
>     [ 2117.846746] SCAS FPGA 0000:01:00.0: [fpga+]: IOCTL:
>     IOCTL_FPGA_PIN_PAGE
>     [ 2117.846746] Testers hell
>     [ 2117.846749] pinned 2
>     [ 2117.846749] setting page: 0
>     [ 2117.846750] setting page: 1
>     [ 2117.846755] Mapped: 2
>     [ 2117.846759] Done
>     [ 2124.136132] [fpga+]: IOCTL 2148040970
>     [ 2124.136134] [fpga+]: MINOR: 0
>     [ 2124.136135] [fpga+]: ARG: 140735560133008
>     [ 2124.136138] SCAS FPGA 0000:01:00.0: [fpga+]: IOCTL:
>     IOCTL_FPGA_PIN_PAGE
>     [ 2124.136138] Testers hell
>     [ 2124.136141] pinned 2
>     [ 2124.136142] setting page: 0
>     [ 2124.136142] setting page: 1
>     [ 2124.136147] Mapped: 2
>     [ 2124.136152] Done
>     [ 2124.136251] BUG: Bad page map in process dma
>     pte:8000000f7b983867 pmd:fa9d31067
>     [ 2124.136254] page:ffffea003dee60c0 count:1 mapcount:-30705
>     mapping:ffff880fadddb901 index:0xc0054000afddc901
>     [ 2124.136254] flags:
>     0x2ffff000008006c(referenced|uptodate|lru|active|swapbacked)
>     [ 2124.136257] page dumped because: bad pte
>     [ 2124.136258] addr:0000000001382000 vm_flags:08100073
>     anon_vma:ffff880fadddb900 mapping:          (null) index:1382
>     [ 2124.136261] CPU: 6 PID: 6902 Comm: dma Tainted: G    B   W OE 
>     3.18.5hotdog #2
>     [ 2124.136261] Hardware name: System manufacturer System Product
>     Name/RAMPAGE IV BLACK EDITION, BIOS 0701 06/04/2014
>     [ 2124.136262]  ffff880fb842eac8 ffff880fd4093c58 ffffffff8176ab0c
>     0000000000000000
>     [ 2124.136264]  0000000001382000 ffff880fd4093ca8 ffffffff8119215a
>     8000000f7b983867
>     [ 2124.136265]  0000000000001382 ffff880fd4093ca8 ffff880fa9d31c10
>     ffffea003dee60c0
>     [ 2124.136267] Call Trace:
>     [ 2124.136270]  [<ffffffff8176ab0c>] dump_stack+0x46/0x58
>     [ 2124.136274]  [<ffffffff8119215a>] print_bad_pte+0x1aa/0x250
>     [ 2124.136275]  [<ffffffff8119367f>] unmap_single_vma+0x76f/0x8f0
>     [ 2124.136278]  [<ffffffff81175cf0>] ? put_pages_list+0x70/0x70
>     [ 2124.136279]  [<ffffffff811942d1>] unmap_vmas+0x51/0xa0
>     [ 2124.136281]  [<ffffffff8119d48c>] exit_mmap+0x9c/0x170
>     [ 2124.136284]  [<ffffffff8106c1c4>] mmput+0x64/0x130
>     [ 2124.136286]  [<ffffffff810716ec>] do_exit+0x27c/0xa80
>     [ 2124.136287]  [<ffffffff81071f7f>] do_group_exit+0x3f/0xa0
>     [ 2124.136290]  [<ffffffff81071ff4>] SyS_exit_group+0x14/0x20
>     [ 2124.136292]  [<ffffffff81772f6d>] system_call_fastpath+0x16/0x1b
>     [ 2124.136319] BUG: Bad page state in process dma  pfn:f7b983
>     [ 2124.136320] page:ffffea003dee60c0 count:0 mapcount:-30705
>     mapping:          (null) index:0xc0054000afddc901
>     [ 2124.136321] flags:
>     0x2ffff000008000c(referenced|uptodate|swapbacked)
>     [ 2124.136322] page dumped because: nonzero mapcount
>     [ 2124.136323] Modules linked in: fpga+(OE) hid_generic nfsv3
>     rpcsec_gss_krb5 nfsv4 vmw_vsock_vmci_transport vsock vmw_vmci
>     autofs4 nfsd auth_rpcgss nfs_acl nfs snd_hda_codec_realtek
>     snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel
>     snd_hda_controller snd_hda_codec x86_pkg_temp_thermal
>     intel_powerclamp snd_hwdep coretemp rfcomm snd_pcm kvm_intel kvm
>     bnep lockd bluetooth snd_seq_midi snd_seq_midi_event snd_rawmidi
>     snd_seq grace sunrpc eeepc_wmi asus_wmi snd_seq_device snd_timer
>     sparse_keymap joydev video mei_me sb_edac snd edac_core mei
>     serio_raw lpc_ich soundcore fscache binfmt_misc nls_iso8859_1
>     parport_pc ppdev mac_hid tpm_infineon cp210x usbserial lp parport
>     dm_crypt hid_microsoft usbhid hid radeon i2c_algo_bit ttm
>     drm_kms_helper mxm_wmi drm crct10dif_pclmul crc32_pclmul e1000e
>     ghash_clmulni_intel aesni_intel psmouse aes_x86_64 lrw gf128mul
>     glue_helper ablk_helper cryptd aacraid ahci ptp libahci pps_core
>     wmi [last unloaded: fpga+]
>     [ 2124.136351] CPU: 6 PID: 6902 Comm: dma Tainted: G    B   W OE 
>     3.18.5hotdog #2
>     [ 2124.136352] Hardware name: System manufacturer System Product
>     Name/RAMPAGE IV BLACK EDITION, BIOS 0701 06/04/2014
>     [ 2124.136353]  ffffffff81aa55c7 ffff880fd4093bb8 ffffffff8176ab0c
>     0000000000000000
>     [ 2124.136354]  ffffea003dee60c0 ffff880fd4093be8 ffffffff81767527
>     ff0000ffffffffff
>     [ 2124.136355]  ffffea003dee60c0 0000000000000000 00ffff0000000000
>     ffff880fd4093c38
>     [ 2124.136356] Call Trace:
>     [ 2124.136358]  [<ffffffff8176ab0c>] dump_stack+0x46/0x58
>     [ 2124.136360]  [<ffffffff81767527>] bad_page.part.50+0xe0/0xfe
>     [ 2124.136362]  [<ffffffff8116c91d>] free_pages_prepare+0x16d/0x190
>     [ 2124.136363]  [<ffffffff8116f1a5>] free_hot_cold_page+0x35/0x180
>     [ 2124.136365]  [<ffffffff8116f33e>] free_hot_cold_page_list+0x4e/0xa0
>     [ 2124.136366]  [<ffffffff81176076>] release_pages+0x1f6/0x270
>     [ 2124.136368]  [<ffffffff811a88bd>]
>     free_pages_and_swap_cache+0x8d/0xa0
>     [ 2124.136370]  [<ffffffff81191b04>] tlb_flush_mmu_free+0x34/0x60
>     [ 2124.136371]  [<ffffffff81192524>] tlb_flush_mmu+0x24/0x30
>     [ 2124.136373]  [<ffffffff81192544>] tlb_finish_mmu+0x14/0x40
>     [ 2124.136374]  [<ffffffff8119d4bb>] exit_mmap+0xcb/0x170
>     [ 2124.136376]  [<ffffffff8106c1c4>] mmput+0x64/0x130
>     [ 2124.136378]  [<ffffffff810716ec>] do_exit+0x27c/0xa80
>     [ 2124.136379]  [<ffffffff81071f7f>] do_group_exit+0x3f/0xa0
>     [ 2124.136381]  [<ffffffff81071ff4>] SyS_exit_group+0x14/0x20
>     [ 2124.136382]  [<ffffffff81772f6d>] system_call_fastpath+0x16/0x1b
>     ted at ted-headlessBrute:~$
>
>
>
>
>
> On 29/01/15 19:50, Malte Vesper wrote:
>> Hi,
>> I am trying to use pci_map_sg in combination with get_user_pages_fast 
>> for a driver. However my code screws the processes memory map over. 
>> For easier testing I bundled the piinin/unpinning and mapping 
>> unmapping into one ioctl call. The following code misses all the 
>> checks on return values, since it is only inteneded as a MWE.
>>
>> When I run my code I get a "*BUG: Bad page map in process ...*", 
>> after "Done" is printed.
>>
>> I tried following DMA-API-howto.txt and looking at other code, but I 
>> fail to see where I go wrong.
>>
>> Regards, Ted
>>
>>
>> Code from the IOCTL handler:
>>
>>         case IOCTL_FPGA_PIN_PAGE:
>>         {
>>             struct pageInfo pageInfo;
>>
>>             dev_dbg(&pcidev->dev, pr_fmt("IOCTL: 
>> IOCTL_FPGA_PIN_PAGE\n"));
>>
>>             if(!copy_from_user(&pageInfo, (void*)arg, sizeof(struct 
>> pageInfo))) {
>>                 //horrible test
>>                 const int noPages = pageInfo.size/PAGE_SIZE;
>>                 int pinned;
>>                 int mapped  = 0;
>>                 struct scatterlist* scatterlist;
>>                 printk("Test start\n");
>>                 //userspacestartpointer, nopages, write?, page* array
>>                 struct page** pages=kmalloc(sizeof(struct 
>> page*)*noPages, GFP_KERNEL);
>>                 pinned=*get_user_pages_fast*((unsigned 
>> long)pageInfo.start, noPages, 1, pages);
>>
>>                 scatterlist = kmalloc(sizeof(struct 
>> scatterlist)*pinned, GFP_KERNEL);
>>
>>                 for(int i=0; i<pinned; ++i) {
>>                     sg_set_page(&scatterlist[i], pages[i], PAGE_SIZE, 0);
>>                 }
>>
>>                 mapped =*pci_map_sg*(pcidev, scatterlist, pinned, 
>> DMA_BIDIRECTIONAL);
>> *pci_unmap_sg*(pcidev, scatterlist, pinned, DMA_BIDIRECTIONAL);
>>
>>                 for(int i=0; i<pinned; ++i) {
>> *put_page*(pages[i]); //I did place a print here and got two pages 
>> unpinned as I expected
>>                 }
>>                 kfree(scatterlist);
>>                 kfree(pages);
>>                 printk("Done\n");
>>                 /*pageInfo.pageId = getPageId(getArea(&pageInfo));
>>                 copy_to_user((void*)arg, &pageInfo, sizeof(struct 
>> pageInfo));*/
>>                 return pageInfo.pageId;
>>             } else {
>>                 return -EFAULT;
>>             }
>>         }
>>             break;
>>
>> P.S.: I used the following code to allocate the memory in userspace, 
>> and I use chunk and chunksize to fill pageInfo.start and 
>> pageInfo.size respectivly:
>>
>>     const int pagesize = sysconf(_SC_PAGESIZE);
>>     const int chunksize = 2*pagesize;
>>     void* chunk;
>>
>>     if(int error = posix_memalign(&chunk, pagesize, chunksize)) {
>>         std::cout << "Could not get a page." << std::endl;
>>         return error;
>>     }
>>
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20150203/65c27cd2/attachment-0001.html