From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751748AbbCQFbu (ORCPT ); Tue, 17 Mar 2015 01:31:50 -0400 Received: from mail-by2on0129.outbound.protection.outlook.com ([207.46.100.129]:23712 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750782AbbCQFbr (ORCPT ); Tue, 17 Mar 2015 01:31:47 -0400 From: Yuan Yao To: , CC: , , Subject: [PATCH 1/2] dma: Add Freescale qDMA engine driver support Date: Tue, 17 Mar 2015 13:28:38 +0800 Message-ID: <1426570119-8634-1-git-send-email-yao.yuan@freescale.com> X-Mailer: git-send-email 2.1.0.27.g96db324 X-EOPAttributedMessage: 0 Authentication-Results: spf=fail (sender IP is 192.88.168.50) smtp.mailfrom=yao.yuan@freescale.com; lists.infradead.org; dkim=none (message not signed) header.d=none; X-Forefront-Antispam-Report: CIP:192.88.168.50;CTRY:US;IPV:NLI;EFV:NLI;BMV:1;SFV:NSPM;SFS:(10019020)(6009001)(339900001)(189002)(199003)(51234002)(46102003)(62966003)(36756003)(50226001)(551934003)(77156002)(77096005)(50986999)(85426001)(19580395003)(6806004)(19580405001)(87936001)(47776003)(86362001)(575784001)(50466002)(229853001)(33646002)(105606002)(104016003)(48376002)(92566002)(2004002);DIR:OUT;SFP:1102;SCL:1;SRVR:BY2PR0301MB0632;H:tx30smr01.am.freescale.net;FPR:;SPF:Fail;MLV:sfv;MX:1;A:1;LANG:en; MIME-Version: 1.0 Content-Type: text/plain X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0301MB0632; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5002010)(5005006);SRVR:BY2PR0301MB0632;BCL:0;PCL:0;RULEID:;SRVR:BY2PR0301MB0632; X-Forefront-PRVS: 0518EEFB48 X-OriginatorOrg: freescale.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2015 05:31:43.7691 (UTC) X-MS-Exchange-CrossTenant-Id: 710a03f5-10f6-4d38-9ff4-a80b81da590d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=710a03f5-10f6-4d38-9ff4-a80b81da590d;Ip=[192.88.168.50];Helo=[tx30smr01.am.freescale.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR0301MB0632 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add Freescale Queue Direct Memory Access (qDMA) controller support. The qDMA supports channel virtualization by allowing DMA jobs to be enqueued into different command queues. Core can initiate a DMA transaction by preparing a command descriptor (CD) for each DMA job and enqueuing this job to a command queue. This module can be found on LS-1021 LS1043 and LS2085 SoCs. Signed-off-by: Yuan Yao --- Documentation/devicetree/bindings/dma/fsl-qdma.txt | 51 ++ drivers/dma/Kconfig | 11 + drivers/dma/Makefile | 1 + drivers/dma/fsl-qdma.c | 929 +++++++++++++++++++++ 4 files changed, 992 insertions(+) create mode 100644 Documentation/devicetree/bindings/dma/fsl-qdma.txt create mode 100644 drivers/dma/fsl-qdma.c diff --git a/Documentation/devicetree/bindings/dma/fsl-qdma.txt b/Documentation/devicetree/bindings/dma/fsl-qdma.txt new file mode 100644 index 0000000..676ae3d --- /dev/null +++ b/Documentation/devicetree/bindings/dma/fsl-qdma.txt @@ -0,0 +1,50 @@ +* Freescale queue Direct Memory Access Controller(qDMA) Controller + + The qDMA controller transfers blocks of data between one source and one or more +destinations. The blocks of data transferred can be represented in memory as contiguous +or non-contiguous using scatter/gather table(s). Channel virtualization is supported +through enqueuing of DMA jobs to, or dequeuing DMA jobs from, different work +queues. + +* qDMA Controller +Required properties: +- compatible : + - "fsl,ls1021a-qdma" for qDMA used similar to that on LS1021a SoC +- reg : Specifies base physical address(s) and size of the qDMA registers. + The region is qDMA control register's address and size. +- interrupts : A list of interrupt-specifiers, one for each entry in + interrupt-names. +- interrupt-names : Should contain: + "qdma-controller" - the controller interrupt + "qdma-queue" - the queue interrupt +- status-sizes : Number of circular status descriptor queue size +- channels : Number of channels supported by the controller +- queues : Number of queues supported by the controller +- queue-sizes : Number of circular descriptor queue size for each queue +- queue-group : The group for each queue belong to +- queue-weight : The weight for each queue belog to +- default-queue : The default queue for request a new channel + +Optional properties: +- big-endian: If present registers and hardware scatter/gather descriptors + of the qDMA are implemented in big endian mode, otherwise in little + mode. + + +Examples: + + qdma: qdma@8390000 { + compatible = "fsl,ls1021a-qdma"; + reg = <0x0 0x8390000 0x0 0x10000>; + interrupts = , + ; + interrupt-names = "qdma-controller", "qdma-queue"; + status-sizes = <64>; + channels = <8>; + queues = <2>; + queue-sizes = <256 256>; + queue-group = <0 1>; + queue-weight = <0 0>; + default-queue = <0>; + big-endian; + }; diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig index 074ffad..fa52c9e 100644 --- a/drivers/dma/Kconfig +++ b/drivers/dma/Kconfig @@ -388,6 +388,17 @@ config FSL_EDMA multiplexing capability for DMA request sources(slot). This module can be found on Freescale Vybrid and LS-1 SoCs. +config FSL_QDMA + tristate "Freescale qDMA engine support" + depends on OF + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + help + Support the Freescale qDMA engine with command queue and legacy mode. + Channel virtualization is supported through enqueuing of DMA jobs to, + or dequeuing DMA jobs from, different work queues. + This module can be found on Freescale LS SoCs. + config XILINX_VDMA tristate "Xilinx AXI VDMA Engine" depends on (ARCH_ZYNQ || MICROBLAZE) diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile index bf44858..5f2b95e 100644 --- a/drivers/dma/Makefile +++ b/drivers/dma/Makefile @@ -46,6 +46,7 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o obj-$(CONFIG_K3_DMA) += k3dma.o obj-$(CONFIG_MOXART_DMA) += moxart-dma.o obj-$(CONFIG_FSL_EDMA) += fsl-edma.o +obj-$(CONFIG_FSL_QDMA) += fsl-qdma.o obj-$(CONFIG_QCOM_BAM_DMA) += qcom_bam_dma.o obj-y += xilinx/ obj-$(CONFIG_INTEL_MIC_X100_DMA) += mic_x100_dma.o diff --git a/drivers/dma/fsl-qdma.c b/drivers/dma/fsl-qdma.c new file mode 100644 index 0000000..00f3c33 --- /dev/null +++ b/drivers/dma/fsl-qdma.c @@ -0,0 +1,929 @@ +/* + * drivers/dma/fsl-qdma.c + * + * Copyright 2014-2015 Freescale Semiconductor, Inc. + * + * Driver for the Freescale qDMA engine with command queue and legacy mode. + * Channel virtualization is supported through enqueuing of DMA jobs to, + * or dequeuing DMA jobs from, different work queues. + * This module can be found on Freescale LS SoCs. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "virt-dma.h" + +#define FSL_QDMA_DMR 0x8000 +#define FSL_QDMA_DSR 0x8004 +#define FSL_QDMA_DEIER 0x9e00 +#define FSL_QDMA_DEDR 0x9e04 +#define FSL_QDMA_DECFDW0R 0x9e10 +#define FSL_QDMA_DECFDW1R 0x9e14 +#define FSL_QDMA_DECFDW2R 0x9e18 +#define FSL_QDMA_DECFDW3R 0x9e1c +#define FSL_QDMA_DECFQIDR 0x9e30 +#define FSL_QDMA_DECBR 0x9e34 + +#define FSL_QDMA_BCQMR(x) (0xa0c0 + 0x100 * (x)) +#define FSL_QDMA_BCQSR(x) (0xa0c4 + 0x100 * (x)) +#define FSL_QDMA_SQDPAR 0xa80c +#define FSL_QDMA_SQEPAR 0xa814 +#define FSL_QDMA_BSQMR 0xa800 +#define FSL_QDMA_BSQSR 0xa804 +#define FSL_QDMA_BCQIER 0xa0e0 +#define FSL_QDMA_BSQICR 0xa828 +#define FSL_QDMA_CQIER 0xaa10 +#define FSL_QDMA_SQCCMR 0xaa20 +#define FSL_QDMA_BCQIDR 0xa0e4 +#define FSL_QDMA_CQDPA_SADDR(x) (0xa0cc + 0x100 * (x)) +#define FSL_QDMA_CQEPA_SADDR(x) (0xa0d4 + 0x100 * (x)) + +#define FSL_QDMA_SQICR_ICEN + +#define FSL_QDMA_CQIDR_CQT 0xff000000 +#define FSL_QDMA_CQIDR_SQPE 0x800000 +#define FSL_QDMA_CQIDR_SQT 0x8000 + +#define FSL_QDMA_BCQIER_CQTIE 0x8000 +#define FSL_QDMA_BCQIER_CQPEIE 0x800000 +#define FSL_QDMA_BSQICR_ICEN 0x80000000 +#define FSL_QDMA_BSQICR_ICST(x) ((x) << 16) +#define FSL_QDMA_BSQICR_ICTT(x) ((x) & 0xffff) +#define FSL_QDMA_CQIER_MEIE 0x80000000 +#define FSL_QDMA_CQIER_TEIE 0x1 + +#define FSL_QDMA_QUEUE_MAX 8 + +#define FSL_QDMA_BCQMR_EN 0x80000000 +#define FSL_QDMA_BCQMR_EI 0x40000000 +#define FSL_QDMA_BCQMR_CD_THLD(x) ((x) << 20) +#define FSL_QDMA_BCQMR_CQ_SIZE(x) ((x) << 16) + +#define FSL_QDMA_BCQSR_QF 0x10000 + +#define FSL_QDMA_BSQMR_EN 0x80000000 +#define FSL_QDMA_BSQMR_DI 0x40000000 +#define FSL_QDMA_BSQMR_CQ_SIZE(x) ((x) << 16) + +#define FSL_QDMA_BSQSR_QE 0x20000 + +#define FSL_QDMA_DMR_DQD 0x40000000 +#define FSL_QDMA_DSR_DB 0x80000000 + +#define FSL_QDMA_BASE_BUFFER_SIZE 96 +#define FSL_QDMA_CIRCULAR_DESC_SIZE_MIN 64 +#define FSL_QDMA_CIRCULAR_DESC_SIZE_MAX 16384 +#define FSL_QDMA_QUEUE_NUM_MAX 8 + +#define FSL_QDMA_CMD_DWTTYPE_OFFSET 28 +#define FSL_QDMA_CMD_NS_OFFSET 27 +#define FSL_QDMA_CMD_DQOS_OFFSET 24 +#define FSL_QDMA_CMD_WTHROTL_OFFSET 20 +#define FSL_QDMA_CMD_DSEN_OFFSET 19 +#define FSL_QDMA_CMD_LWC_OFFSET 16 + +struct fsl_qdma_ccdf { + u8 status; + u32 resv0:22; + u32 ser:1; + u32 resv1:1; + u32 resv2:20; + u32 offset:9; + u32 format:3; + union { + struct { + u32 addr_lo; /* low 32-bits of 40-bit address */ + u32 addr_hi:8; /* high 8-bits of 40-bit address */ + u32 resv3:16; + u32 queue:3; + u32 resv4:3; + u32 dd:2; /* dynamic debug */ + }; + struct { + u64 addr:40; + /* More efficient address accessor */ + u64 __notaddress:24; + }; + }; +} __packed; + +struct fsl_qdma_csgf { + u32 offset:13; + u32 resv0:19; + u32 length:30; + u32 f:1; + u32 e:1; + union { + struct { + u32 addr_lo; /* low 32-bits of 40-bit address */ + u32 addr_hi:8; /* high 8-bits of 40-bit address */ + u32 resv1:24; + }; + struct { + u64 addr:40; + /* More efficient address accessor */ + u64 __notaddress:24; + }; + }; +} __packed; + +struct fsl_qdma_sdf { + u32 resv0:32; + u32 ssd:12; /* souce stride distance */ + u32 sss:12; /* souce stride size */ + u32 resv1:8; + u32 resv2:32; + u32 cmd; +} __packed; + +struct fsl_qdma_ddf { + u32 resv0:32; + u32 dsd:12; /* Destination stride distance */ + u32 dss:12; /* Destination stride size */ + u32 resv1:8; + u32 resv2:32; + u32 cmd; +} __packed; + +struct fsl_qdma_tcd { + u16 saddr_high; + u32 saddr; + u32 nbytes; + u16 daddr_high; + u32 daddr; +}; + +struct fsl_qdma_sw_tcd { + dma_addr_t ptcd; + struct fsl_qdma_tcd *tcd; +}; + +struct fsl_qdma_chan_config { + enum dma_transfer_direction dir; + enum dma_slave_buswidth addr_width; + u32 burst; + u32 attr; +}; + +struct fsl_qdma_chan { + struct virt_dma_chan vchan; + struct virt_dma_desc vdesc; + enum dma_status status; + u32 slave_id; + struct fsl_qdma_engine *qdma; + struct fsl_qdma_queue *queue; + struct list_head qcomp; +}; + +struct fsl_qdma_queue { + struct fsl_qdma_ccdf *virt_head; + struct fsl_qdma_ccdf *virt_tail; + struct list_head comp_used; + struct list_head comp_free; + struct dma_pool *comp_pool; + spinlock_t queue_lock; + dma_addr_t bus_addr; + u32 n_cq; + u32 id; + struct fsl_qdma_ccdf *cq; +}; + +struct fsl_qdma_comp { + dma_addr_t bus_addr; + void *virt_addr; + struct fsl_qdma_chan *qchan; + struct virt_dma_desc vdesc; + struct list_head list; +}; + +struct fsl_qdma_engine { + struct dma_device dma_dev; + void __iomem *membase; + u32 n_chans; + u32 n_queues; + struct mutex fsl_qdma_mutex; + int controller_irq; + int queue_irq; + bool big_endian; + struct fsl_qdma_queue *queue; + struct fsl_qdma_queue *status; + struct fsl_qdma_chan chans[]; + +}; + +static u32 qdma_readl(struct fsl_qdma_engine *qdma, void __iomem *addr) +{ + if (qdma->big_endian) + return ioread32be(addr); + else + return ioread32(addr); +} + +static void qdma_writel(struct fsl_qdma_engine *qdma, u32 val, + void __iomem *addr) +{ + if (qdma->big_endian) + iowrite32be(val, addr); + else + iowrite32(val, addr); +} + +static struct fsl_qdma_chan *to_fsl_qdma_chan(struct dma_chan *chan) +{ + return container_of(chan, struct fsl_qdma_chan, vchan.chan); +} + +static struct fsl_qdma_comp *to_fsl_qdma_comp(struct virt_dma_desc *vd) +{ + return container_of(vd, struct fsl_qdma_comp, vdesc); +} + +static int fsl_qdma_alloc_chan_resources(struct dma_chan *chan) +{ + /* + * In QDMA mode, We don't need to do anything. + */ + return 0; +} + +static void fsl_qdma_free_chan_resources(struct dma_chan *chan) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&fsl_chan->vchan.lock, flags); + vchan_get_all_descriptors(&fsl_chan->vchan, &head); + spin_unlock_irqrestore(&fsl_chan->vchan.lock, flags); + + vchan_dma_desc_free_list(&fsl_chan->vchan, &head); +} + +static void fsl_qdma_comp_fill(struct fsl_qdma_comp *fsl_comp, + u32 dst, u32 src, u32 len) +{ + struct fsl_qdma_ccdf *ccdf; + struct fsl_qdma_csgf *csgf_desc, *csgf_src, *csgf_dest; + struct fsl_qdma_sdf *sdf; + struct fsl_qdma_ddf *ddf; + + ccdf = (struct fsl_qdma_ccdf *)fsl_comp->virt_addr; + csgf_desc = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 1; + csgf_src = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 2; + csgf_dest = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 3; + sdf = (struct fsl_qdma_sdf *)fsl_comp->virt_addr + 4; + ddf = (struct fsl_qdma_ddf *)fsl_comp->virt_addr + 5; + + memset(fsl_comp->virt_addr, 0, FSL_QDMA_BASE_BUFFER_SIZE); + /* Head Command Descriptor(Frame Descriptor) */ + ccdf->addr = fsl_comp->bus_addr + 16; + ccdf->format = 1; /* Compound S/G format */ + ccdf->ser = 1; /* Status notification is enqueued to status queue. */ + /* Compound Command Descriptor(Frame List Table) */ + csgf_desc->addr = fsl_comp->bus_addr + 64; + csgf_desc->length = 32; /* It must be 32 as Compound S/G Descriptor */ + csgf_src->addr = src; + csgf_src->length = len; + csgf_dest->addr = dst; + csgf_dest->length = len; + csgf_dest->f = 1; /* This entry is the last entry. */ + /* Descriptor Buffer */ + sdf->cmd = 0x4 << FSL_QDMA_CMD_DWTTYPE_OFFSET; + ddf->cmd = 0x4 << FSL_QDMA_CMD_DWTTYPE_OFFSET; +} + +/* + * Request a command descriptor for enqueue. + */ +static struct fsl_qdma_comp *fsl_qdma_request_enqueue_desc( + struct fsl_qdma_chan *fsl_chan) +{ + struct fsl_qdma_comp *comp_temp; + struct fsl_qdma_queue *queue = fsl_chan->queue; + unsigned long flags; + + spin_lock_irqsave(&queue->queue_lock, flags); + if (list_empty(&queue->comp_free)) { + spin_unlock_irqrestore(&queue->queue_lock, flags); + comp_temp = kzalloc(sizeof(*comp_temp), GFP_KERNEL); + if (!comp_temp) + return NULL; + comp_temp->virt_addr = dma_pool_alloc(queue->comp_pool, + GFP_NOWAIT, + &comp_temp->bus_addr); + if (!comp_temp->virt_addr) + return NULL; + comp_temp->qchan = fsl_chan; + return comp_temp; + } + comp_temp = list_first_entry(&queue->comp_free, struct fsl_qdma_comp, + list); + comp_temp->qchan = fsl_chan; + list_del(&comp_temp->list); + spin_unlock_irqrestore(&queue->queue_lock, flags); + return comp_temp; +} + +static struct fsl_qdma_queue *fsl_qdma_alloc_queue_resources( + struct platform_device *pdev, + unsigned int queue_num) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_queue *queue_head, *queue_temp; + int ret, len, i; + unsigned int queue_size[FSL_QDMA_QUEUE_MAX]; + + if (queue_num > FSL_QDMA_QUEUE_MAX) + queue_num = FSL_QDMA_QUEUE_MAX; + len = sizeof(*queue_head) * queue_num; + queue_head = devm_kzalloc(&pdev->dev, len, GFP_KERNEL); + if (!queue_head) + return NULL; + + ret = of_property_read_u32_array(np, "queue-sizes", queue_size, + queue_num); + if (ret) { + dev_err(&pdev->dev, "Can't get queue-sizes.\n"); + return NULL; + } + + for (i = 0; i < queue_num; i++) { + if (queue_size[i] > FSL_QDMA_CIRCULAR_DESC_SIZE_MAX + || queue_size[i] < FSL_QDMA_CIRCULAR_DESC_SIZE_MIN) { + dev_err(&pdev->dev, "Get wrong queue-sizes.\n"); + return NULL; + } + queue_temp = queue_head + i; + queue_temp->cq = dma_alloc_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + queue_size[i], + &queue_temp->bus_addr, + GFP_KERNEL); + if (!queue_temp->cq) + return NULL; + queue_temp->n_cq = queue_size[i]; + queue_temp->id = i; + queue_temp->virt_head = queue_temp->cq; + queue_temp->virt_tail = queue_temp->cq; + /* + * The dma pool for queue command buffer + */ + queue_temp->comp_pool = dma_pool_create("comp_pool", + &pdev->dev, + FSL_QDMA_BASE_BUFFER_SIZE, + 16, 0); + if (!queue_temp->comp_pool) { + dma_free_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + queue_size[i], + queue_temp->cq, + queue_temp->bus_addr); + return NULL; + } + /* + * List for queue command buffer + */ + INIT_LIST_HEAD(&queue_temp->comp_used); + INIT_LIST_HEAD(&queue_temp->comp_free); + spin_lock_init(&queue_temp->queue_lock); + } + + return queue_head; +} + +static struct fsl_qdma_queue *fsl_qdma_prep_status_queue( + struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_queue *status_head; + unsigned int status_size; + int ret; + + ret = of_property_read_u32(np, "status-sizes", &status_size); + if (ret) { + dev_err(&pdev->dev, "Can't get status-sizes.\n"); + return NULL; + } + if (status_size > FSL_QDMA_CIRCULAR_DESC_SIZE_MAX + || status_size < FSL_QDMA_CIRCULAR_DESC_SIZE_MIN) { + dev_err(&pdev->dev, "Get wrong status_size.\n"); + return NULL; + } + status_head = devm_kzalloc(&pdev->dev, sizeof(*status_head), + GFP_KERNEL); + if (!status_head) + return NULL; + + /* + * Buffer for queue command + */ + status_head->cq = dma_alloc_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + status_size, + &status_head->bus_addr, + GFP_KERNEL); + if (!status_head->cq) + return NULL; + status_head->n_cq = status_size; + status_head->virt_head = status_head->cq; + status_head->virt_tail = status_head->cq; + status_head->comp_pool = NULL; + + return status_head; +} + +static int fsl_qdma_halt(struct fsl_qdma_engine *fsl_qdma) +{ + void __iomem *addr = fsl_qdma->membase; + int i, count = 5; + u32 reg; + + /* Disable the command queue and wait for idle state. */ + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DMR); + reg |= FSL_QDMA_DMR_DQD; + qdma_writel(fsl_qdma, reg, fsl_qdma->membase + FSL_QDMA_DMR); + for (i = 0; i < FSL_QDMA_QUEUE_NUM_MAX; i++) + qdma_writel(fsl_qdma, 0, addr + FSL_QDMA_BCQMR(i)); + + while (1) { + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DSR); + if (!(reg & FSL_QDMA_DSR_DB)) + break; + if (count-- < 0) + return -EBUSY; + udelay(100); + } + + /* Disable status queue. */ + qdma_writel(fsl_qdma, 0, addr + FSL_QDMA_BSQMR); + + /* + * Clear the command queue interrupt detect register for all queues. + */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_BCQIDR); + + return 0; +} + +static void fsl_qdma_queue_transfer_complete(struct fsl_qdma_engine *fsl_qdma) +{ + struct fsl_qdma_queue *fsl_queue = fsl_qdma->queue; + struct fsl_qdma_queue *fsl_status = fsl_qdma->status; + struct fsl_qdma_queue *temp_queue; + struct fsl_qdma_comp *fsl_comp; + void __iomem *addr = fsl_qdma->membase; + u32 reg, i; + u32 *status_addr; + + while (1) { + status_addr = (u32 *)fsl_status->virt_head++; + if (fsl_status->virt_head == fsl_status->cq + fsl_status->n_cq) + fsl_status->virt_head = fsl_status->cq; + /* + * Sacn all the queues. + * Match which queue completed this transfer. + */ + for (i = 0; i < fsl_qdma->n_queues; i++) { + temp_queue = fsl_queue + i; + if (list_empty(&temp_queue->comp_used)) + continue; + fsl_comp = list_first_entry(&temp_queue->comp_used, + struct fsl_qdma_comp, + list); + if (fsl_comp->bus_addr + 16 != *(u32 *)(status_addr+2)) + continue; + spin_lock(&temp_queue->queue_lock); + list_del(&fsl_comp->list); + spin_unlock(&temp_queue->queue_lock); + + reg = qdma_readl(fsl_qdma, addr + FSL_QDMA_BSQMR); + reg |= FSL_QDMA_BSQMR_DI; + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BSQMR); + + spin_lock(&fsl_comp->qchan->vchan.lock); + vchan_cookie_complete(&fsl_comp->vdesc); + fsl_comp->qchan->status = DMA_COMPLETE; + spin_unlock(&fsl_comp->qchan->vchan.lock); + break; + } + reg = qdma_readl(fsl_qdma, addr + FSL_QDMA_BSQSR); + if (reg & FSL_QDMA_BSQSR_QE) + break; + } +} + +static irqreturn_t fsl_qdma_controller_handler(int irq, void *dev_id) +{ + struct fsl_qdma_engine *fsl_qdma = dev_id; + unsigned int intr; + void __iomem *base_addr; + + base_addr = fsl_qdma->membase; + intr = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DEDR); + + if (intr) + dev_err(fsl_qdma->dma_dev.dev, + "Programming Errors! Code:0x%x\n", intr); + qdma_writel(fsl_qdma, 0xffffffff, fsl_qdma->membase + FSL_QDMA_DEDR); + + return IRQ_HANDLED; +} + +static irqreturn_t fsl_qdma_queue_handler(int irq, void *dev_id) +{ + struct fsl_qdma_engine *fsl_qdma = dev_id; + unsigned int intr; + void __iomem *base_addr; + + base_addr = fsl_qdma->membase; + intr = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_BCQIDR); + + if ((intr & FSL_QDMA_CQIDR_SQT) != 0) + fsl_qdma_queue_transfer_complete(fsl_qdma); + + qdma_writel(fsl_qdma, 0xffffffff, fsl_qdma->membase + FSL_QDMA_BCQIDR); + return IRQ_HANDLED; +} + +static int fsl_qdma_irq_init(struct platform_device *pdev, + struct fsl_qdma_engine *fsl_qdma) +{ + int ret; + + fsl_qdma->controller_irq = platform_get_irq_byname(pdev, + "qdma-controller"); + if (fsl_qdma->controller_irq < 0) { + dev_err(&pdev->dev, "Can't get qdma controller irq.\n"); + return fsl_qdma->controller_irq; + } + + fsl_qdma->queue_irq = platform_get_irq_byname(pdev, "qdma-queue"); + if (fsl_qdma->queue_irq < 0) { + dev_err(&pdev->dev, "Can't get qdma queue irq.\n"); + return fsl_qdma->queue_irq; + } + + ret = devm_request_irq(&pdev->dev, fsl_qdma->controller_irq, + fsl_qdma_controller_handler, 0, "qDMA controller", + fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't register qDMA controller IRQ.\n"); + return ret; + } + ret = devm_request_irq(&pdev->dev, fsl_qdma->queue_irq, + fsl_qdma_queue_handler, 0, "qDMA queue", fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't register qDMA queue IRQ.\n"); + return ret; + } + + return 0; +} + +static int fsl_qdma_reg_init(struct fsl_qdma_engine *fsl_qdma) +{ + struct fsl_qdma_queue *fsl_queue = fsl_qdma->queue; + struct fsl_qdma_queue *temp; + void __iomem *addr = fsl_qdma->membase; + int i, ret; + u32 reg; + + /* Try to halt the qDMA engine first. */ + ret = fsl_qdma_halt(fsl_qdma); + if (ret) { + dev_err(fsl_qdma->dma_dev.dev, "DMA halt failed!"); + return ret; + } + + /* + * Clear the command queue interrupt detect register for all queues. + */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_BCQIDR); + + for (i = 0; i < fsl_qdma->n_queues; i++) { + temp = fsl_queue + i; + /* + * Initialize Command Queue registers to point to the first + * command descriptor in memory. + * Dequeue Pointer Address Registers + * Enqueue Pointer Address Registers + */ + qdma_writel(fsl_qdma, temp->bus_addr, + addr + FSL_QDMA_CQDPA_SADDR(i)); + qdma_writel(fsl_qdma, temp->bus_addr, + addr + FSL_QDMA_CQEPA_SADDR(i)); + + /* Initialize the queue mode. */ + reg = FSL_QDMA_BCQMR_EN; + reg |= FSL_QDMA_BCQMR_CD_THLD(ilog2(temp->n_cq)-4); + reg |= FSL_QDMA_BCQMR_CQ_SIZE(ilog2(temp->n_cq)-6); + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BCQMR(i)); + } + + /* + * Initialize status queue registers to point to the first + * command descriptor in memory. + * Dequeue Pointer Address Registers + * Enqueue Pointer Address Registers + */ + qdma_writel(fsl_qdma, fsl_qdma->status->bus_addr, + addr + FSL_QDMA_SQEPAR); + qdma_writel(fsl_qdma, fsl_qdma->status->bus_addr, + addr + FSL_QDMA_SQDPAR); + /* Initialize status queue interrupt. */ + qdma_writel(fsl_qdma, FSL_QDMA_BCQIER_CQTIE, addr + FSL_QDMA_BCQIER); + qdma_writel(fsl_qdma, FSL_QDMA_BSQICR_ICEN | + FSL_QDMA_BSQICR_ICST(ilog2(16) + 1) | + FSL_QDMA_BSQICR_ICTT(0xffff), + addr + FSL_QDMA_BSQICR); + qdma_writel(fsl_qdma, FSL_QDMA_CQIER_MEIE | FSL_QDMA_CQIER_TEIE, + addr + FSL_QDMA_CQIER); + /* Initialize controller interrupt register. */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_DEDR); + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_DEIER); + + /* Set the statue queue critical watermark level. */ + qdma_writel(fsl_qdma, 0x300000, addr + FSL_QDMA_SQCCMR); + + /* Initialize the status queue mode. */ + reg = FSL_QDMA_BSQMR_EN; + reg |= FSL_QDMA_BSQMR_CQ_SIZE(ilog2(fsl_qdma->status->n_cq)-6); + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BSQMR); + + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DMR); + reg &= ~FSL_QDMA_DMR_DQD; + qdma_writel(fsl_qdma, reg, fsl_qdma->membase + FSL_QDMA_DMR); + + return 0; +} + +static struct dma_async_tx_descriptor *fsl_qdma_prep_dma_sg( + struct dma_chan *chan, + struct scatterlist *dst_sg, unsigned int dst_nents, + struct scatterlist *src_sg, unsigned int src_nents, + unsigned long flags) +{ + return NULL; +} + +static struct dma_async_tx_descriptor * +fsl_qdma_prep_memcpy(struct dma_chan *chan, dma_addr_t dst, + dma_addr_t src, size_t len, unsigned long flags) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + struct fsl_qdma_comp *fsl_comp; + + fsl_comp = fsl_qdma_request_enqueue_desc(fsl_chan); + fsl_qdma_comp_fill(fsl_comp, dst, src, len); + + return vchan_tx_prep(&fsl_chan->vchan, &fsl_comp->vdesc, flags); +} + +static void fsl_qdma_enqueue_desc(struct fsl_qdma_chan *fsl_chan) +{ + void __iomem *addr = fsl_chan->qdma->membase; + struct fsl_qdma_queue *fsl_queue = fsl_chan->queue; + struct fsl_qdma_comp *fsl_comp; + struct virt_dma_desc *vdesc; + u32 reg; + + reg = qdma_readl(fsl_chan->qdma, addr + FSL_QDMA_BCQSR(fsl_queue->id)); + if (reg & FSL_QDMA_BCQSR_QF) { + dev_err(fsl_chan->qdma->dma_dev.dev, "Enqueue rejection!\n"); + return; + } + + vdesc = vchan_next_desc(&fsl_chan->vchan); + if (!vdesc) + return; + list_del(&vdesc->node); + fsl_comp = to_fsl_qdma_comp(vdesc); + + memcpy(fsl_queue->virt_head++, fsl_comp->virt_addr, 16); + if (fsl_queue->virt_head == fsl_queue->cq + fsl_queue->n_cq) + fsl_queue->virt_head = fsl_queue->cq; + + list_add_tail(&fsl_comp->list, &fsl_queue->comp_used); + reg = qdma_readl(fsl_chan->qdma, addr + FSL_QDMA_BCQMR(fsl_queue->id)); + reg |= FSL_QDMA_BCQMR_EI; + qdma_writel(fsl_chan->qdma, reg, addr + FSL_QDMA_BCQMR(fsl_queue->id)); + fsl_chan->status = DMA_IN_PROGRESS; +} + +static enum dma_status fsl_qdma_tx_status(struct dma_chan *chan, + dma_cookie_t cookie, struct dma_tx_state *txstate) +{ + return dma_cookie_status(chan, cookie, txstate); +} + +static void fsl_qdma_free_desc(struct virt_dma_desc *vdesc) +{ + struct fsl_qdma_comp *fsl_comp; + struct fsl_qdma_queue *fsl_queue; + unsigned long flags; + + fsl_comp = to_fsl_qdma_comp(vdesc); + fsl_queue = fsl_comp->qchan->queue; + + spin_lock_irqsave(&fsl_queue->queue_lock, flags); + list_add_tail(&fsl_comp->list, &fsl_queue->comp_free); + spin_unlock_irqrestore(&fsl_queue->queue_lock, flags); +} + +static void fsl_qdma_issue_pending(struct dma_chan *chan) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + struct fsl_qdma_queue *fsl_queue = fsl_chan->queue; + unsigned long flags; + + spin_lock_irqsave(&fsl_queue->queue_lock, flags); + spin_lock(&fsl_chan->vchan.lock); + if (vchan_issue_pending(&fsl_chan->vchan)) + fsl_qdma_enqueue_desc(fsl_chan); + spin_unlock(&fsl_chan->vchan.lock); + spin_unlock_irqrestore(&fsl_queue->queue_lock, flags); +} + +static int fsl_qdma_probe(struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_engine *fsl_qdma; + struct fsl_qdma_chan *fsl_chan; + struct resource *res; + unsigned int len, chans, queues; + int ret, i; + + ret = of_property_read_u32(np, "channels", &chans); + if (ret) { + dev_err(&pdev->dev, "Can't get channels.\n"); + return ret; + } + + len = sizeof(*fsl_qdma) + sizeof(*fsl_chan) * chans; + fsl_qdma = devm_kzalloc(&pdev->dev, len, GFP_KERNEL); + if (!fsl_qdma) + return -ENOMEM; + + ret = of_property_read_u32(np, "queues", &queues); + if (ret) { + dev_err(&pdev->dev, "Can't get queues.\n"); + return ret; + } + + fsl_qdma->queue = fsl_qdma_alloc_queue_resources(pdev, queues); + if (!fsl_qdma->queue) + return -ENOMEM; + + fsl_qdma->status = fsl_qdma_prep_status_queue(pdev); + if (!fsl_qdma->status) + return -ENOMEM; + + fsl_qdma->n_chans = chans; + fsl_qdma->n_queues = queues; + mutex_init(&fsl_qdma->fsl_qdma_mutex); + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + fsl_qdma->membase = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(fsl_qdma->membase)) + return PTR_ERR(fsl_qdma->membase); + + ret = fsl_qdma_irq_init(pdev, fsl_qdma); + if (ret) + return ret; + + fsl_qdma->big_endian = of_property_read_bool(np, "big-endian"); + INIT_LIST_HEAD(&fsl_qdma->dma_dev.channels); + for (i = 0; i < fsl_qdma->n_chans; i++) { + struct fsl_qdma_chan *fsl_chan = &fsl_qdma->chans[i]; + + fsl_chan->qdma = fsl_qdma; + fsl_chan->queue = fsl_qdma->queue + i%fsl_qdma->n_queues; + fsl_chan->vchan.desc_free = fsl_qdma_free_desc; + INIT_LIST_HEAD(&fsl_chan->qcomp); + vchan_init(&fsl_chan->vchan, &fsl_qdma->dma_dev); + } + + dma_cap_set(DMA_PRIVATE, fsl_qdma->dma_dev.cap_mask); + dma_cap_set(DMA_SLAVE, fsl_qdma->dma_dev.cap_mask); + dma_cap_set(DMA_MEMCPY, fsl_qdma->dma_dev.cap_mask); + + fsl_qdma->dma_dev.dev = &pdev->dev; + fsl_qdma->dma_dev.device_alloc_chan_resources + = fsl_qdma_alloc_chan_resources; + fsl_qdma->dma_dev.device_free_chan_resources + = fsl_qdma_free_chan_resources; + fsl_qdma->dma_dev.device_tx_status = fsl_qdma_tx_status; + fsl_qdma->dma_dev.device_prep_dma_memcpy = fsl_qdma_prep_memcpy; + fsl_qdma->dma_dev.device_prep_dma_sg = fsl_qdma_prep_dma_sg; + fsl_qdma->dma_dev.device_issue_pending = fsl_qdma_issue_pending; + + platform_set_drvdata(pdev, fsl_qdma); + + ret = dma_async_device_register(&fsl_qdma->dma_dev); + if (ret) { + dev_err(&pdev->dev, "Can't register Freescale qDMA engine.\n"); + return ret; + } + + ret = fsl_qdma_reg_init(fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't Initialize the qDMA engine.\n"); + return ret; + } + + + return 0; +} + +static int fsl_qdma_remove(struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_engine *fsl_qdma = platform_get_drvdata(pdev); + struct fsl_qdma_queue *queue_temp; + struct fsl_qdma_queue *status = fsl_qdma->status; + struct fsl_qdma_comp *comp_temp, *_comp_temp; + int i; + + of_dma_controller_free(np); + dma_async_device_unregister(&fsl_qdma->dma_dev); + + /* Free descriptor areas */ + for (i = 0; i < fsl_qdma->n_queues; i++) { + queue_temp = fsl_qdma->queue + i; + list_for_each_entry_safe(comp_temp, _comp_temp, + &queue_temp->comp_used, list) { + dma_pool_free(queue_temp->comp_pool, + comp_temp->virt_addr, + comp_temp->bus_addr); + list_del(&comp_temp->list); + kfree(comp_temp); + } + list_for_each_entry_safe(comp_temp, _comp_temp, + &queue_temp->comp_free, list) { + dma_pool_free(queue_temp->comp_pool, + comp_temp->virt_addr, + comp_temp->bus_addr); + list_del(&comp_temp->list); + kfree(comp_temp); + } + dma_free_coherent(&pdev->dev, sizeof(struct fsl_qdma_ccdf) * + queue_temp->n_cq, queue_temp->cq, + queue_temp->bus_addr); + dma_pool_destroy(queue_temp->comp_pool); + } + + dma_free_coherent(&pdev->dev, sizeof(struct fsl_qdma_ccdf) * + status->n_cq, status->cq, status->bus_addr); + return 0; +} + +static const struct of_device_id fsl_qdma_dt_ids[] = { + { .compatible = "fsl,ls1021a-qdma", }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, fsl_qdma_dt_ids); + +static struct platform_driver fsl_qdma_driver = { + .driver = { + .name = "fsl-qdma", + .owner = THIS_MODULE, + .of_match_table = fsl_qdma_dt_ids, + }, + .probe = fsl_qdma_probe, + .remove = fsl_qdma_remove, +}; + +static int __init fsl_qdma_init(void) +{ + return platform_driver_register(&fsl_qdma_driver); +} +subsys_initcall(fsl_qdma_init); + +static void __exit fsl_qdma_exit(void) +{ + platform_driver_unregister(&fsl_qdma_driver); +} +module_exit(fsl_qdma_exit); + +MODULE_ALIAS("platform:fsl-qdma"); +MODULE_DESCRIPTION("Freescale qDMA engine driver"); +MODULE_LICENSE("GPL v2"); -- 2.1.0.27.g96db324 From mboxrd@z Thu Jan 1 00:00:00 1970 From: yao.yuan@freescale.com (Yuan Yao) Date: Tue, 17 Mar 2015 13:28:38 +0800 Subject: [PATCH 1/2] dma: Add Freescale qDMA engine driver support Message-ID: <1426570119-8634-1-git-send-email-yao.yuan@freescale.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Add Freescale Queue Direct Memory Access (qDMA) controller support. The qDMA supports channel virtualization by allowing DMA jobs to be enqueued into different command queues. Core can initiate a DMA transaction by preparing a command descriptor (CD) for each DMA job and enqueuing this job to a command queue. This module can be found on LS-1021 LS1043 and LS2085 SoCs. Signed-off-by: Yuan Yao --- Documentation/devicetree/bindings/dma/fsl-qdma.txt | 51 ++ drivers/dma/Kconfig | 11 + drivers/dma/Makefile | 1 + drivers/dma/fsl-qdma.c | 929 +++++++++++++++++++++ 4 files changed, 992 insertions(+) create mode 100644 Documentation/devicetree/bindings/dma/fsl-qdma.txt create mode 100644 drivers/dma/fsl-qdma.c diff --git a/Documentation/devicetree/bindings/dma/fsl-qdma.txt b/Documentation/devicetree/bindings/dma/fsl-qdma.txt new file mode 100644 index 0000000..676ae3d --- /dev/null +++ b/Documentation/devicetree/bindings/dma/fsl-qdma.txt @@ -0,0 +1,50 @@ +* Freescale queue Direct Memory Access Controller(qDMA) Controller + + The qDMA controller transfers blocks of data between one source and one or more +destinations. The blocks of data transferred can be represented in memory as contiguous +or non-contiguous using scatter/gather table(s). Channel virtualization is supported +through enqueuing of DMA jobs to, or dequeuing DMA jobs from, different work +queues. + +* qDMA Controller +Required properties: +- compatible : + - "fsl,ls1021a-qdma" for qDMA used similar to that on LS1021a SoC +- reg : Specifies base physical address(s) and size of the qDMA registers. + The region is qDMA control register's address and size. +- interrupts : A list of interrupt-specifiers, one for each entry in + interrupt-names. +- interrupt-names : Should contain: + "qdma-controller" - the controller interrupt + "qdma-queue" - the queue interrupt +- status-sizes : Number of circular status descriptor queue size +- channels : Number of channels supported by the controller +- queues : Number of queues supported by the controller +- queue-sizes : Number of circular descriptor queue size for each queue +- queue-group : The group for each queue belong to +- queue-weight : The weight for each queue belog to +- default-queue : The default queue for request a new channel + +Optional properties: +- big-endian: If present registers and hardware scatter/gather descriptors + of the qDMA are implemented in big endian mode, otherwise in little + mode. + + +Examples: + + qdma: qdma at 8390000 { + compatible = "fsl,ls1021a-qdma"; + reg = <0x0 0x8390000 0x0 0x10000>; + interrupts = , + ; + interrupt-names = "qdma-controller", "qdma-queue"; + status-sizes = <64>; + channels = <8>; + queues = <2>; + queue-sizes = <256 256>; + queue-group = <0 1>; + queue-weight = <0 0>; + default-queue = <0>; + big-endian; + }; diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig index 074ffad..fa52c9e 100644 --- a/drivers/dma/Kconfig +++ b/drivers/dma/Kconfig @@ -388,6 +388,17 @@ config FSL_EDMA multiplexing capability for DMA request sources(slot). This module can be found on Freescale Vybrid and LS-1 SoCs. +config FSL_QDMA + tristate "Freescale qDMA engine support" + depends on OF + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + help + Support the Freescale qDMA engine with command queue and legacy mode. + Channel virtualization is supported through enqueuing of DMA jobs to, + or dequeuing DMA jobs from, different work queues. + This module can be found on Freescale LS SoCs. + config XILINX_VDMA tristate "Xilinx AXI VDMA Engine" depends on (ARCH_ZYNQ || MICROBLAZE) diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile index bf44858..5f2b95e 100644 --- a/drivers/dma/Makefile +++ b/drivers/dma/Makefile @@ -46,6 +46,7 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o obj-$(CONFIG_K3_DMA) += k3dma.o obj-$(CONFIG_MOXART_DMA) += moxart-dma.o obj-$(CONFIG_FSL_EDMA) += fsl-edma.o +obj-$(CONFIG_FSL_QDMA) += fsl-qdma.o obj-$(CONFIG_QCOM_BAM_DMA) += qcom_bam_dma.o obj-y += xilinx/ obj-$(CONFIG_INTEL_MIC_X100_DMA) += mic_x100_dma.o diff --git a/drivers/dma/fsl-qdma.c b/drivers/dma/fsl-qdma.c new file mode 100644 index 0000000..00f3c33 --- /dev/null +++ b/drivers/dma/fsl-qdma.c @@ -0,0 +1,929 @@ +/* + * drivers/dma/fsl-qdma.c + * + * Copyright 2014-2015 Freescale Semiconductor, Inc. + * + * Driver for the Freescale qDMA engine with command queue and legacy mode. + * Channel virtualization is supported through enqueuing of DMA jobs to, + * or dequeuing DMA jobs from, different work queues. + * This module can be found on Freescale LS SoCs. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License as published by the + * Free Software Foundation; either version 2 of the License, or (at your + * option) any later version. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "virt-dma.h" + +#define FSL_QDMA_DMR 0x8000 +#define FSL_QDMA_DSR 0x8004 +#define FSL_QDMA_DEIER 0x9e00 +#define FSL_QDMA_DEDR 0x9e04 +#define FSL_QDMA_DECFDW0R 0x9e10 +#define FSL_QDMA_DECFDW1R 0x9e14 +#define FSL_QDMA_DECFDW2R 0x9e18 +#define FSL_QDMA_DECFDW3R 0x9e1c +#define FSL_QDMA_DECFQIDR 0x9e30 +#define FSL_QDMA_DECBR 0x9e34 + +#define FSL_QDMA_BCQMR(x) (0xa0c0 + 0x100 * (x)) +#define FSL_QDMA_BCQSR(x) (0xa0c4 + 0x100 * (x)) +#define FSL_QDMA_SQDPAR 0xa80c +#define FSL_QDMA_SQEPAR 0xa814 +#define FSL_QDMA_BSQMR 0xa800 +#define FSL_QDMA_BSQSR 0xa804 +#define FSL_QDMA_BCQIER 0xa0e0 +#define FSL_QDMA_BSQICR 0xa828 +#define FSL_QDMA_CQIER 0xaa10 +#define FSL_QDMA_SQCCMR 0xaa20 +#define FSL_QDMA_BCQIDR 0xa0e4 +#define FSL_QDMA_CQDPA_SADDR(x) (0xa0cc + 0x100 * (x)) +#define FSL_QDMA_CQEPA_SADDR(x) (0xa0d4 + 0x100 * (x)) + +#define FSL_QDMA_SQICR_ICEN + +#define FSL_QDMA_CQIDR_CQT 0xff000000 +#define FSL_QDMA_CQIDR_SQPE 0x800000 +#define FSL_QDMA_CQIDR_SQT 0x8000 + +#define FSL_QDMA_BCQIER_CQTIE 0x8000 +#define FSL_QDMA_BCQIER_CQPEIE 0x800000 +#define FSL_QDMA_BSQICR_ICEN 0x80000000 +#define FSL_QDMA_BSQICR_ICST(x) ((x) << 16) +#define FSL_QDMA_BSQICR_ICTT(x) ((x) & 0xffff) +#define FSL_QDMA_CQIER_MEIE 0x80000000 +#define FSL_QDMA_CQIER_TEIE 0x1 + +#define FSL_QDMA_QUEUE_MAX 8 + +#define FSL_QDMA_BCQMR_EN 0x80000000 +#define FSL_QDMA_BCQMR_EI 0x40000000 +#define FSL_QDMA_BCQMR_CD_THLD(x) ((x) << 20) +#define FSL_QDMA_BCQMR_CQ_SIZE(x) ((x) << 16) + +#define FSL_QDMA_BCQSR_QF 0x10000 + +#define FSL_QDMA_BSQMR_EN 0x80000000 +#define FSL_QDMA_BSQMR_DI 0x40000000 +#define FSL_QDMA_BSQMR_CQ_SIZE(x) ((x) << 16) + +#define FSL_QDMA_BSQSR_QE 0x20000 + +#define FSL_QDMA_DMR_DQD 0x40000000 +#define FSL_QDMA_DSR_DB 0x80000000 + +#define FSL_QDMA_BASE_BUFFER_SIZE 96 +#define FSL_QDMA_CIRCULAR_DESC_SIZE_MIN 64 +#define FSL_QDMA_CIRCULAR_DESC_SIZE_MAX 16384 +#define FSL_QDMA_QUEUE_NUM_MAX 8 + +#define FSL_QDMA_CMD_DWTTYPE_OFFSET 28 +#define FSL_QDMA_CMD_NS_OFFSET 27 +#define FSL_QDMA_CMD_DQOS_OFFSET 24 +#define FSL_QDMA_CMD_WTHROTL_OFFSET 20 +#define FSL_QDMA_CMD_DSEN_OFFSET 19 +#define FSL_QDMA_CMD_LWC_OFFSET 16 + +struct fsl_qdma_ccdf { + u8 status; + u32 resv0:22; + u32 ser:1; + u32 resv1:1; + u32 resv2:20; + u32 offset:9; + u32 format:3; + union { + struct { + u32 addr_lo; /* low 32-bits of 40-bit address */ + u32 addr_hi:8; /* high 8-bits of 40-bit address */ + u32 resv3:16; + u32 queue:3; + u32 resv4:3; + u32 dd:2; /* dynamic debug */ + }; + struct { + u64 addr:40; + /* More efficient address accessor */ + u64 __notaddress:24; + }; + }; +} __packed; + +struct fsl_qdma_csgf { + u32 offset:13; + u32 resv0:19; + u32 length:30; + u32 f:1; + u32 e:1; + union { + struct { + u32 addr_lo; /* low 32-bits of 40-bit address */ + u32 addr_hi:8; /* high 8-bits of 40-bit address */ + u32 resv1:24; + }; + struct { + u64 addr:40; + /* More efficient address accessor */ + u64 __notaddress:24; + }; + }; +} __packed; + +struct fsl_qdma_sdf { + u32 resv0:32; + u32 ssd:12; /* souce stride distance */ + u32 sss:12; /* souce stride size */ + u32 resv1:8; + u32 resv2:32; + u32 cmd; +} __packed; + +struct fsl_qdma_ddf { + u32 resv0:32; + u32 dsd:12; /* Destination stride distance */ + u32 dss:12; /* Destination stride size */ + u32 resv1:8; + u32 resv2:32; + u32 cmd; +} __packed; + +struct fsl_qdma_tcd { + u16 saddr_high; + u32 saddr; + u32 nbytes; + u16 daddr_high; + u32 daddr; +}; + +struct fsl_qdma_sw_tcd { + dma_addr_t ptcd; + struct fsl_qdma_tcd *tcd; +}; + +struct fsl_qdma_chan_config { + enum dma_transfer_direction dir; + enum dma_slave_buswidth addr_width; + u32 burst; + u32 attr; +}; + +struct fsl_qdma_chan { + struct virt_dma_chan vchan; + struct virt_dma_desc vdesc; + enum dma_status status; + u32 slave_id; + struct fsl_qdma_engine *qdma; + struct fsl_qdma_queue *queue; + struct list_head qcomp; +}; + +struct fsl_qdma_queue { + struct fsl_qdma_ccdf *virt_head; + struct fsl_qdma_ccdf *virt_tail; + struct list_head comp_used; + struct list_head comp_free; + struct dma_pool *comp_pool; + spinlock_t queue_lock; + dma_addr_t bus_addr; + u32 n_cq; + u32 id; + struct fsl_qdma_ccdf *cq; +}; + +struct fsl_qdma_comp { + dma_addr_t bus_addr; + void *virt_addr; + struct fsl_qdma_chan *qchan; + struct virt_dma_desc vdesc; + struct list_head list; +}; + +struct fsl_qdma_engine { + struct dma_device dma_dev; + void __iomem *membase; + u32 n_chans; + u32 n_queues; + struct mutex fsl_qdma_mutex; + int controller_irq; + int queue_irq; + bool big_endian; + struct fsl_qdma_queue *queue; + struct fsl_qdma_queue *status; + struct fsl_qdma_chan chans[]; + +}; + +static u32 qdma_readl(struct fsl_qdma_engine *qdma, void __iomem *addr) +{ + if (qdma->big_endian) + return ioread32be(addr); + else + return ioread32(addr); +} + +static void qdma_writel(struct fsl_qdma_engine *qdma, u32 val, + void __iomem *addr) +{ + if (qdma->big_endian) + iowrite32be(val, addr); + else + iowrite32(val, addr); +} + +static struct fsl_qdma_chan *to_fsl_qdma_chan(struct dma_chan *chan) +{ + return container_of(chan, struct fsl_qdma_chan, vchan.chan); +} + +static struct fsl_qdma_comp *to_fsl_qdma_comp(struct virt_dma_desc *vd) +{ + return container_of(vd, struct fsl_qdma_comp, vdesc); +} + +static int fsl_qdma_alloc_chan_resources(struct dma_chan *chan) +{ + /* + * In QDMA mode, We don't need to do anything. + */ + return 0; +} + +static void fsl_qdma_free_chan_resources(struct dma_chan *chan) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&fsl_chan->vchan.lock, flags); + vchan_get_all_descriptors(&fsl_chan->vchan, &head); + spin_unlock_irqrestore(&fsl_chan->vchan.lock, flags); + + vchan_dma_desc_free_list(&fsl_chan->vchan, &head); +} + +static void fsl_qdma_comp_fill(struct fsl_qdma_comp *fsl_comp, + u32 dst, u32 src, u32 len) +{ + struct fsl_qdma_ccdf *ccdf; + struct fsl_qdma_csgf *csgf_desc, *csgf_src, *csgf_dest; + struct fsl_qdma_sdf *sdf; + struct fsl_qdma_ddf *ddf; + + ccdf = (struct fsl_qdma_ccdf *)fsl_comp->virt_addr; + csgf_desc = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 1; + csgf_src = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 2; + csgf_dest = (struct fsl_qdma_csgf *)fsl_comp->virt_addr + 3; + sdf = (struct fsl_qdma_sdf *)fsl_comp->virt_addr + 4; + ddf = (struct fsl_qdma_ddf *)fsl_comp->virt_addr + 5; + + memset(fsl_comp->virt_addr, 0, FSL_QDMA_BASE_BUFFER_SIZE); + /* Head Command Descriptor(Frame Descriptor) */ + ccdf->addr = fsl_comp->bus_addr + 16; + ccdf->format = 1; /* Compound S/G format */ + ccdf->ser = 1; /* Status notification is enqueued to status queue. */ + /* Compound Command Descriptor(Frame List Table) */ + csgf_desc->addr = fsl_comp->bus_addr + 64; + csgf_desc->length = 32; /* It must be 32 as Compound S/G Descriptor */ + csgf_src->addr = src; + csgf_src->length = len; + csgf_dest->addr = dst; + csgf_dest->length = len; + csgf_dest->f = 1; /* This entry is the last entry. */ + /* Descriptor Buffer */ + sdf->cmd = 0x4 << FSL_QDMA_CMD_DWTTYPE_OFFSET; + ddf->cmd = 0x4 << FSL_QDMA_CMD_DWTTYPE_OFFSET; +} + +/* + * Request a command descriptor for enqueue. + */ +static struct fsl_qdma_comp *fsl_qdma_request_enqueue_desc( + struct fsl_qdma_chan *fsl_chan) +{ + struct fsl_qdma_comp *comp_temp; + struct fsl_qdma_queue *queue = fsl_chan->queue; + unsigned long flags; + + spin_lock_irqsave(&queue->queue_lock, flags); + if (list_empty(&queue->comp_free)) { + spin_unlock_irqrestore(&queue->queue_lock, flags); + comp_temp = kzalloc(sizeof(*comp_temp), GFP_KERNEL); + if (!comp_temp) + return NULL; + comp_temp->virt_addr = dma_pool_alloc(queue->comp_pool, + GFP_NOWAIT, + &comp_temp->bus_addr); + if (!comp_temp->virt_addr) + return NULL; + comp_temp->qchan = fsl_chan; + return comp_temp; + } + comp_temp = list_first_entry(&queue->comp_free, struct fsl_qdma_comp, + list); + comp_temp->qchan = fsl_chan; + list_del(&comp_temp->list); + spin_unlock_irqrestore(&queue->queue_lock, flags); + return comp_temp; +} + +static struct fsl_qdma_queue *fsl_qdma_alloc_queue_resources( + struct platform_device *pdev, + unsigned int queue_num) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_queue *queue_head, *queue_temp; + int ret, len, i; + unsigned int queue_size[FSL_QDMA_QUEUE_MAX]; + + if (queue_num > FSL_QDMA_QUEUE_MAX) + queue_num = FSL_QDMA_QUEUE_MAX; + len = sizeof(*queue_head) * queue_num; + queue_head = devm_kzalloc(&pdev->dev, len, GFP_KERNEL); + if (!queue_head) + return NULL; + + ret = of_property_read_u32_array(np, "queue-sizes", queue_size, + queue_num); + if (ret) { + dev_err(&pdev->dev, "Can't get queue-sizes.\n"); + return NULL; + } + + for (i = 0; i < queue_num; i++) { + if (queue_size[i] > FSL_QDMA_CIRCULAR_DESC_SIZE_MAX + || queue_size[i] < FSL_QDMA_CIRCULAR_DESC_SIZE_MIN) { + dev_err(&pdev->dev, "Get wrong queue-sizes.\n"); + return NULL; + } + queue_temp = queue_head + i; + queue_temp->cq = dma_alloc_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + queue_size[i], + &queue_temp->bus_addr, + GFP_KERNEL); + if (!queue_temp->cq) + return NULL; + queue_temp->n_cq = queue_size[i]; + queue_temp->id = i; + queue_temp->virt_head = queue_temp->cq; + queue_temp->virt_tail = queue_temp->cq; + /* + * The dma pool for queue command buffer + */ + queue_temp->comp_pool = dma_pool_create("comp_pool", + &pdev->dev, + FSL_QDMA_BASE_BUFFER_SIZE, + 16, 0); + if (!queue_temp->comp_pool) { + dma_free_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + queue_size[i], + queue_temp->cq, + queue_temp->bus_addr); + return NULL; + } + /* + * List for queue command buffer + */ + INIT_LIST_HEAD(&queue_temp->comp_used); + INIT_LIST_HEAD(&queue_temp->comp_free); + spin_lock_init(&queue_temp->queue_lock); + } + + return queue_head; +} + +static struct fsl_qdma_queue *fsl_qdma_prep_status_queue( + struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_queue *status_head; + unsigned int status_size; + int ret; + + ret = of_property_read_u32(np, "status-sizes", &status_size); + if (ret) { + dev_err(&pdev->dev, "Can't get status-sizes.\n"); + return NULL; + } + if (status_size > FSL_QDMA_CIRCULAR_DESC_SIZE_MAX + || status_size < FSL_QDMA_CIRCULAR_DESC_SIZE_MIN) { + dev_err(&pdev->dev, "Get wrong status_size.\n"); + return NULL; + } + status_head = devm_kzalloc(&pdev->dev, sizeof(*status_head), + GFP_KERNEL); + if (!status_head) + return NULL; + + /* + * Buffer for queue command + */ + status_head->cq = dma_alloc_coherent(&pdev->dev, + sizeof(struct fsl_qdma_ccdf) * + status_size, + &status_head->bus_addr, + GFP_KERNEL); + if (!status_head->cq) + return NULL; + status_head->n_cq = status_size; + status_head->virt_head = status_head->cq; + status_head->virt_tail = status_head->cq; + status_head->comp_pool = NULL; + + return status_head; +} + +static int fsl_qdma_halt(struct fsl_qdma_engine *fsl_qdma) +{ + void __iomem *addr = fsl_qdma->membase; + int i, count = 5; + u32 reg; + + /* Disable the command queue and wait for idle state. */ + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DMR); + reg |= FSL_QDMA_DMR_DQD; + qdma_writel(fsl_qdma, reg, fsl_qdma->membase + FSL_QDMA_DMR); + for (i = 0; i < FSL_QDMA_QUEUE_NUM_MAX; i++) + qdma_writel(fsl_qdma, 0, addr + FSL_QDMA_BCQMR(i)); + + while (1) { + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DSR); + if (!(reg & FSL_QDMA_DSR_DB)) + break; + if (count-- < 0) + return -EBUSY; + udelay(100); + } + + /* Disable status queue. */ + qdma_writel(fsl_qdma, 0, addr + FSL_QDMA_BSQMR); + + /* + * Clear the command queue interrupt detect register for all queues. + */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_BCQIDR); + + return 0; +} + +static void fsl_qdma_queue_transfer_complete(struct fsl_qdma_engine *fsl_qdma) +{ + struct fsl_qdma_queue *fsl_queue = fsl_qdma->queue; + struct fsl_qdma_queue *fsl_status = fsl_qdma->status; + struct fsl_qdma_queue *temp_queue; + struct fsl_qdma_comp *fsl_comp; + void __iomem *addr = fsl_qdma->membase; + u32 reg, i; + u32 *status_addr; + + while (1) { + status_addr = (u32 *)fsl_status->virt_head++; + if (fsl_status->virt_head == fsl_status->cq + fsl_status->n_cq) + fsl_status->virt_head = fsl_status->cq; + /* + * Sacn all the queues. + * Match which queue completed this transfer. + */ + for (i = 0; i < fsl_qdma->n_queues; i++) { + temp_queue = fsl_queue + i; + if (list_empty(&temp_queue->comp_used)) + continue; + fsl_comp = list_first_entry(&temp_queue->comp_used, + struct fsl_qdma_comp, + list); + if (fsl_comp->bus_addr + 16 != *(u32 *)(status_addr+2)) + continue; + spin_lock(&temp_queue->queue_lock); + list_del(&fsl_comp->list); + spin_unlock(&temp_queue->queue_lock); + + reg = qdma_readl(fsl_qdma, addr + FSL_QDMA_BSQMR); + reg |= FSL_QDMA_BSQMR_DI; + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BSQMR); + + spin_lock(&fsl_comp->qchan->vchan.lock); + vchan_cookie_complete(&fsl_comp->vdesc); + fsl_comp->qchan->status = DMA_COMPLETE; + spin_unlock(&fsl_comp->qchan->vchan.lock); + break; + } + reg = qdma_readl(fsl_qdma, addr + FSL_QDMA_BSQSR); + if (reg & FSL_QDMA_BSQSR_QE) + break; + } +} + +static irqreturn_t fsl_qdma_controller_handler(int irq, void *dev_id) +{ + struct fsl_qdma_engine *fsl_qdma = dev_id; + unsigned int intr; + void __iomem *base_addr; + + base_addr = fsl_qdma->membase; + intr = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DEDR); + + if (intr) + dev_err(fsl_qdma->dma_dev.dev, + "Programming Errors! Code:0x%x\n", intr); + qdma_writel(fsl_qdma, 0xffffffff, fsl_qdma->membase + FSL_QDMA_DEDR); + + return IRQ_HANDLED; +} + +static irqreturn_t fsl_qdma_queue_handler(int irq, void *dev_id) +{ + struct fsl_qdma_engine *fsl_qdma = dev_id; + unsigned int intr; + void __iomem *base_addr; + + base_addr = fsl_qdma->membase; + intr = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_BCQIDR); + + if ((intr & FSL_QDMA_CQIDR_SQT) != 0) + fsl_qdma_queue_transfer_complete(fsl_qdma); + + qdma_writel(fsl_qdma, 0xffffffff, fsl_qdma->membase + FSL_QDMA_BCQIDR); + return IRQ_HANDLED; +} + +static int fsl_qdma_irq_init(struct platform_device *pdev, + struct fsl_qdma_engine *fsl_qdma) +{ + int ret; + + fsl_qdma->controller_irq = platform_get_irq_byname(pdev, + "qdma-controller"); + if (fsl_qdma->controller_irq < 0) { + dev_err(&pdev->dev, "Can't get qdma controller irq.\n"); + return fsl_qdma->controller_irq; + } + + fsl_qdma->queue_irq = platform_get_irq_byname(pdev, "qdma-queue"); + if (fsl_qdma->queue_irq < 0) { + dev_err(&pdev->dev, "Can't get qdma queue irq.\n"); + return fsl_qdma->queue_irq; + } + + ret = devm_request_irq(&pdev->dev, fsl_qdma->controller_irq, + fsl_qdma_controller_handler, 0, "qDMA controller", + fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't register qDMA controller IRQ.\n"); + return ret; + } + ret = devm_request_irq(&pdev->dev, fsl_qdma->queue_irq, + fsl_qdma_queue_handler, 0, "qDMA queue", fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't register qDMA queue IRQ.\n"); + return ret; + } + + return 0; +} + +static int fsl_qdma_reg_init(struct fsl_qdma_engine *fsl_qdma) +{ + struct fsl_qdma_queue *fsl_queue = fsl_qdma->queue; + struct fsl_qdma_queue *temp; + void __iomem *addr = fsl_qdma->membase; + int i, ret; + u32 reg; + + /* Try to halt the qDMA engine first. */ + ret = fsl_qdma_halt(fsl_qdma); + if (ret) { + dev_err(fsl_qdma->dma_dev.dev, "DMA halt failed!"); + return ret; + } + + /* + * Clear the command queue interrupt detect register for all queues. + */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_BCQIDR); + + for (i = 0; i < fsl_qdma->n_queues; i++) { + temp = fsl_queue + i; + /* + * Initialize Command Queue registers to point to the first + * command descriptor in memory. + * Dequeue Pointer Address Registers + * Enqueue Pointer Address Registers + */ + qdma_writel(fsl_qdma, temp->bus_addr, + addr + FSL_QDMA_CQDPA_SADDR(i)); + qdma_writel(fsl_qdma, temp->bus_addr, + addr + FSL_QDMA_CQEPA_SADDR(i)); + + /* Initialize the queue mode. */ + reg = FSL_QDMA_BCQMR_EN; + reg |= FSL_QDMA_BCQMR_CD_THLD(ilog2(temp->n_cq)-4); + reg |= FSL_QDMA_BCQMR_CQ_SIZE(ilog2(temp->n_cq)-6); + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BCQMR(i)); + } + + /* + * Initialize status queue registers to point to the first + * command descriptor in memory. + * Dequeue Pointer Address Registers + * Enqueue Pointer Address Registers + */ + qdma_writel(fsl_qdma, fsl_qdma->status->bus_addr, + addr + FSL_QDMA_SQEPAR); + qdma_writel(fsl_qdma, fsl_qdma->status->bus_addr, + addr + FSL_QDMA_SQDPAR); + /* Initialize status queue interrupt. */ + qdma_writel(fsl_qdma, FSL_QDMA_BCQIER_CQTIE, addr + FSL_QDMA_BCQIER); + qdma_writel(fsl_qdma, FSL_QDMA_BSQICR_ICEN | + FSL_QDMA_BSQICR_ICST(ilog2(16) + 1) | + FSL_QDMA_BSQICR_ICTT(0xffff), + addr + FSL_QDMA_BSQICR); + qdma_writel(fsl_qdma, FSL_QDMA_CQIER_MEIE | FSL_QDMA_CQIER_TEIE, + addr + FSL_QDMA_CQIER); + /* Initialize controller interrupt register. */ + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_DEDR); + qdma_writel(fsl_qdma, 0xffffffff, addr + FSL_QDMA_DEIER); + + /* Set the statue queue critical watermark level. */ + qdma_writel(fsl_qdma, 0x300000, addr + FSL_QDMA_SQCCMR); + + /* Initialize the status queue mode. */ + reg = FSL_QDMA_BSQMR_EN; + reg |= FSL_QDMA_BSQMR_CQ_SIZE(ilog2(fsl_qdma->status->n_cq)-6); + qdma_writel(fsl_qdma, reg, addr + FSL_QDMA_BSQMR); + + reg = qdma_readl(fsl_qdma, fsl_qdma->membase + FSL_QDMA_DMR); + reg &= ~FSL_QDMA_DMR_DQD; + qdma_writel(fsl_qdma, reg, fsl_qdma->membase + FSL_QDMA_DMR); + + return 0; +} + +static struct dma_async_tx_descriptor *fsl_qdma_prep_dma_sg( + struct dma_chan *chan, + struct scatterlist *dst_sg, unsigned int dst_nents, + struct scatterlist *src_sg, unsigned int src_nents, + unsigned long flags) +{ + return NULL; +} + +static struct dma_async_tx_descriptor * +fsl_qdma_prep_memcpy(struct dma_chan *chan, dma_addr_t dst, + dma_addr_t src, size_t len, unsigned long flags) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + struct fsl_qdma_comp *fsl_comp; + + fsl_comp = fsl_qdma_request_enqueue_desc(fsl_chan); + fsl_qdma_comp_fill(fsl_comp, dst, src, len); + + return vchan_tx_prep(&fsl_chan->vchan, &fsl_comp->vdesc, flags); +} + +static void fsl_qdma_enqueue_desc(struct fsl_qdma_chan *fsl_chan) +{ + void __iomem *addr = fsl_chan->qdma->membase; + struct fsl_qdma_queue *fsl_queue = fsl_chan->queue; + struct fsl_qdma_comp *fsl_comp; + struct virt_dma_desc *vdesc; + u32 reg; + + reg = qdma_readl(fsl_chan->qdma, addr + FSL_QDMA_BCQSR(fsl_queue->id)); + if (reg & FSL_QDMA_BCQSR_QF) { + dev_err(fsl_chan->qdma->dma_dev.dev, "Enqueue rejection!\n"); + return; + } + + vdesc = vchan_next_desc(&fsl_chan->vchan); + if (!vdesc) + return; + list_del(&vdesc->node); + fsl_comp = to_fsl_qdma_comp(vdesc); + + memcpy(fsl_queue->virt_head++, fsl_comp->virt_addr, 16); + if (fsl_queue->virt_head == fsl_queue->cq + fsl_queue->n_cq) + fsl_queue->virt_head = fsl_queue->cq; + + list_add_tail(&fsl_comp->list, &fsl_queue->comp_used); + reg = qdma_readl(fsl_chan->qdma, addr + FSL_QDMA_BCQMR(fsl_queue->id)); + reg |= FSL_QDMA_BCQMR_EI; + qdma_writel(fsl_chan->qdma, reg, addr + FSL_QDMA_BCQMR(fsl_queue->id)); + fsl_chan->status = DMA_IN_PROGRESS; +} + +static enum dma_status fsl_qdma_tx_status(struct dma_chan *chan, + dma_cookie_t cookie, struct dma_tx_state *txstate) +{ + return dma_cookie_status(chan, cookie, txstate); +} + +static void fsl_qdma_free_desc(struct virt_dma_desc *vdesc) +{ + struct fsl_qdma_comp *fsl_comp; + struct fsl_qdma_queue *fsl_queue; + unsigned long flags; + + fsl_comp = to_fsl_qdma_comp(vdesc); + fsl_queue = fsl_comp->qchan->queue; + + spin_lock_irqsave(&fsl_queue->queue_lock, flags); + list_add_tail(&fsl_comp->list, &fsl_queue->comp_free); + spin_unlock_irqrestore(&fsl_queue->queue_lock, flags); +} + +static void fsl_qdma_issue_pending(struct dma_chan *chan) +{ + struct fsl_qdma_chan *fsl_chan = to_fsl_qdma_chan(chan); + struct fsl_qdma_queue *fsl_queue = fsl_chan->queue; + unsigned long flags; + + spin_lock_irqsave(&fsl_queue->queue_lock, flags); + spin_lock(&fsl_chan->vchan.lock); + if (vchan_issue_pending(&fsl_chan->vchan)) + fsl_qdma_enqueue_desc(fsl_chan); + spin_unlock(&fsl_chan->vchan.lock); + spin_unlock_irqrestore(&fsl_queue->queue_lock, flags); +} + +static int fsl_qdma_probe(struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_engine *fsl_qdma; + struct fsl_qdma_chan *fsl_chan; + struct resource *res; + unsigned int len, chans, queues; + int ret, i; + + ret = of_property_read_u32(np, "channels", &chans); + if (ret) { + dev_err(&pdev->dev, "Can't get channels.\n"); + return ret; + } + + len = sizeof(*fsl_qdma) + sizeof(*fsl_chan) * chans; + fsl_qdma = devm_kzalloc(&pdev->dev, len, GFP_KERNEL); + if (!fsl_qdma) + return -ENOMEM; + + ret = of_property_read_u32(np, "queues", &queues); + if (ret) { + dev_err(&pdev->dev, "Can't get queues.\n"); + return ret; + } + + fsl_qdma->queue = fsl_qdma_alloc_queue_resources(pdev, queues); + if (!fsl_qdma->queue) + return -ENOMEM; + + fsl_qdma->status = fsl_qdma_prep_status_queue(pdev); + if (!fsl_qdma->status) + return -ENOMEM; + + fsl_qdma->n_chans = chans; + fsl_qdma->n_queues = queues; + mutex_init(&fsl_qdma->fsl_qdma_mutex); + + res = platform_get_resource(pdev, IORESOURCE_MEM, 0); + fsl_qdma->membase = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(fsl_qdma->membase)) + return PTR_ERR(fsl_qdma->membase); + + ret = fsl_qdma_irq_init(pdev, fsl_qdma); + if (ret) + return ret; + + fsl_qdma->big_endian = of_property_read_bool(np, "big-endian"); + INIT_LIST_HEAD(&fsl_qdma->dma_dev.channels); + for (i = 0; i < fsl_qdma->n_chans; i++) { + struct fsl_qdma_chan *fsl_chan = &fsl_qdma->chans[i]; + + fsl_chan->qdma = fsl_qdma; + fsl_chan->queue = fsl_qdma->queue + i%fsl_qdma->n_queues; + fsl_chan->vchan.desc_free = fsl_qdma_free_desc; + INIT_LIST_HEAD(&fsl_chan->qcomp); + vchan_init(&fsl_chan->vchan, &fsl_qdma->dma_dev); + } + + dma_cap_set(DMA_PRIVATE, fsl_qdma->dma_dev.cap_mask); + dma_cap_set(DMA_SLAVE, fsl_qdma->dma_dev.cap_mask); + dma_cap_set(DMA_MEMCPY, fsl_qdma->dma_dev.cap_mask); + + fsl_qdma->dma_dev.dev = &pdev->dev; + fsl_qdma->dma_dev.device_alloc_chan_resources + = fsl_qdma_alloc_chan_resources; + fsl_qdma->dma_dev.device_free_chan_resources + = fsl_qdma_free_chan_resources; + fsl_qdma->dma_dev.device_tx_status = fsl_qdma_tx_status; + fsl_qdma->dma_dev.device_prep_dma_memcpy = fsl_qdma_prep_memcpy; + fsl_qdma->dma_dev.device_prep_dma_sg = fsl_qdma_prep_dma_sg; + fsl_qdma->dma_dev.device_issue_pending = fsl_qdma_issue_pending; + + platform_set_drvdata(pdev, fsl_qdma); + + ret = dma_async_device_register(&fsl_qdma->dma_dev); + if (ret) { + dev_err(&pdev->dev, "Can't register Freescale qDMA engine.\n"); + return ret; + } + + ret = fsl_qdma_reg_init(fsl_qdma); + if (ret) { + dev_err(&pdev->dev, "Can't Initialize the qDMA engine.\n"); + return ret; + } + + + return 0; +} + +static int fsl_qdma_remove(struct platform_device *pdev) +{ + struct device_node *np = pdev->dev.of_node; + struct fsl_qdma_engine *fsl_qdma = platform_get_drvdata(pdev); + struct fsl_qdma_queue *queue_temp; + struct fsl_qdma_queue *status = fsl_qdma->status; + struct fsl_qdma_comp *comp_temp, *_comp_temp; + int i; + + of_dma_controller_free(np); + dma_async_device_unregister(&fsl_qdma->dma_dev); + + /* Free descriptor areas */ + for (i = 0; i < fsl_qdma->n_queues; i++) { + queue_temp = fsl_qdma->queue + i; + list_for_each_entry_safe(comp_temp, _comp_temp, + &queue_temp->comp_used, list) { + dma_pool_free(queue_temp->comp_pool, + comp_temp->virt_addr, + comp_temp->bus_addr); + list_del(&comp_temp->list); + kfree(comp_temp); + } + list_for_each_entry_safe(comp_temp, _comp_temp, + &queue_temp->comp_free, list) { + dma_pool_free(queue_temp->comp_pool, + comp_temp->virt_addr, + comp_temp->bus_addr); + list_del(&comp_temp->list); + kfree(comp_temp); + } + dma_free_coherent(&pdev->dev, sizeof(struct fsl_qdma_ccdf) * + queue_temp->n_cq, queue_temp->cq, + queue_temp->bus_addr); + dma_pool_destroy(queue_temp->comp_pool); + } + + dma_free_coherent(&pdev->dev, sizeof(struct fsl_qdma_ccdf) * + status->n_cq, status->cq, status->bus_addr); + return 0; +} + +static const struct of_device_id fsl_qdma_dt_ids[] = { + { .compatible = "fsl,ls1021a-qdma", }, + { /* sentinel */ } +}; +MODULE_DEVICE_TABLE(of, fsl_qdma_dt_ids); + +static struct platform_driver fsl_qdma_driver = { + .driver = { + .name = "fsl-qdma", + .owner = THIS_MODULE, + .of_match_table = fsl_qdma_dt_ids, + }, + .probe = fsl_qdma_probe, + .remove = fsl_qdma_remove, +}; + +static int __init fsl_qdma_init(void) +{ + return platform_driver_register(&fsl_qdma_driver); +} +subsys_initcall(fsl_qdma_init); + +static void __exit fsl_qdma_exit(void) +{ + platform_driver_unregister(&fsl_qdma_driver); +} +module_exit(fsl_qdma_exit); + +MODULE_ALIAS("platform:fsl-qdma"); +MODULE_DESCRIPTION("Freescale qDMA engine driver"); +MODULE_LICENSE("GPL v2"); -- 2.1.0.27.g96db324