linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jiho Chu <jiho.chu@samsung.com>
To: gregkh@linuxfoundation.org, arnd@arndb.de, linux-kernel@vger.kernel.org
Cc: yelini.jeong@samsung.com, myungjoo.ham@samsung.com,
	Jiho Chu <jiho.chu@samsung.com>
Subject: [PATCH 0/9] Samsung Trinity NPU device driver
Date: Mon, 25 Jul 2022 15:52:59 +0900	[thread overview]
Message-ID: <20220725065308.2457024-1-jiho.chu@samsung.com> (raw)
In-Reply-To: <CGME20220725065308epcas1p2f6de3d74792854bb312cca4b310badac@epcas1p2.samsung.com>

Hello,

My name is Jiho Chu, and working for device driver and system daemon for
several years at Samsung Electronics. 

Trinity Neural Processing Unit (NPU) series are hardware accelerators
for neural network processing in embedded systems, which are integrated
into application processors or SoCs. Trinity NPU is compatible with AMBA
bus architecture and first launched in 2018 with its first version for
vision processing, Trinity Version1 (TRIV1). Its second version, TRIV2,
is released in Dec, 2021. Another Trinity NPU for audio processing is
referred as TRIA.

TRIV2 is shipped for many models of 2022 Samsung TVs, providing
acceleration for various AI-based applications, which include image
recognition and picture quality improvements for streaming video, which
can be accessed via GStreamer and its neural network plugins,
NNStreamer.

In this patch set, it includes Trinity Vision 2 kernel device driver.
Trinity Vision 2 supports accelerating image inference process for
Convolution Neural Network (CNN). The CNN workload is executed by Deep
Learning Accelerator (DLA), and general Neural Network Layers are
executed by Digital Signal Processor (DSP). And there is a Control
Processor (CP) which can control DLA and DSP. These three IPs (DLA, DSP,
CP) are composing Trinity Vision 2 NPU, and the device driver mainly
supervise the CP to manage entire NPU.

Controlling DLA and DSP operations is performed with internal command
instructions. and the instructions for the Trinity is similar with
general processor's ISA, but it is specialized for Neural Processing
operations. The virtual ISA (vISA) is designed for calculating multiple
data with single operation, like modern SIMD processor. The device
driver loads a program to CP at start up, and the program can decode a
binary which is built with the vISA. We calls this decoding program as a
Instruction Decoding Unit (IDU) program. While running the NPU, the CP
executes IDU program to fetch and decode instructions which made up of
vISA, by the scheduling policy of the device driver.

These DLA, DSP and CP are loosely coupled using ARM's AMBA, so the
Trinity can easily communicate with most ARM processors. Each IPs
designed to have memory-mapped registers which can be used to control
the IP, and the CP provides Wait-For-Event (WFE) operation to subscribe
interrupt signals from the DLA and DSP. Also, embedded Direct Memory
Access Controller (DMAC) manages data communications between internal
SRAM and outer main memory, IOMMU module supports unified memory space.

A user can control the Trinity NPU with IOCTLs provided by driver. These
controls includes memory management operations to transfer model data
(HWMEM_ALLOC/HWMEM_DEALLOC), NPU workload control operations to submit
workload (RUN/STOP), and statistics operations to check current NPU
status. (STAT)

The device driver also implemented features for developers. It provides
sysfs control attributes like stop, suspend, sched_test, and profile.
Also, it provides status attributes like app status, a number of total
requests, a number of active requests and memory usages. For the tracing
operations, several ftrace events are defined and embedded for several
important points.

I would highly appreciate your feedback.
Review, question or anythings.


Thanks.
Jiho Chu

Jiho Chu (9):
  trinity: Add base driver
  tirnity: Add dma memory module
  trinity: Add load/unload IDU files
  trinity: Add schduler module
  trinity: Add sysfs debugfs module
  trinity: Add pm and ioctl feature
  trinity: Add profile module
  trinity: Add trace module
  MAINTAINERS: add TRINITY driver

 MAINTAINERS                                   |    7 +
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/trinity/Kconfig                  |   27 +
 drivers/misc/trinity/Makefile                 |   12 +
 drivers/misc/trinity/sched/core.c             |  170 ++
 drivers/misc/trinity/sched/priority.c         |  335 +++
 drivers/misc/trinity/sched/priority.h         |   18 +
 drivers/misc/trinity/sched/sched.h            |   52 +
 drivers/misc/trinity/trinity.c                | 1282 +++++++++++
 drivers/misc/trinity/trinity_common.h         |  434 ++++
 drivers/misc/trinity/trinity_debug.c          |  358 ++++
 drivers/misc/trinity/trinity_hwmem.c          |  438 ++++
 drivers/misc/trinity/trinity_hwmem.h          |   45 +
 drivers/misc/trinity/trinity_pm.c             |   76 +
 drivers/misc/trinity/trinity_resv_mem.c       |  264 +++
 drivers/misc/trinity/trinity_resv_mem.h       |   41 +
 drivers/misc/trinity/trinity_stat.c           |  893 ++++++++
 drivers/misc/trinity/trinity_stat.h           |   56 +
 drivers/misc/trinity/trinity_sysfs.c          |  864 ++++++++
 drivers/misc/trinity/trinity_trace.c          |   15 +
 drivers/misc/trinity/trinity_trace.h          |  406 ++++
 drivers/misc/trinity/trinity_vision2_drv.c    | 1893 +++++++++++++++++
 .../misc/trinity/trinity_vision2_profile.h    |  324 +++
 drivers/misc/trinity/trinity_vision2_regs.h   |  210 ++
 include/uapi/misc/trinity.h                   |  458 ++++
 26 files changed, 8680 insertions(+)
 create mode 100644 drivers/misc/trinity/Kconfig
 create mode 100644 drivers/misc/trinity/Makefile
 create mode 100644 drivers/misc/trinity/sched/core.c
 create mode 100644 drivers/misc/trinity/sched/priority.c
 create mode 100644 drivers/misc/trinity/sched/priority.h
 create mode 100644 drivers/misc/trinity/sched/sched.h
 create mode 100644 drivers/misc/trinity/trinity.c
 create mode 100644 drivers/misc/trinity/trinity_common.h
 create mode 100644 drivers/misc/trinity/trinity_debug.c
 create mode 100644 drivers/misc/trinity/trinity_hwmem.c
 create mode 100644 drivers/misc/trinity/trinity_hwmem.h
 create mode 100644 drivers/misc/trinity/trinity_pm.c
 create mode 100644 drivers/misc/trinity/trinity_resv_mem.c
 create mode 100644 drivers/misc/trinity/trinity_resv_mem.h
 create mode 100644 drivers/misc/trinity/trinity_stat.c
 create mode 100644 drivers/misc/trinity/trinity_stat.h
 create mode 100644 drivers/misc/trinity/trinity_sysfs.c
 create mode 100644 drivers/misc/trinity/trinity_trace.c
 create mode 100644 drivers/misc/trinity/trinity_trace.h
 create mode 100644 drivers/misc/trinity/trinity_vision2_drv.c
 create mode 100644 drivers/misc/trinity/trinity_vision2_profile.h
 create mode 100644 drivers/misc/trinity/trinity_vision2_regs.h
 create mode 100644 include/uapi/misc/trinity.h

-- 
2.25.1


       reply	other threads:[~2022-07-25  6:53 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CGME20220725065308epcas1p2f6de3d74792854bb312cca4b310badac@epcas1p2.samsung.com>
2022-07-25  6:52 ` Jiho Chu [this message]
     [not found]   ` <CGME20220725065309epcas1p4565e7bb0fea1aaf3e5e300de00774c2a@epcas1p4.samsung.com>
2022-07-25  6:53     ` [PATCH 1/9] trinity: Add base driver Jiho Chu
2022-07-27 11:54       ` Krzysztof Kozlowski
2022-07-27 13:22       ` Greg KH
2022-09-01 19:04         ` Dafna Hirschfeld
2022-09-02  5:39           ` Greg KH
     [not found]       ` <CGME20220725065309epcas1p4565e7bb0fea1aaf3e5e300de00774c2a@epcms1p2>
2022-07-28  2:05         ` MyungJoo Ham
2022-09-01 18:36       ` Mark Brown
2022-09-02  8:23         ` Jiho Chu
     [not found]   ` <CGME20220725065309epcas1p3c691bbc54c84775524b97c4b717c7ce7@epcas1p3.samsung.com>
2022-07-25  6:53     ` [PATCH 2/9] tirnity: Add dma memory module Jiho Chu
2022-07-27 13:15       ` Greg KH
     [not found]   ` <CGME20220725065309epcas1p20c847655e7332c818fc0fd2c50fb0e27@epcas1p2.samsung.com>
2022-07-25  6:53     ` [PATCH 3/9] trinity: Add load/unload IDU files Jiho Chu
2022-07-27 13:14       ` Greg KH
2022-09-17  7:39         ` Jiho Chu
     [not found]   ` <CGME20220725065309epcas1p42ba84c5241d69192ea73904ed6af17d7@epcas1p4.samsung.com>
2022-07-25  6:53     ` [PATCH 4/9] trinity: Add schduler module Jiho Chu
2022-07-27 13:09       ` Greg KH
     [not found]   ` <CGME20220725065309epcas1p413498a418cbf58570f8009ae7fd91015@epcas1p4.samsung.com>
2022-07-25  6:53     ` [PATCH 5/9] trinity: Add sysfs debugfs module Jiho Chu
2022-07-27 13:25       ` Greg KH
     [not found]   ` <CGME20220725065310epcas1p3688f336bfd5c732145575524f3365a0f@epcas1p3.samsung.com>
2022-07-25  6:53     ` [PATCH 6/9] trinity: Add pm and ioctl feature Jiho Chu
     [not found]   ` <CGME20220725065310epcas1p2ae58294d9cf44e622ed8cc7e5a8b988c@epcas1p2.samsung.com>
2022-07-25  6:53     ` [PATCH 7/9] trinity: Add profile module Jiho Chu
     [not found]   ` <CGME20220725065310epcas1p1841fde6ae768a98543418c81790c0832@epcas1p1.samsung.com>
2022-07-25  6:53     ` [PATCH 8/9] trinity: Add trace module Jiho Chu
     [not found]   ` <CGME20220725065310epcas1p2735e463512b0db489f2af532f15dae6e@epcas1p2.samsung.com>
2022-07-25  6:53     ` [PATCH 9/9] MAINTAINERS: add TRINITY driver Jiho Chu
2022-07-25  9:02   ` [PATCH 0/9] Samsung Trinity NPU device driver Greg KH
2022-07-25  9:10     ` Oded Gabbay
     [not found]     ` <CGME20220725065308epcas1p2f6de3d74792854bb312cca4b310badac@epcms1p5>
2022-07-26  2:09       ` MyungJoo Ham
2022-07-26  6:59         ` Krzysztof Kozlowski
2022-07-26  7:51           ` Arnd Bergmann
2022-07-26 11:24             ` Oded Gabbay
2022-07-29 17:50             ` Pavel Machek
     [not found]       ` <20220726050305epcms1p5ef19a54322263c768ea71d59da7e2616@epcms1p5>
2022-07-26 14:35         ` 추지호/Robot Intelligence팀(SR)/Staff Engineer/삼성전자
2022-07-26  6:57   ` Krzysztof Kozlowski
2022-07-27 11:51     ` Jiho Chu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220725065308.2457024-1-jiho.chu@samsung.com \
    --to=jiho.chu@samsung.com \
    --cc=arnd@arndb.de \
    --cc=gregkh@linuxfoundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=myungjoo.ham@samsung.com \
    --cc=yelini.jeong@samsung.com \
    --subject='Re: [PATCH 0/9] Samsung Trinity NPU device driver' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).