Report from 2013 ARM kernel summit

Report from 2013 ARM kernel summit

[Grant Likely]

Here is the plain text version. Reply and comment to your heart's content.

Once again, thanks to Olof Johannson, Kevin Hilman, Mark Brown and
Paul Walmsley for the help in preparing this report.
________

Over the last 3 years it has become a tradition to co-locate a meetup of
the ARM kernel maintainers and core developers with kernel summit. This
year 47 of us descended upon Edinburgh for two days of meetings before
ksummit. Here is a summary of the conversations over the two days, but
the raw notes are also available[1] and are worth a read for a great
deal more detail.

Day 1
-----
ARM Maintainership Trees

Wednesday morning kicked off with a largely uncontroversial review of
the ARM maintainership trees. Olof Johansson provided a status report of
the ARM-SoC tree. Generally he's pretty happy with how things have been
working, but he had some recommendations on what he would like to see in
pull requests. For one, it makes his life easier if pull requests are
based on the earliest -rc possible simply so that it doesn't pull in
mainline commits that he's not pulled into ARM-SoC yet. Russell King
agreed on this point saying it was easier when it didn't look like pull
requests contained loads of other stuff.

Olof is a little concerned about the flood of patches and pull requests
showing up at around -rc6 time. Typically the only time that changes are
not getting pulled into the arch and arm-soc trees is during the merge
window, immediately before -rc1. Outside of that, pull requests should
be sent as early as possible instead of "right before the next merge
window". If you miss the merge window, don't let that be a reason to
delay a pull request by another 10 weeks. Also, once you do sent a pull
request, don't change the tag. Russell said he will often pull a branch,
test it, but then not actually merge it until a few days later to see if
it has changed.

The conversation digressed slightly here to talk about testing. There
are several automated build systems testing the ARM tree at the moment,
including Fengguang Wu's 0-day build test and test runs maintained by
Olof, Kevin Hilman and Paul Walmsley. Pretty much everyone agreed about
the value of the tools, but some questions were asked about how the test
reports can be made more useful and what can be done to increase
coverage. Test reports from Olof's, Kevin's and Paul's systems are going
to kernel-build-reports at lists.linaro.org for those who are interested,
but there is not yet any automatic notification to maintainers when a
tree breaks. Kevin and Paul also boot test the kernel on the limited
selection of ARM boards they have access to. A few requests were voiced
include more boot tests on non-ARM boards, a web status dashboard front
end, and testing for power management regressions. However, as valuable
as any or all of these features would be, Olof, Kevin and Paul's time is
limited. Anyone volunteering to implement said features would be most
welcome.

Russell King and Catalin Marinas also got a turn to talk about how
things were working for arm and arm64 maintenance respectively. Again
there wasn't much to say. The current process is working well for
Russell, but he took the opportunity request folks to send him a
reminder about a week after sending him a pull request. With 8k-9k of
traffic on the list per month it is easy for things to get lost.
Similarly, the current process is working well for Catalin although he
doesn't yet have to deal with a multitude of individual SoC ports. Most
of it is core changes to fill out important features like ftrace and
kprobes.

Will Deacon (ARM64 co-maintainer) is currently concerned that people are
writing code, but not spending very much time reviewing. The sentiment
was repeated by others in the room, but without any concrete solutions
aside from continuing to encourage developers to review other patches.

Kernel Consolidation

The next topic was a review of the consolidation work on ARM
sub-architectures. Linus Walleij showed a spreadsheet[2] listing all of
the sub-architectures, and progress towards single zImage, device tree,
and other common infrastructure pieces. Of the 62 sub-architectures, 26
are mostly complete, 14 are in various states of progress, and 8 haven't
been started yet. The remaining 14 aren't expected to ever be converted
either because it is expected to be removed or because there is a
technical reason it will not be converted. When it was asked if the
unconverted platforms could be removed, Arnd Bergmann pointed out that a
lot of them are strongarm/xscale based which has an active hobbyist
community. It would be unfriendly to pull that out from under them.

Stephen Warren asked about defconfig policy. There are still a lot of
defconfig files and he was wondering if some of them need to be removed.
multi_v7_defconfig could be used instead. However, supporting a large
number of devices with one defconfig currently means a lot of drivers
must be built directly into the kernel. Tony Lindgren asked if most of
multi_v7_defconfig could be switched to using modules, but doing so is
inconvenient for a lot of kernel developers because it requires an
initrd. There was a brief discussion about putting a simple tool into
the kernel build scripts to generate a minimal initrd with just enough
to meet the needs of kernel developers. Tony and Kevin volunteered to
investigate further and Mike Turquette suggested looking at Dracut.[3]
For everything else, one defconfig per subarchitecture sounded
reasonable to the architecture maintainers.

It is also a concern that very few people are enabling the drivers as
modules and there are already known init order problems for some drivers
as modules (like interrupt and GPIO controllers). Grant and Tony both
expressed that init order problem is important to solve because it
forces better organization and separation between drivers. Kevin closed
the discussion by proposing that making multi_v7_defconfig use modules
be a long term goal after the above concerns are addressed.

Early Init, Deferred Probe and Init Order

Finally before breaking for lunch some time was spent discussing what to
do with devices that are being set up before initcall time and therefore
before the driver model is available. The early_platform infrastructure
is intended to support these devices by allowing a platform_device
structure to be created early and then registered into the driver model
at initcall time. Magnus Damm has said that early_platform has worked
well for SHMobile, but Russell is concerned that early_platform seems
like a hack, a concern seconded by Arnd and Grant. The problem is that
early_platform adds non-trivial complexity to the driver model for a
very small number of devices.

As an alternative, it was suggested that early running code shouldn't
try to use the driver model at all, but rather limit itself to only
essentials required to get to initcalls. For instance, would it be
possible to have SoC early setup code that enables clocks and
regulators, but doesn't attach a full driver until initcall time.
Getting to initcalls requires very little hardware and anything
unnecessary during early init really should be pushed out to that time.
Even initialization of secondary interrupt controllers can be delayed if
not required by the system timer. The examples of of_clk_init() and
of_irq_init() were mentioned as a way to do early initialization, but
only when deferring to initcalls is not possible. Kumar Gala made the
point that adding more of_<blah>_init() functions is not going to
scale.

The big problem with deferring to initcalls is that the kernel doesn?t
have any information about dependencies between devices and so doesn?t
know what order to call modules in. Right now driver probe order is
roughly determined by kernel link order first, and registration order
second for devices populated during or after initcalls. Olof commented
that the current driver model is based on the assumption that devices
live in a single hierarchy. There is no easy way to add in dependency
data with the current structure, and in most cases dependencies aren?t
specifically on other devices, but rather on the service provided by
another device (ie, a GPIO line).

Kumar is concerned that deferred probe won?t work in all situations. For
instance, what about a device that has an optional dependency? Does it
fail to probe in the hope that the dependency will show up later? It was
suggested by Mark Brown that if the resource is described in the device
tree then it is indeed the correct behaviour to defer until it arrives.
Or, if the driver really is able to proceed, then the driver should
assume responsibility for obtaining the resource at a later time.

The suggestion was made that it would be a really nice to have the core
kernel sort out probe order rather than using deferred probe. Grant
replied that deferred probe was designed as the simplest possible
solution to the problem, and by no means is he attached to it if someone
can come up with a better approach. He did consider other options that
put dependency resolution into the core, but it ended up pulling all
kinds of GPIO, IRQ, Clock and DMA details into the core which made it
quite complex.

Non-probeable System Architecture

Will Deacon raised the topic of how to deal with some of the more
complex details of system architecture. The architecture details that
are going to increase complexity include cache coherency domains (CCI),
endpoint-to-endpoint DMA, GIC MSI mapping and interconnects which lose
meta-data when translating. For example, he presented the problem of a
PCIe device attached through a PCIe bridge performing DMA access through
an SMMU. A requester ID (RID) is mapped onto a Stream ID (SID) which it
turn maps onto a Device ID (DID), but RIDs don?t necessarily map
directly onto an SID and the kernel needs to handle translating between
the ids in order to program the SMMU correctly. This is important to get
right, otherwise the SMMU will either block legitimate access or allow
devices to make invalid requests.

Greg Kroah-Hartman made the point that PCIe is supposed to be completely
probable and that on x86 firmware takes care of any mapping required by
the hardware. Having to do this in the kernel is just crazy. Will
responded that this isn?t actually a PCIe problem, but rather how the
PCIe bridge is attached to the SMMU. Also, unlike on x86, most ARM
platform builders don?t expect firmware to abstract away the hardware
mapping so the kernel needs to know the mechanism for translating IDs.
For v8, ARM is pushing for a common firmware interface in the form of
trusted firmware[4] running in the secure world (EL3) and PSCI[5] for
the interface. However, the mapping problems described by Will fall
outside of the current scope for trusted firmware.

No firm conclusions were reached in this session other than to continue
on the approach of encoding the data into the device tree.

ARMv8 Secure Firmware requirements

ARM has introduced the Power State Coordination Interface (PSCI)[6] in
an attempt to standardize the interaction between Linux (or any OS) and
supervisory software running at the different privilege levels in order
to manage power state transitions. Before ARMv8, there was no standard
for managing SoC power states. Since the use of privilege levels was not
standard across SoCs or not used at all, every SoC had their own way of
managing power state transitions.

With the introduction of ARMv8, there is now an attempt to standardize
using PSCI, which provides a standard for arm32 and arm64.  However,
while ARM encourages use of the new standards, the ARM ecosystem does
not have a reputation of following standards.  Also, since ARMv8
hardware is not widely available, it?s not clear how broadly PSCI will
be used.

In theory, PSCI would simplify the core kernel code by hiding much of
the complexity in firmware. This is the ideal world, and the one ARM is
encouraging. In the real world however, there are several complexities
that were raised: non-PSCI based systems, firmware vendors wanting to
have minimal firmware for audit reasons, buggy/broken firmware, errata
workarounds requiring secure privileges, etc.

Unfortunately, there is not yet a one-size-fits-all solution to this, so
it?s likely the kernel will have to handle both PSCI and some SoC
specific stuff. The goal of the arm64 maintainers though is to keep SoC
specifics to an absolute minimum.

ARMv8 Board Support

Catalin Marinas led the next session on the status of ARMv8 board
support. The executive summary is: there will be no board support in
arch/arm64. Everything should live under the appropriate
subsystem/driver dir now. The reality is that this isn?t quite ready
yet, but that is the direction we?re headed.

Exploit Mitigation

Kees Cook lead an afternoon session giving an overview of the exploit
mitigation work he is doing. His slides are available at
http://outflux.net/slides/2013/arm/mitigation.pdf. Kees first used the
time to review how kernel bugs can get exploited as a security
vulnerability. There are no shortage of kernel bugs to be found, so
Kees? focus is on how to makes those bugs harder to translate into
exploits.

Kees first covered the tools available for fuzzing and static analysis.
Fuzzing, or subjecting the software to random or otherwise invalid
input, is a great way to look for unexpected cases in the kernel. Kees
has high praise for the Trinity project[7] and he recommends running
Trinity for 10 minutes as part of automated tests. Fuzz testing can also
be driven by external hardware if available. For example, Facedancer[8]
is a hardware USB device which can generate arbitrary USB traffic and is
driven by Python scripts. Kees was able to find 12 bugs in the USB HID
using Facedancer.

Kees also recommends using static analysis tools such as smatch and
coccinelle to look for common patterns of bugs before they can be turned
into exploits. However, while both static analysis and fuzzing are great
for finding bugs, they don?t make it any harder to exploit as-yet
undiscovered bugs. For that, hardening the kernel is needed.

The low hanging fruit for hardening is using page permissions to protect
against certain classes of attacks. For instance, enforcing RO data to
be in read-only pages, disabling execution from data pages, and prevent
userspace execution and read/write from the kernel. On ARM, most of
these protections are implemented, but protecting against kernel
read/write of userspace pages is proving difficult. Kees would
appreciate help getting it to work. He also would like to identify
write-seldom data and explicitly mark those pages as read-only, but that
approach is largely unexplored in the upstream kernel.

Moving toward the more experimental efforts, there is work right now in
bringing Address Space Randomization (ASR) to kernel space. The idea of
ASR is to make it difficult to predict where a function actually exists
in memory on a running system and therefore difficult to craft a stack
smash attack that exploits it. ASR is well established in userspace, but
it is difficult to implement for the kernel. There is support available
now on x86 for the text sections and patches are coming soon to add
randomization for modules, kmalloc and vmalloc. ARM however has some
extra problems related to memory not always being based at address 0
which makes it tricky to work out physical to virtual translations.

Arnd also raised the concern that the number of locations to which the
kernel can be relocated is constrained to a small number of
possibilities. Fewer than 1024 in a typical 32-bit system. Even if an
attacker can only succeed one out of 1024 attempts, that is still a huge
number of systems if the pool of systems is in the millions.

Finally, Kees covered some core Linux features useful for Linux that
aren?t fully implemented on ARM. x86 and s390 have CONFIG_DEBUG_RODATA
and CONFIG_DEBUG_SET_MODULE_RONX, both of which are used to mark
read-only or non-executable memory as write-protected and optionally
no-execute in the page tables. CONFIG_X86_PTDUMP provides details of the
page tables in debugfs which is useful for debugging page table
permission changes. Kees would like to see all of the above implemented
on ARM. Russell took interest in the second option and had a draft patch
posted before the end of the week.[9] There is some existing work
ongoing for ARM security providing CONFIG_STRICT_MEMORY_RWX[10] and
grsecurity/PaX[11] which Kees highlighted on his final slide.

Memory features and Config data in Device Tree

The final sessions of the day touched on a couple of device tree related
issues. The first was how to describe memory features in the device tree
and the second was what to do with platform configuration data. Laura
Abbott brought up the memory features topic to discuss the handling of
memory that should not or must not get mapped as normal memory. Some
examples of why a memory region would be set aside are, to be used by
the contiguous memory allocator (CMA), to describe
hot-pluggable/power-managed memory, or to describe NUMA behaviour. The
topic was discussed briefly and then deferred to the second day because
it overlapped with the reserved memory binding discussion scheduled for
the 2nd day, with a recommendation to consult with Benjamin
Herrenschmidt and Steve Capper about how to model NUMA platforms.

The discussion did lead into a question about what to do with purely
software configuration data that is needed at boot time. For example,
Laura used the example of the ION memory manager which is entirely a
software construct. The kernel needs to set aside memory to be used by
ION at boot time, but the amount of memory isn?t a property of the
physical hardware, nor can the kernel determine the correct size
automatically and nor can userspace provide the right size. The natural
thing to do is put the data into the device tree, but device tree policy
is to avoid putting Linux-specific details into the tree since it is
supposed to describe hardware independent of the operating system. This
discussion was also tabled for the next day, but not before Grant made
the statement that a lot of configuration data is really about intended
operational configuration, even if it isn?t strictly a hardware
description. It is perfectly reasonable to put that information into the
device tree.

Day 2
-----

Device Tree Process is Broken

Day 2 turned into the device tree day starting off with the big problem
that the process for creating new bindings is completely broken. The
original plan was to split into two rooms right from the beginning of
the second day, but when it was realized that the DT process problem
affects everyone[12], the decision was made to keep the entire group
together to discuss the issue.

Several major problems have been identified with the device tree process,
1. There is disagreement on whether or not the DT should be treated as
a stable ABI,
2. subsystem maintainers don?t know what to do with device tree changes,
3. developers feel they need to lock down the perfect binding before
DT bindings will get merged, and
4. traffic on the devicetree mailing list is overwhelming. Reviewers
are burning out.

In addition, the mailing list thread raised the meta question, ?Is DT on
ARM the solution, or is there something better??[13] Grant Likely chimed
in on this last question with the thought that right now we?ve basically
got four options; go back to board files, stick with the DT approach,
switch to ACPI, or invent something entirely new. There is no problem
with investigating alternatives, but development cannot stop in the mean
time. The immediate DT problems still need to be solved. Kumar Gala also
pointed out that the problems aren?t DT-specific. Exactly the same set
of problems would need to be solved if we were using something else,
like ACPI.

Hardware Configuration Data

The configuration data question was sorted out quickly after a brief
discussion. General agreement was that configuration data relating to
how the hardware should be used is okay. Linux specific data should
still be avoided. For example, how much contiguous buffer memory to set
aside, and preferred clock rates are reasonable, but names of Linux
modules to load are not. However, this can only be a guideline and
sometimes Linux specific details need to be there. When that happens,
the Linux-specific properties should be prefixed with ?linux,?.
Regardless, the DT should always provide enough information for the
system to boot into an operational state without resorting to extra
parameters on the kernel command line.

Where that data should be put was also discussed. David Brown suggested
that perhaps the DT should split into separate trees, one for hardware
and one for configuration, but Kumar pointed out that doing so would
replicate the SoC hierarchy under a different node which increases the
complexity. Putting all configuration data under the /chosen node was
one suggestion. Historically the /chosen node was created to pass data
from OpenFirmware to the kernel, so it is a reasonable choice. However,
if the configuration is related to a specific device and isn?t likely to
be changed by the user then it is better to keep the data in a device
node.

Device Tree as Stable ABI

The big question of the day was whether or not device tree should be
considered a stable ABI. One side of the debate suggests that once a
binding is set, it should never change because doing so would break
booting on boards using the older device tree. The other side asserts
that enabling new features and new hardware requires binding changes, so
insisting on a stable ABI forces binding authors to write a ?perfect?
binding on the first try before they understand the hardware.

Setting the context for the problem, Thomas Petazzoni compared the
device tree to the kernel?s other stable API, syscalls. Compared to what
is being done with device tree, the syscall ABI evolves very slowly
because it has to be stable, but it is also much more limited in scope.
Additionally, syscalls to a large extent attempt to abstract the
hardware, whereas device tree attempts to describe it in detail.

The first issue to deal with was what does it actually mean to have a
stable ABI? From the user perspective, it means that upgrading the
kernel should not cause breakage. How that works out in real life
depends on how the kernel is upgraded. Laurent Pinchard pointed out that
if DT is in mainline and always upgraded with the kernel then there will
never be any breakage. However, shipping the DT with the kernel
increases the burden on the distribution vendor to provide DTs for all
known hardware. It is preferable in that case for firmware to provide a
DT. David Brown suggested the solution may be to stabilize the ABI for
general purpose server/client systems, but don?t burden embedded
developers with the same requirement.

>From Russell King?s perspective, if a binding has been released in a
mainline kernel, then by default it needs to be considered a stable ABI.
If there are unstable bindings, then there must be a way to mark them as
unstable. David Woodhouse agreed with the addition that it needs to be
enforced by the DT tools at build time. A staging tree was proposed for
bindings with some rules about their lifecycle. If a staging binding is
accepted, then it needs to either become stable within a fixed period of
time (perhaps 6 months), show forward progress, or be removed from the
tree for inactivity. Again the tooling issue was raised and the goal is
to make the DT tools perform binding validation.

That still leaves the question of, what does a stable binding look like?
Certainly a stable binding means that a newer kernel will not break on
an older device tree, but that doesn?t mean the binding is frozen for
all time. Grant said there are ways to change bindings that don?t result
in breakage. For instance, if a new property is added, then default to
the previous behaviour if it is missing. It is perfectly acceptable for
a binding author to start simple and extend the binding as needed. If a
binding truly needs an incompatible change, then change the compatible
string at the same time. The driver can bind against both the old and
the new. These guidelines aren?t new, but they desperately need to be
documented.

Also, if a binding changes, and nobody notices, has anything been
broken? There was a fair bit of discussion about how bindings develop
organically and, especially for new hardware, require some time to sort
out. Allowing some grace for new hardware support is appropriate.

It is clear from the discussion that a lot of details and expectations
need to be documented. However, as a starting point the following broad
statements were proposed to the general agreement of the room:
1. By default, bindings released with the kernel shall be considered stable
   1. Breaking end users should be avoided.
   2. Unstable bindings will be marked as unstable with a timeline for
      stabilization
   3. However, if nobody complains, has anything been broken?
   4. Bindings can be changed in incompatible ways if it can be
      reasonably argued that nobody will be affected by the change
1. Policy documentation is desperately need. Action on Grant to organize
   doc writing.
2. Statement to be written on what is stable ABI and what is not.

Some of the above work has been started. A smaller group split off to
hammer out a statement on DT binding policy which is available in the
form of presentation slides[14], and will be written up as a document in
the coming weeks. Also, Stephen Warren had already written a good
guidance document for crafting device tree bindings.[15]

List overload

The other major problem faced by device tree users is that patches are
getting stalled in the review process. The traffic on the device tree
mailing list is so high that it overwhelms the device tree binding
reviewers. A suggestion was made that perhaps only the binding changes
should be posted to the mailing list, excluding the code change, but it
was pointed out that sometimes the code provides useful context about
why the binding is designed the way it was, so it is helpful to have it
in the same series. Regardless, from this point forward, the DT
reviewers are going to focus primarily on the binding documentation. Any
review of implementation will be somewhat incidental.

In addition, many subsystem maintainers have little knowledge about what
device tree bindings are supposed to look like, and so some avoid
merging patches which haven?t been acked by a DT reviewer. Grant
suggested that rules should be provided for what to do with unreviewed
DT bindings, and provide the guidance that if a binding hasn?t been
reviewed in a reasonable period of time (two weeks was suggested) then
the subsystem maintainer can use their own discretion. Will Deacon and
Olof strongly objected with the concern that it will be a free pass to
get anything into the kernel. After further discussion the tentative
agreement in the case of sleepy DT reviewers was to leave the decision
to the subsystem maintainer, provided that it is contained to within a
single device driver and the maintainer is satisfied that the patch
looks reasonable. Common subsystem bindings that affect multiple drivers
still need DT review since the impact is far greater.

The output of this discussion can also be found in the DT binding policy
slide deck.[16]

Device Tree Schema and Tooling

Several of the issues regarding device tree that have been causing
problems have been attributed to limitations in the tooling, especially
its ability to automatically identify errors. Proposals for defining
schemas to allow validation of device tree bindings from Tomasz Figa and
Benoit Cousson were discussed but no firm conclusions were drawn since
it is not yet clear exactly what needs to be validated.

The next step to progress this will be to do a review of the existing
bindings in order to scope out the features that will need to be checked
so that a format for defining schema can be developed.

Device Tree Reserved Memory

There was a small group discussion to hammer out a binding for reserved
memory regions. Using the binding proposed by Marek Szyprowski[17] it
was reworked to properly support platforms with multiple memory nodes
and generalize the usage model. The results of the discussion with a new
proposal have already been posted to the mailing list[18].

Power management

The power management breakout started with Kevin Hilman highlighting
that the power management core maintainers have recently moved towards
favouring a runtime PM centric view of the world. This is good news as
this has been popular with much of the embedded world. There was some
discussion of the limited current adoption of runtime PM, laziness was
blamed for much of this - providing standard runtime PM integration in
subsystems has been one way of making this easier.

There followed a discussion of standardising the interfaces for
interacting with controllers with PM functionality, such as the M3 found
in the TI processor in the BeagleBone Black. Concerns were raised that
too much functionality is being moved into these processors, often
limited to specific use cases like system suspend which caused problems
when implementing runtime PM, and that often it is hard for upstream to
influence.

PSCI was covered but since it focuses on CPU complexes it isn?t
sufficient for many of the use cases where the entire SoC is involved.

remoteproc and rpmsg were discussed, remoteproc did not seem problematic
but there were concerns that rpmsg may be too heavyweight for the
microcontrollers used for power management. Linus Walleij observed that
these had been a good inspiration for interfaces to modems at ST/E.
There was agreement that providing strong examples of good practice and
reusable code is one of the best ways for upstream to influence firmware
design.

The immediate conclusion was that we should place what?s there at the
minute under a single directory in drivers and then work on
standardising the interfaces to the rest of the system and providing
high quality references.

UEFI secure boot on ARM
In the last slot of the day, Matthew Garrett and James Bottomley joined
the group to spend some time talking about the lessons learned working
on UEFI secure boot. Some ARM vendors intend to use Secure Boot on ARM
devices and it would be good to have some influence on implementation so
that it is in a form that works well with Linux.

In James? ideal world, ARM device would ship in setup mode so the users
are always able to install their own keys, but failing that it would be
best to have a standard mechanism to unlock the bootloader and re-enter
setup mode. On devices intended to run Windows, Microsoft is the signing
authority.

The Shim[19] bootloader and the Linux Foundation secure boot system[20]
are both designed to be signed by a third party (Microsoft) and then
allow Linux to be booted without a signature from said third party. Shim
also provides a consistent method for managing keys which wouldn?t be
necessary if vendors can be convinced to implement a consistent method
in UEFI.

In the ARM world, it is highly likely that systems will be designed only
to run Linux and probably won?t have any need to use Microsoft as a
signing authority. Without a central authority (and setting up an
authority is expensive) it is most likely that vendors will need to be
their own root of trust. However, if vendors can be convinced that
allowing the firmware to be unlocked and put into setup mode, then there
should be no need to use Shim.

As for implementation, as far as James and Matthew know, the upstream
Tianocore project has everything needed to work with secure boot. There
are details of course about platform secure variable storage, but that
should not prevent getting the tools and signing working.

Finally, the tools for signing ARM PECOFF binaries need work. Jeremy
Kerr?s sbsign and Peter Jones? pesign tools both should be investigated.
It is believed that they don?t currently work on 32-bit.

Next Steps and Conclusion

Wrapping up the event, Olof asked the question about how this event
worked, what can be done differently, and should another gathering be
organized. This year the summit was split into two rooms on the second
day which was good for deep dive discussions. Time was split about 60%
discussions with the whole group and 40% smaller groups and hacking
time. Everyone seemed happy with the level of activity and would do the
same thing.

Scheduling this year was coordinated on a shared Google calendar, and
could best be described as fuzzy. Topics were moved around or
added/removed as needed. Some expressed that it was difficult to know
where to be at times, partially because the rooms were quite far apart,
but in general the format worked. Suggestions were made for using strict
time-slots or an ?unconference? system next time.

For doing it again, the consensus was a resounding yes, hopefully more
frequently than once a year. Possible venues were discussed including
co-location with Linaro Connect or ELC. Co-locating with Linaro is a
concern for some since Linaro is a membership organization and not all
ARM maintainers work for member companies. Linux Foundation events like
ELC is a good fit since the LF team has been very supportive in hosting
Linux development summits like this one. Olof and Kevin will investigate
organizing a new summit.
________________
[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-November/209580.html
[2] http://goo.gl/GD9MAO
[3] https://dracut.wiki.kernel.org/index.php/Main_Page
[4] http://goo.gl/FqQ4YI
[5] http://goo.gl/zrXE39 (behind registration wall)
[6] Ibid.
[7] http://codemonkey.org.uk/projects/trinity
[8] http://goodfet.sourceforge.net/hardware/facedancer11
[9] http://comments.gmane.org/gmane.linux.kernel.cross-arch/19992
[10] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-October/203261.html
[11] http://forums.grsecurity.net/viewtopic.php?f=7&t=3292
[12] Quite possibly influenced by the long running thread on the
ksummit-2013-discuss mailing list.
http://thread.gmane.org/gmane.linux.drivers.devicetree/48898
[13] Ibid
[14] http://goo.gl/pXtY2b
[15] http://www.spinics.net/lists/devicetree/msg03897.html
[16] Ibid
[17] http://permalink.gmane.org/gmane.linux.ports.arm.kernel/263219
[18] http://permalink.gmane.org/gmane.linux.drivers.devicetree/50160
[19] https://github.com/mjg59/shim
[20] http://blog.hansenpartnership.com/linux-foundation-secure-boot-system-released

Re: Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Missed some MLs. Resending....

Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:

> Early Init, Deferred Probe and Init Order
...
> The big problem with deferring to initcalls is that the kernel doesn?t
> have any information about dependencies between devices and so doesn?t
> know what order to call modules in. Right now driver probe order is
> roughly determined by kernel link order first, and registration order
> second for devices populated during or after initcalls. Olof commented
> that the current driver model is based on the assumption that devices
> live in a single hierarchy. There is no easy way to add in dependency
> data with the current structure, and in most cases dependencies aren?t
> specifically on other devices, but rather on the service provided by
> another device (ie, a GPIO line).

I think that an IOMMU device is also the service provider for other
devices, where an single IOMMU device resides in the bus to provide
the address translation to the bus children.

> Kumar is concerned that deferred probe won?t work in all situations. For
> instance, what about a device that has an optional dependency? Does it
> fail to probe in the hope that the dependency will show up later? It was

Other devices don't know if the system has IOMMU or not. IOMMU'able
devices' population can be deferred till IOMMU device is populated,
but they don't know if there's really IOMMU device exists or not at
their population time.

....
> The suggestion was made that it would be a really nice to have the core
> kernel sort out probe order rather than using deferred probe. Grant
> replied that deferred probe was designed as the simplest possible
> solution to the problem, and by no means is he attached to it if someone
> can come up with a better approach. He did consider other options that
> put dependency resolution into the core, but it ended up pulling all
> kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> quite complex.

I have a problem to control the poplution order for IOMMU in Tegra
SoC, and currently I haven't found any framework so that I inserted an
IOMMU hook in the driver core[1] since I discussed with Stephen Warren
and we thought that the number of those subsystem which requires this
kind of control isn't so many. This patch depends on a certain DT
bindings("#stream-id-cells") if a device has an dependency on IOMMU or
not. But at least this doesn't require anything for a client driver
side at all. The exsiting driver can be used as they are. The current
design is:

1, When a device is populated, it checks if that device is IOMMU'able
   or not. This is identified by "#stream-id-cells" in DT. If
   a device is normal(non IOMMU), a device is populated. If a device
   is IOMMU'able, it continues to be checked.

2, When an IOMMU device has been already populated,
   "dev->bus->iommu_ops" is set. If so a device(client) is registered
   as IOMMU'able, and the rest of population process continues. If not
   set yet, a device population would be deferred til
   "dev->bus->iommu_ops" is set.

3, After an IOMMU device has been already populated, deferred
   population will be continued.

I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
or not, but I haven't any better idea right now.

Any suggestion/hint would be really appreciated.

[1]
http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006933.html

Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Missed some MLs. Resending....

Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:

> Early Init, Deferred Probe and Init Order
...
> The big problem with deferring to initcalls is that the kernel doesn?t
> have any information about dependencies between devices and so doesn?t
> know what order to call modules in. Right now driver probe order is
> roughly determined by kernel link order first, and registration order
> second for devices populated during or after initcalls. Olof commented
> that the current driver model is based on the assumption that devices
> live in a single hierarchy. There is no easy way to add in dependency
> data with the current structure, and in most cases dependencies aren?t
> specifically on other devices, but rather on the service provided by
> another device (ie, a GPIO line).

I think that an IOMMU device is also the service provider for other
devices, where an single IOMMU device resides in the bus to provide
the address translation to the bus children.

> Kumar is concerned that deferred probe won?t work in all situations. For
> instance, what about a device that has an optional dependency? Does it
> fail to probe in the hope that the dependency will show up later? It was

Other devices don't know if the system has IOMMU or not. IOMMU'able
devices' population can be deferred till IOMMU device is populated,
but they don't know if there's really IOMMU device exists or not at
their population time.

....
> The suggestion was made that it would be a really nice to have the core
> kernel sort out probe order rather than using deferred probe. Grant
> replied that deferred probe was designed as the simplest possible
> solution to the problem, and by no means is he attached to it if someone
> can come up with a better approach. He did consider other options that
> put dependency resolution into the core, but it ended up pulling all
> kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> quite complex.

I have a problem to control the poplution order for IOMMU in Tegra
SoC, and currently I haven't found any framework so that I inserted an
IOMMU hook in the driver core[1] since I discussed with Stephen Warren
and we thought that the number of those subsystem which requires this
kind of control isn't so many. This patch depends on a certain DT
bindings("#stream-id-cells") if a device has an dependency on IOMMU or
not. But at least this doesn't require anything for a client driver
side at all. The exsiting driver can be used as they are. The current
design is:

1, When a device is populated, it checks if that device is IOMMU'able
   or not. This is identified by "#stream-id-cells" in DT. If
   a device is normal(non IOMMU), a device is populated. If a device
   is IOMMU'able, it continues to be checked.

2, When an IOMMU device has been already populated,
   "dev->bus->iommu_ops" is set. If so a device(client) is registered
   as IOMMU'able, and the rest of population process continues. If not
   set yet, a device population would be deferred til
   "dev->bus->iommu_ops" is set.

3, After an IOMMU device has been already populated, deferred
   population will be continued.

I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
or not, but I haven't any better idea right now.

Any suggestion/hint would be really appreciated.

[1]
http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006933.html

Re: Report from 2013 ARM kernel summit

[Grant Likely]

On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> Missed some MLs. Resending....
> 
> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:
> 
> > Early Init, Deferred Probe and Init Order
> ...
> > The big problem with deferring to initcalls is that the kernel doesn?t
> > have any information about dependencies between devices and so doesn?t
> > know what order to call modules in. Right now driver probe order is
> > roughly determined by kernel link order first, and registration order
> > second for devices populated during or after initcalls. Olof commented
> > that the current driver model is based on the assumption that devices
> > live in a single hierarchy. There is no easy way to add in dependency
> > data with the current structure, and in most cases dependencies aren?t
> > specifically on other devices, but rather on the service provided by
> > another device (ie, a GPIO line).
> 
> I think that an IOMMU device is also the service provider for other
> devices, where an single IOMMU device resides in the bus to provide
> the address translation to the bus children.
> 
> > Kumar is concerned that deferred probe won?t work in all situations. For
> > instance, what about a device that has an optional dependency? Does it
> > fail to probe in the hope that the dependency will show up later? It was
> 
> Other devices don't know if the system has IOMMU or not. IOMMU'able
> devices' population can be deferred till IOMMU device is populated,
> but they don't know if there's really IOMMU device exists or not at
> their population time.

Deferring the population might complicate the common code by changing
the time that devices get created. Right now the code is oriented around
doing it all at once.

However, it would be possible to cause a probe deferal on devices that
requires an IOMMU in a way that it would be triggered by the bus instead
of by the individual device drivers.

I don't really know the details well enough to comment intelliently.
Craft up the patches and see what it looks like.

> > The suggestion was made that it would be a really nice to have the core
> > kernel sort out probe order rather than using deferred probe. Grant
> > replied that deferred probe was designed as the simplest possible
> > solution to the problem, and by no means is he attached to it if someone
> > can come up with a better approach. He did consider other options that
> > put dependency resolution into the core, but it ended up pulling all
> > kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> > quite complex.
> 
> I have a problem to control the poplution order for IOMMU in Tegra
> SoC, and currently I haven't found any framework so that I inserted an
> IOMMU hook in the driver core[1] since I discussed with Stephen Warren
> and we thought that the number of those subsystem which requires this
> kind of control isn't so many. This patch depends on a certain DT
> bindings("#stream-id-cells") if a device has an dependency on IOMMU or
> not. But at least this doesn't require anything for a client driver
> side at all. The exsiting driver can be used as they are. The current
> design is:
> 
> 1, When a device is populated, it checks if that device is IOMMU'able
>    or not. This is identified by "#stream-id-cells" in DT. If
>    a device is normal(non IOMMU), a device is populated. If a device
>    is IOMMU'able, it continues to be checked.
> 
> 2, When an IOMMU device has been already populated,
>    "dev->bus->iommu_ops" is set. If so a device(client) is registered
>    as IOMMU'able, and the rest of population process continues. If not
>    set yet, a device population would be deferred til
>    "dev->bus->iommu_ops" is set.
> 
> 3, After an IOMMU device has been already populated, deferred
>    population will be continued.
> 
> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> or not, but I haven't any better idea right now.

It seems a little fragile to me too. I'd rather the IOMMU requirement be
described more explicitly.

g.

Report from 2013 ARM kernel summit

[Grant Likely]

On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> Missed some MLs. Resending....
> 
> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:
> 
> > Early Init, Deferred Probe and Init Order
> ...
> > The big problem with deferring to initcalls is that the kernel doesn?t
> > have any information about dependencies between devices and so doesn?t
> > know what order to call modules in. Right now driver probe order is
> > roughly determined by kernel link order first, and registration order
> > second for devices populated during or after initcalls. Olof commented
> > that the current driver model is based on the assumption that devices
> > live in a single hierarchy. There is no easy way to add in dependency
> > data with the current structure, and in most cases dependencies aren?t
> > specifically on other devices, but rather on the service provided by
> > another device (ie, a GPIO line).
> 
> I think that an IOMMU device is also the service provider for other
> devices, where an single IOMMU device resides in the bus to provide
> the address translation to the bus children.
> 
> > Kumar is concerned that deferred probe won?t work in all situations. For
> > instance, what about a device that has an optional dependency? Does it
> > fail to probe in the hope that the dependency will show up later? It was
> 
> Other devices don't know if the system has IOMMU or not. IOMMU'able
> devices' population can be deferred till IOMMU device is populated,
> but they don't know if there's really IOMMU device exists or not at
> their population time.

Deferring the population might complicate the common code by changing
the time that devices get created. Right now the code is oriented around
doing it all at once.

However, it would be possible to cause a probe deferal on devices that
requires an IOMMU in a way that it would be triggered by the bus instead
of by the individual device drivers.

I don't really know the details well enough to comment intelliently.
Craft up the patches and see what it looks like.

> > The suggestion was made that it would be a really nice to have the core
> > kernel sort out probe order rather than using deferred probe. Grant
> > replied that deferred probe was designed as the simplest possible
> > solution to the problem, and by no means is he attached to it if someone
> > can come up with a better approach. He did consider other options that
> > put dependency resolution into the core, but it ended up pulling all
> > kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> > quite complex.
> 
> I have a problem to control the poplution order for IOMMU in Tegra
> SoC, and currently I haven't found any framework so that I inserted an
> IOMMU hook in the driver core[1] since I discussed with Stephen Warren
> and we thought that the number of those subsystem which requires this
> kind of control isn't so many. This patch depends on a certain DT
> bindings("#stream-id-cells") if a device has an dependency on IOMMU or
> not. But at least this doesn't require anything for a client driver
> side at all. The exsiting driver can be used as they are. The current
> design is:
> 
> 1, When a device is populated, it checks if that device is IOMMU'able
>    or not. This is identified by "#stream-id-cells" in DT. If
>    a device is normal(non IOMMU), a device is populated. If a device
>    is IOMMU'able, it continues to be checked.
> 
> 2, When an IOMMU device has been already populated,
>    "dev->bus->iommu_ops" is set. If so a device(client) is registered
>    as IOMMU'able, and the rest of population process continues. If not
>    set yet, a device population would be deferred til
>    "dev->bus->iommu_ops" is set.
> 
> 3, After an IOMMU device has been already populated, deferred
>    population will be continued.
> 
> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> or not, but I haven't any better idea right now.

It seems a little fragile to me too. I'd rather the IOMMU requirement be
described more explicitly.

g.

Re: Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Hi Grant,

Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:

> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> > Missed some MLs. Resending....
> > 
> > Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:
> > 
> > > Early Init, Deferred Probe and Init Order
> > ...
> > > The big problem with deferring to initcalls is that the kernel doesn?t
> > > have any information about dependencies between devices and so doesn?t
> > > know what order to call modules in. Right now driver probe order is
> > > roughly determined by kernel link order first, and registration order
> > > second for devices populated during or after initcalls. Olof commented
> > > that the current driver model is based on the assumption that devices
> > > live in a single hierarchy. There is no easy way to add in dependency
> > > data with the current structure, and in most cases dependencies aren?t
> > > specifically on other devices, but rather on the service provided by
> > > another device (ie, a GPIO line).
> > 
> > I think that an IOMMU device is also the service provider for other
> > devices, where an single IOMMU device resides in the bus to provide
> > the address translation to the bus children.
> > 
> > > Kumar is concerned that deferred probe won?t work in all situations. For
> > > instance, what about a device that has an optional dependency? Does it
> > > fail to probe in the hope that the dependency will show up later? It was
> > 
> > Other devices don't know if the system has IOMMU or not. IOMMU'able
> > devices' population can be deferred till IOMMU device is populated,
> > but they don't know if there's really IOMMU device exists or not at
> > their population time.
> 
> Deferring the population might complicate the common code by changing
> the time that devices get created. Right now the code is oriented around
> doing it all at once.
> 
> However, it would be possible to cause a probe deferal on devices that
> requires an IOMMU in a way that it would be triggered by the bus instead
> of by the individual device drivers.
> 
> I don't really know the details well enough to comment intelliently.
> Craft up the patches and see what it looks like.

I have just sent v5 as below:
  [PATCHv5 0/9] Unifying Tegra IOMMU(SMMU) driver among Tegra SoCs
  http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Driver core part is:
  [PATCHv5 2/9] driver/core: populate devices in order for IOMMUs
  http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007003.html

Any comment would be really appreciated.

> > > The suggestion was made that it would be a really nice to have the core
> > > kernel sort out probe order rather than using deferred probe. Grant
> > > replied that deferred probe was designed as the simplest possible
> > > solution to the problem, and by no means is he attached to it if someone
> > > can come up with a better approach. He did consider other options that
> > > put dependency resolution into the core, but it ended up pulling all
> > > kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> > > quite complex.
> > 
> > I have a problem to control the poplution order for IOMMU in Tegra
> > SoC, and currently I haven't found any framework so that I inserted an
> > IOMMU hook in the driver core[1] since I discussed with Stephen Warren
> > and we thought that the number of those subsystem which requires this
> > kind of control isn't so many. This patch depends on a certain DT
> > bindings("#stream-id-cells") if a device has an dependency on IOMMU or
> > not. But at least this doesn't require anything for a client driver
> > side at all. The exsiting driver can be used as they are. The current
> > design is:
> > 
> > 1, When a device is populated, it checks if that device is IOMMU'able
> >    or not. This is identified by "#stream-id-cells" in DT. If
> >    a device is normal(non IOMMU), a device is populated. If a device
> >    is IOMMU'able, it continues to be checked.
> > 
> > 2, When an IOMMU device has been already populated,
> >    "dev->bus->iommu_ops" is set. If so a device(client) is registered
> >    as IOMMU'able, and the rest of population process continues. If not
> >    set yet, a device population would be deferred til
> >    "dev->bus->iommu_ops" is set.
> > 
> > 3, After an IOMMU device has been already populated, deferred
> >    population will be continued.
> > 
> > I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> > or not, but I haven't any better idea right now.
> 
> It seems a little fragile to me too. I'd rather the IOMMU requirement be
> described more explicitly.

I think that Will Deacon can do better than I.

Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Hi Grant,

Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:

> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> > Missed some MLs. Resending....
> > 
> > Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 8 Nov 2013 14:20:46 +0100:
> > 
> > > Early Init, Deferred Probe and Init Order
> > ...
> > > The big problem with deferring to initcalls is that the kernel doesn?t
> > > have any information about dependencies between devices and so doesn?t
> > > know what order to call modules in. Right now driver probe order is
> > > roughly determined by kernel link order first, and registration order
> > > second for devices populated during or after initcalls. Olof commented
> > > that the current driver model is based on the assumption that devices
> > > live in a single hierarchy. There is no easy way to add in dependency
> > > data with the current structure, and in most cases dependencies aren?t
> > > specifically on other devices, but rather on the service provided by
> > > another device (ie, a GPIO line).
> > 
> > I think that an IOMMU device is also the service provider for other
> > devices, where an single IOMMU device resides in the bus to provide
> > the address translation to the bus children.
> > 
> > > Kumar is concerned that deferred probe won?t work in all situations. For
> > > instance, what about a device that has an optional dependency? Does it
> > > fail to probe in the hope that the dependency will show up later? It was
> > 
> > Other devices don't know if the system has IOMMU or not. IOMMU'able
> > devices' population can be deferred till IOMMU device is populated,
> > but they don't know if there's really IOMMU device exists or not at
> > their population time.
> 
> Deferring the population might complicate the common code by changing
> the time that devices get created. Right now the code is oriented around
> doing it all at once.
> 
> However, it would be possible to cause a probe deferal on devices that
> requires an IOMMU in a way that it would be triggered by the bus instead
> of by the individual device drivers.
> 
> I don't really know the details well enough to comment intelliently.
> Craft up the patches and see what it looks like.

I have just sent v5 as below:
  [PATCHv5 0/9] Unifying Tegra IOMMU(SMMU) driver among Tegra SoCs
  http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Driver core part is:
  [PATCHv5 2/9] driver/core: populate devices in order for IOMMUs
  http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007003.html

Any comment would be really appreciated.

> > > The suggestion was made that it would be a really nice to have the core
> > > kernel sort out probe order rather than using deferred probe. Grant
> > > replied that deferred probe was designed as the simplest possible
> > > solution to the problem, and by no means is he attached to it if someone
> > > can come up with a better approach. He did consider other options that
> > > put dependency resolution into the core, but it ended up pulling all
> > > kinds of GPIO, IRQ, Clock and DMA details into the core which made it
> > > quite complex.
> > 
> > I have a problem to control the poplution order for IOMMU in Tegra
> > SoC, and currently I haven't found any framework so that I inserted an
> > IOMMU hook in the driver core[1] since I discussed with Stephen Warren
> > and we thought that the number of those subsystem which requires this
> > kind of control isn't so many. This patch depends on a certain DT
> > bindings("#stream-id-cells") if a device has an dependency on IOMMU or
> > not. But at least this doesn't require anything for a client driver
> > side at all. The exsiting driver can be used as they are. The current
> > design is:
> > 
> > 1, When a device is populated, it checks if that device is IOMMU'able
> >    or not. This is identified by "#stream-id-cells" in DT. If
> >    a device is normal(non IOMMU), a device is populated. If a device
> >    is IOMMU'able, it continues to be checked.
> > 
> > 2, When an IOMMU device has been already populated,
> >    "dev->bus->iommu_ops" is set. If so a device(client) is registered
> >    as IOMMU'able, and the rest of population process continues. If not
> >    set yet, a device population would be deferred til
> >    "dev->bus->iommu_ops" is set.
> > 
> > 3, After an IOMMU device has been already populated, deferred
> >    population will be continued.
> > 
> > I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> > or not, but I haven't any better idea right now.
> 
> It seems a little fragile to me too. I'd rather the IOMMU requirement be
> described more explicitly.

I think that Will Deacon can do better than I.

Re: Report from 2013 ARM kernel summit

[Will Deacon]

On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
> > On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> > > 1, When a device is populated, it checks if that device is IOMMU'able
> > >    or not. This is identified by "#stream-id-cells" in DT. If
> > >    a device is normal(non IOMMU), a device is populated. If a device
> > >    is IOMMU'able, it continues to be checked.

[...]

> > > I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> > > or not, but I haven't any better idea right now.
> > 
> > It seems a little fragile to me too. I'd rather the IOMMU requirement be
> > described more explicitly.
> 
> I think that Will Deacon can do better than I.

I already commented briefly here:

  http://www.spinics.net/lists/devicetree/msg11513.html

basically deferring to DT people :)

Anyway, I'm happy to tighten up the IOMMU requirement description but
*not* at the expense of breaking what we currently have for the ARM SMMU,
which is being used by Calxeda.

Adding Andreas and Rob for input on potential binding additions to the SMMU.

Will

Report from 2013 ARM kernel summit

[Will Deacon]

On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
> > On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> > > 1, When a device is populated, it checks if that device is IOMMU'able
> > >    or not. This is identified by "#stream-id-cells" in DT. If
> > >    a device is normal(non IOMMU), a device is populated. If a device
> > >    is IOMMU'able, it continues to be checked.

[...]

> > > I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> > > or not, but I haven't any better idea right now.
> > 
> > It seems a little fragile to me too. I'd rather the IOMMU requirement be
> > described more explicitly.
> 
> I think that Will Deacon can do better than I.

I already commented briefly here:

  http://www.spinics.net/lists/devicetree/msg11513.html

basically deferring to DT people :)

Anyway, I'm happy to tighten up the IOMMU requirement description but
*not* at the expense of breaking what we currently have for the ARM SMMU,
which is being used by Calxeda.

Adding Andreas and Rob for input on potential binding additions to the SMMU.

Will

Re: Report from 2013 ARM kernel summit

[Rob Herring]

On 11/19/2013 11:35 AM, Will Deacon wrote:
> On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>>>> 1, When a device is populated, it checks if that device is IOMMU'able
>>>>    or not. This is identified by "#stream-id-cells" in DT. If
>>>>    a device is normal(non IOMMU), a device is populated. If a device
>>>>    is IOMMU'able, it continues to be checked.
> 
> [...]
> 
>>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>>>> or not, but I haven't any better idea right now.
>>>
>>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>>> described more explicitly.

I don't see how this can work. Typically you find a property and then
read the relevant #*-cells to determine the size. Having multiple cell
properties is asking for errors.

>>
>> I think that Will Deacon can do better than I.
> 
> I already commented briefly here:
> 
>   http://www.spinics.net/lists/devicetree/msg11513.html
> 
> basically deferring to DT people :)
> 
> Anyway, I'm happy to tighten up the IOMMU requirement description but
> *not* at the expense of breaking what we currently have for the ARM SMMU,
> which is being used by Calxeda.
> 
> Adding Andreas and Rob for input on potential binding additions to the SMMU.

The above proposal would be an incompatible change. However, I think we
could still deal with a change in this binding at this stage.

One way approach to handle this without changing the binding would be to
scan the DT for all iommu's up front and create a list of all nodes and
their iommu parent. The fact that the hierarchy is described in a way
that doesn't fit Linux well is really a Linux implementation detail.

If changing the binding, a simple approach would be to allow
'smmu-parent' to be a bus and/or device property and not just for
chained iommu's. This could be a global or bus property that is
inherited. Like interrupt-parent, you would have to deal with the parent
being itself. Also, perhaps iommu-parent would be a better name. In any
case, I'd like to see this all be a generic iommu binding.

Rob

Report from 2013 ARM kernel summit

[Rob Herring]

On 11/19/2013 11:35 AM, Will Deacon wrote:
> On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
>>>> 1, When a device is populated, it checks if that device is IOMMU'able
>>>>    or not. This is identified by "#stream-id-cells" in DT. If
>>>>    a device is normal(non IOMMU), a device is populated. If a device
>>>>    is IOMMU'able, it continues to be checked.
> 
> [...]
> 
>>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>>>> or not, but I haven't any better idea right now.
>>>
>>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>>> described more explicitly.

I don't see how this can work. Typically you find a property and then
read the relevant #*-cells to determine the size. Having multiple cell
properties is asking for errors.

>>
>> I think that Will Deacon can do better than I.
> 
> I already commented briefly here:
> 
>   http://www.spinics.net/lists/devicetree/msg11513.html
> 
> basically deferring to DT people :)
> 
> Anyway, I'm happy to tighten up the IOMMU requirement description but
> *not* at the expense of breaking what we currently have for the ARM SMMU,
> which is being used by Calxeda.
> 
> Adding Andreas and Rob for input on potential binding additions to the SMMU.

The above proposal would be an incompatible change. However, I think we
could still deal with a change in this binding at this stage.

One way approach to handle this without changing the binding would be to
scan the DT for all iommu's up front and create a list of all nodes and
their iommu parent. The fact that the hierarchy is described in a way
that doesn't fit Linux well is really a Linux implementation detail.

If changing the binding, a simple approach would be to allow
'smmu-parent' to be a bus and/or device property and not just for
chained iommu's. This could be a global or bus property that is
inherited. Like interrupt-parent, you would have to deal with the parent
being itself. Also, perhaps iommu-parent would be a better name. In any
case, I'd like to see this all be a generic iommu binding.

Rob

Re: Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Hi Rob,

Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:

> On 11/19/2013 11:35 AM, Will Deacon wrote:
> > On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
> >> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
> >>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> >>>> 1, When a device is populated, it checks if that device is IOMMU'able
> >>>>    or not. This is identified by "#stream-id-cells" in DT. If
> >>>>    a device is normal(non IOMMU), a device is populated. If a device
> >>>>    is IOMMU'able, it continues to be checked.
> > 
> > [...]
> > 
> >>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> >>>> or not, but I haven't any better idea right now.
> >>>
> >>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
> >>> described more explicitly.
> 
> I don't see how this can work. Typically you find a property and then
> read the relevant #*-cells to determine the size. Having multiple cell
> properties is asking for errors.

The above was mentioned for PATCHv4 series[1], which used the arm,smmu
DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
iommu.

In PATCHv5[2], we took the following DT binding where multiple cell
properties seem to work ok.

  smmu_a: iommu@xxxxxxxx {
  	#iommu-cells = <2>;
  	....
  };
  
  smmu_b: iommu@xxxxxxxx {
  	#iommu-cells = <3>;
  	....
  };
  
  device_a {
  	 iommus = <&smmu_a param1 param2>,
  	 	  <&smmu_b param1 param2 param3>;
  };

This can describe the relation between a device and an iommu
independently. The number of params needed for each IOMMU can be
sepcified by #iommu-cells in its iommu entry.

  device_a <-> smmu_a, needs 2 params for a device
  device_a <-> smmu_b, needs 3 params for a device

For example, "smmu_a" can be an bus level global IOMMU where all child
devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
only for "device_a".

"memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
                                      |
                                      |
                                      +--"device_b"

> >> I think that Will Deacon can do better than I.
> > 
> > I already commented briefly here:
> > 
> >   http://www.spinics.net/lists/devicetree/msg11513.html
> > 
> > basically deferring to DT people :)
> > 
> > Anyway, I'm happy to tighten up the IOMMU requirement description but
> > *not* at the expense of breaking what we currently have for the ARM SMMU,
> > which is being used by Calxeda.
> > 
> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> 
> The above proposal would be an incompatible change. However, I think we
> could still deal with a change in this binding at this stage.
> 
> One way approach to handle this without changing the binding would be to
> scan the DT for all iommu's up front and create a list of all nodes and
> their iommu parent. The fact that the hierarchy is described in a way
> that doesn't fit Linux well is really a Linux implementation detail.

I may need some implementation to understand this further.

> If changing the binding, a simple approach would be to allow
> 'smmu-parent' to be a bus and/or device property and not just for
> chained iommu's. This could be a global or bus property that is
> inherited. Like interrupt-parent, you would have to deal with the parent
> being itself. Also, perhaps iommu-parent would be a better name. In any
> case, I'd like to see this all be a generic iommu binding.

I guess that this would work.

One concern to this may be that there might be the case where IOMMU
hierarchy doesn't always follow the bus's one, I guess. I'm not so
sure if it's an good example, but, platform_bus itself doesn't follow
the actual bus hierarchy at all. But if we start to describe the bus
hierarchy(adding a specific bus) we'll loose the benefit of
platform_bus, where we can share almost all devices in a single
driver.

OTOH, do you see any limitations of point-to-point connections between
a device and IOMMUs, described in the above "device_a" and "smmu_[a|b]"?

[1] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006931.html
[2] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Report from 2013 ARM kernel summit

[Hiroshi Doyu]

Hi Rob,

Rob Herring <robherring2@gmail.com> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:

> On 11/19/2013 11:35 AM, Will Deacon wrote:
> > On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
> >> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
> >>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> >>>> 1, When a device is populated, it checks if that device is IOMMU'able
> >>>>    or not. This is identified by "#stream-id-cells" in DT. If
> >>>>    a device is normal(non IOMMU), a device is populated. If a device
> >>>>    is IOMMU'able, it continues to be checked.
> > 
> > [...]
> > 
> >>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
> >>>> or not, but I haven't any better idea right now.
> >>>
> >>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
> >>> described more explicitly.
> 
> I don't see how this can work. Typically you find a property and then
> read the relevant #*-cells to determine the size. Having multiple cell
> properties is asking for errors.

The above was mentioned for PATCHv4 series[1], which used the arm,smmu
DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
iommu.

In PATCHv5[2], we took the following DT binding where multiple cell
properties seem to work ok.

  smmu_a: iommu at xxxxxxxx {
  	#iommu-cells = <2>;
  	....
  };
  
  smmu_b: iommu at xxxxxxxx {
  	#iommu-cells = <3>;
  	....
  };
  
  device_a {
  	 iommus = <&smmu_a param1 param2>,
  	 	  <&smmu_b param1 param2 param3>;
  };

This can describe the relation between a device and an iommu
independently. The number of params needed for each IOMMU can be
sepcified by #iommu-cells in its iommu entry.

  device_a <-> smmu_a, needs 2 params for a device
  device_a <-> smmu_b, needs 3 params for a device

For example, "smmu_a" can be an bus level global IOMMU where all child
devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
only for "device_a".

"memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
                                      |
                                      |
                                      +--"device_b"

> >> I think that Will Deacon can do better than I.
> > 
> > I already commented briefly here:
> > 
> >   http://www.spinics.net/lists/devicetree/msg11513.html
> > 
> > basically deferring to DT people :)
> > 
> > Anyway, I'm happy to tighten up the IOMMU requirement description but
> > *not* at the expense of breaking what we currently have for the ARM SMMU,
> > which is being used by Calxeda.
> > 
> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> 
> The above proposal would be an incompatible change. However, I think we
> could still deal with a change in this binding at this stage.
> 
> One way approach to handle this without changing the binding would be to
> scan the DT for all iommu's up front and create a list of all nodes and
> their iommu parent. The fact that the hierarchy is described in a way
> that doesn't fit Linux well is really a Linux implementation detail.

I may need some implementation to understand this further.

> If changing the binding, a simple approach would be to allow
> 'smmu-parent' to be a bus and/or device property and not just for
> chained iommu's. This could be a global or bus property that is
> inherited. Like interrupt-parent, you would have to deal with the parent
> being itself. Also, perhaps iommu-parent would be a better name. In any
> case, I'd like to see this all be a generic iommu binding.

I guess that this would work.

One concern to this may be that there might be the case where IOMMU
hierarchy doesn't always follow the bus's one, I guess. I'm not so
sure if it's an good example, but, platform_bus itself doesn't follow
the actual bus hierarchy at all. But if we start to describe the bus
hierarchy(adding a specific bus) we'll loose the benefit of
platform_bus, where we can share almost all devices in a single
driver.

OTOH, do you see any limitations of point-to-point connections between
a device and IOMMUs, described in the above "device_a" and "smmu_[a|b]"?

[1] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006931.html
[2] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Re: Report from 2013 ARM kernel summit

[Will Deacon]

On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
> On 11/19/2013 11:35 AM, Will Deacon wrote:
> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> 
> The above proposal would be an incompatible change. However, I think we
> could still deal with a change in this binding at this stage.
> 
> One way approach to handle this without changing the binding would be to
> scan the DT for all iommu's up front and create a list of all nodes and
> their iommu parent. The fact that the hierarchy is described in a way
> that doesn't fit Linux well is really a Linux implementation detail.
> 
> If changing the binding, a simple approach would be to allow
> 'smmu-parent' to be a bus and/or device property and not just for
> chained iommu's. This could be a global or bus property that is
> inherited. Like interrupt-parent, you would have to deal with the parent
> being itself. Also, perhaps iommu-parent would be a better name. In any
> case, I'd like to see this all be a generic iommu binding.

I like that idea. I've recently been toying with removing the chained IOMMU
support, since I don't think anybody is using it who is interested in
mainline. However, making it more general sounds like a better idea.

One potential issue is that I think the nvidia guys want to describe masters
that master via multiple SMMUs (which I believe was the motivation for
moving the stream-ids out into the master nodes, rather than keeping them in
the SMMU). Again, that's not something we can easily add to the arm-smmu,
because the incoming stream-ids are a property of the SMMU node.

So the question is: do we actually need to describe masters that master
through multiple SMMUs as a single node in the devicetree?

Will

Report from 2013 ARM kernel summit

[Will Deacon]

On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
> On 11/19/2013 11:35 AM, Will Deacon wrote:
> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> 
> The above proposal would be an incompatible change. However, I think we
> could still deal with a change in this binding at this stage.
> 
> One way approach to handle this without changing the binding would be to
> scan the DT for all iommu's up front and create a list of all nodes and
> their iommu parent. The fact that the hierarchy is described in a way
> that doesn't fit Linux well is really a Linux implementation detail.
> 
> If changing the binding, a simple approach would be to allow
> 'smmu-parent' to be a bus and/or device property and not just for
> chained iommu's. This could be a global or bus property that is
> inherited. Like interrupt-parent, you would have to deal with the parent
> being itself. Also, perhaps iommu-parent would be a better name. In any
> case, I'd like to see this all be a generic iommu binding.

I like that idea. I've recently been toying with removing the chained IOMMU
support, since I don't think anybody is using it who is interested in
mainline. However, making it more general sounds like a better idea.

One potential issue is that I think the nvidia guys want to describe masters
that master via multiple SMMUs (which I believe was the motivation for
moving the stream-ids out into the master nodes, rather than keeping them in
the SMMU). Again, that's not something we can easily add to the arm-smmu,
because the incoming stream-ids are a property of the SMMU node.

So the question is: do we actually need to describe masters that master
through multiple SMMUs as a single node in the devicetree?

Will

Re: Report from 2013 ARM kernel summit

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 2769 bytes --]

On Wed, Nov 20, 2013 at 10:31:11AM +0000, Will Deacon wrote:
> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
> > On 11/19/2013 11:35 AM, Will Deacon wrote:
> > > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> > 
> > The above proposal would be an incompatible change. However, I think we
> > could still deal with a change in this binding at this stage.
> > 
> > One way approach to handle this without changing the binding would be to
> > scan the DT for all iommu's up front and create a list of all nodes and
> > their iommu parent. The fact that the hierarchy is described in a way
> > that doesn't fit Linux well is really a Linux implementation detail.
> > 
> > If changing the binding, a simple approach would be to allow
> > 'smmu-parent' to be a bus and/or device property and not just for
> > chained iommu's. This could be a global or bus property that is
> > inherited. Like interrupt-parent, you would have to deal with the parent
> > being itself. Also, perhaps iommu-parent would be a better name. In any
> > case, I'd like to see this all be a generic iommu binding.
> 
> I like that idea. I've recently been toying with removing the chained IOMMU
> support, since I don't think anybody is using it who is interested in
> mainline. However, making it more general sounds like a better idea.
> 
> One potential issue is that I think the nvidia guys want to describe masters
> that master via multiple SMMUs (which I believe was the motivation for
> moving the stream-ids out into the master nodes, rather than keeping them in
> the SMMU). Again, that's not something we can easily add to the arm-smmu,
> because the incoming stream-ids are a property of the SMMU node.

If I remember correctly, one of the reasons for the proposal was also
that the interrupt-parent property turned out to be insufficient for
some use-cases, which lead to Grant's proposal of the new interrupts-
extended property. Since that comparison has already been drawn, I think
we can agree that both are used in similar ways. Therefore we should
consider what we've learned from interrupt-parent when designing this
generic IOMMU binding to avoid having to introduce iommu-extended at
some point.

> So the question is: do we actually need to describe masters that master
> through multiple SMMUs as a single node in the devicetree?

I would think so, yes. The alternative would be to have several nodes
that describe the same device, and that conflicts on a different level.

Perhaps it could be done by having separate sub-nodes that each use a
different IOMMU, but that sounds like a much grosser solution. That
pretty much boils down to interrupt-parent/interrupt-map.

Thierry

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Report from 2013 ARM kernel summit

[Thierry Reding]

On Wed, Nov 20, 2013 at 10:31:11AM +0000, Will Deacon wrote:
> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
> > On 11/19/2013 11:35 AM, Will Deacon wrote:
> > > Adding Andreas and Rob for input on potential binding additions to the SMMU.
> > 
> > The above proposal would be an incompatible change. However, I think we
> > could still deal with a change in this binding at this stage.
> > 
> > One way approach to handle this without changing the binding would be to
> > scan the DT for all iommu's up front and create a list of all nodes and
> > their iommu parent. The fact that the hierarchy is described in a way
> > that doesn't fit Linux well is really a Linux implementation detail.
> > 
> > If changing the binding, a simple approach would be to allow
> > 'smmu-parent' to be a bus and/or device property and not just for
> > chained iommu's. This could be a global or bus property that is
> > inherited. Like interrupt-parent, you would have to deal with the parent
> > being itself. Also, perhaps iommu-parent would be a better name. In any
> > case, I'd like to see this all be a generic iommu binding.
> 
> I like that idea. I've recently been toying with removing the chained IOMMU
> support, since I don't think anybody is using it who is interested in
> mainline. However, making it more general sounds like a better idea.
> 
> One potential issue is that I think the nvidia guys want to describe masters
> that master via multiple SMMUs (which I believe was the motivation for
> moving the stream-ids out into the master nodes, rather than keeping them in
> the SMMU). Again, that's not something we can easily add to the arm-smmu,
> because the incoming stream-ids are a property of the SMMU node.

If I remember correctly, one of the reasons for the proposal was also
that the interrupt-parent property turned out to be insufficient for
some use-cases, which lead to Grant's proposal of the new interrupts-
extended property. Since that comparison has already been drawn, I think
we can agree that both are used in similar ways. Therefore we should
consider what we've learned from interrupt-parent when designing this
generic IOMMU binding to avoid having to introduce iommu-extended at
some point.

> So the question is: do we actually need to describe masters that master
> through multiple SMMUs as a single node in the devicetree?

I would think so, yes. The alternative would be to have several nodes
that describe the same device, and that conflicts on a different level.

Perhaps it could be done by having separate sub-nodes that each use a
different IOMMU, but that sounds like a much grosser solution. That
pretty much boils down to interrupt-parent/interrupt-map.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20131120/58474dce/attachment.sig>

Re: Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 7:53 AM, Thierry Reding
<thierry.reding-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Nov 20, 2013 at 10:31:11AM +0000, Will Deacon wrote:
>> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
>> > On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>> >
>> > The above proposal would be an incompatible change. However, I think we
>> > could still deal with a change in this binding at this stage.
>> >
>> > One way approach to handle this without changing the binding would be to
>> > scan the DT for all iommu's up front and create a list of all nodes and
>> > their iommu parent. The fact that the hierarchy is described in a way
>> > that doesn't fit Linux well is really a Linux implementation detail.
>> >
>> > If changing the binding, a simple approach would be to allow
>> > 'smmu-parent' to be a bus and/or device property and not just for
>> > chained iommu's. This could be a global or bus property that is
>> > inherited. Like interrupt-parent, you would have to deal with the parent
>> > being itself. Also, perhaps iommu-parent would be a better name. In any
>> > case, I'd like to see this all be a generic iommu binding.
>>
>> I like that idea. I've recently been toying with removing the chained IOMMU
>> support, since I don't think anybody is using it who is interested in
>> mainline. However, making it more general sounds like a better idea.
>>
>> One potential issue is that I think the nvidia guys want to describe masters
>> that master via multiple SMMUs (which I believe was the motivation for
>> moving the stream-ids out into the master nodes, rather than keeping them in
>> the SMMU). Again, that's not something we can easily add to the arm-smmu,
>> because the incoming stream-ids are a property of the SMMU node.
>
> If I remember correctly, one of the reasons for the proposal was also
> that the interrupt-parent property turned out to be insufficient for
> some use-cases, which lead to Grant's proposal of the new interrupts-
> extended property. Since that comparison has already been drawn, I think
> we can agree that both are used in similar ways. Therefore we should
> consider what we've learned from interrupt-parent when designing this
> generic IOMMU binding to avoid having to introduce iommu-extended at
> some point.

The problem in that case is really the interrupts property not having
a phandle. This case would be a bit different because there is not an
analogous property to interrupts.

However, I don't really like connections described via phandles being
done differently for different bindings. And this one has been a bit
unusual.

>> So the question is: do we actually need to describe masters that master
>> through multiple SMMUs as a single node in the devicetree?
>
> I would think so, yes. The alternative would be to have several nodes
> that describe the same device, and that conflicts on a different level.
>
> Perhaps it could be done by having separate sub-nodes that each use a
> different IOMMU, but that sounds like a much grosser solution. That
> pretty much boils down to interrupt-parent/interrupt-map.

Perhaps not within the DT, but we could have this case with IOMMUs
within PCIe bus and the PCIe RC itself behind an IOMMU.

I'll reply to Hiroshi's email with more on a variation of his proposal.

Rob

Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 7:53 AM, Thierry Reding
<thierry.reding@gmail.com> wrote:
> On Wed, Nov 20, 2013 at 10:31:11AM +0000, Will Deacon wrote:
>> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
>> > On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>> >
>> > The above proposal would be an incompatible change. However, I think we
>> > could still deal with a change in this binding at this stage.
>> >
>> > One way approach to handle this without changing the binding would be to
>> > scan the DT for all iommu's up front and create a list of all nodes and
>> > their iommu parent. The fact that the hierarchy is described in a way
>> > that doesn't fit Linux well is really a Linux implementation detail.
>> >
>> > If changing the binding, a simple approach would be to allow
>> > 'smmu-parent' to be a bus and/or device property and not just for
>> > chained iommu's. This could be a global or bus property that is
>> > inherited. Like interrupt-parent, you would have to deal with the parent
>> > being itself. Also, perhaps iommu-parent would be a better name. In any
>> > case, I'd like to see this all be a generic iommu binding.
>>
>> I like that idea. I've recently been toying with removing the chained IOMMU
>> support, since I don't think anybody is using it who is interested in
>> mainline. However, making it more general sounds like a better idea.
>>
>> One potential issue is that I think the nvidia guys want to describe masters
>> that master via multiple SMMUs (which I believe was the motivation for
>> moving the stream-ids out into the master nodes, rather than keeping them in
>> the SMMU). Again, that's not something we can easily add to the arm-smmu,
>> because the incoming stream-ids are a property of the SMMU node.
>
> If I remember correctly, one of the reasons for the proposal was also
> that the interrupt-parent property turned out to be insufficient for
> some use-cases, which lead to Grant's proposal of the new interrupts-
> extended property. Since that comparison has already been drawn, I think
> we can agree that both are used in similar ways. Therefore we should
> consider what we've learned from interrupt-parent when designing this
> generic IOMMU binding to avoid having to introduce iommu-extended at
> some point.

The problem in that case is really the interrupts property not having
a phandle. This case would be a bit different because there is not an
analogous property to interrupts.

However, I don't really like connections described via phandles being
done differently for different bindings. And this one has been a bit
unusual.

>> So the question is: do we actually need to describe masters that master
>> through multiple SMMUs as a single node in the devicetree?
>
> I would think so, yes. The alternative would be to have several nodes
> that describe the same device, and that conflicts on a different level.
>
> Perhaps it could be done by having separate sub-nodes that each use a
> different IOMMU, but that sounds like a much grosser solution. That
> pretty much boils down to interrupt-parent/interrupt-map.

Perhaps not within the DT, but we could have this case with IOMMUs
within PCIe bus and the PCIe RC itself behind an IOMMU.

I'll reply to Hiroshi's email with more on a variation of his proposal.

Rob

Re: Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> Hi Rob,
>
> Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:
>
>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>> >> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>> >>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>> >>>> 1, When a device is populated, it checks if that device is IOMMU'able
>> >>>>    or not. This is identified by "#stream-id-cells" in DT. If
>> >>>>    a device is normal(non IOMMU), a device is populated. If a device
>> >>>>    is IOMMU'able, it continues to be checked.
>> >
>> > [...]
>> >
>> >>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>> >>>> or not, but I haven't any better idea right now.
>> >>>
>> >>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>> >>> described more explicitly.
>>
>> I don't see how this can work. Typically you find a property and then
>> read the relevant #*-cells to determine the size. Having multiple cell
>> properties is asking for errors.
>
> The above was mentioned for PATCHv4 series[1], which used the arm,smmu
> DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
> iommu.
>
> In PATCHv5[2], we took the following DT binding where multiple cell
> properties seem to work ok.
>
>   smmu_a: iommu@xxxxxxxx {
>         #iommu-cells = <2>;
>         ....
>   };
>
>   smmu_b: iommu@xxxxxxxx {
>         #iommu-cells = <3>;
>         ....
>   };
>
>   device_a {
>          iommus = <&smmu_a param1 param2>,
>                   <&smmu_b param1 param2 param3>;
>   };
>
> This can describe the relation between a device and an iommu
> independently. The number of params needed for each IOMMU can be
> sepcified by #iommu-cells in its iommu entry.
>
>   device_a <-> smmu_a, needs 2 params for a device
>   device_a <-> smmu_b, needs 3 params for a device
>
> For example, "smmu_a" can be an bus level global IOMMU where all child
> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> only for "device_a".
>
> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>                                       |
>                                       |
>                                       +--"device_b"

I think the above binding would be the correct way to describe things
if you have 1 device connected to 2 IOMMUs (directly rather than
chained). IIUC, that is something you have on tegra?

For the topology above where you are chaining iommu's, I think
something like this is more accurately describing the hierarchy:

  smmu_b: iommu@xxxxxxxx {
        #iommu-cells = <3>;
         iommus = <&smmu_a param1 param2>;
       ....
  };
  device_a {
         iommus = <&smmu_b param1 param2 param3>;
  };

I remember discussing this with Will and seem to recall some issue
with describing things this way. But looking at it now, I don't see
what that was.

>> >> I think that Will Deacon can do better than I.
>> >
>> > I already commented briefly here:
>> >
>> >   http://www.spinics.net/lists/devicetree/msg11513.html
>> >
>> > basically deferring to DT people :)
>> >
>> > Anyway, I'm happy to tighten up the IOMMU requirement description but
>> > *not* at the expense of breaking what we currently have for the ARM SMMU,
>> > which is being used by Calxeda.
>> >
>> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>>
>> The above proposal would be an incompatible change. However, I think we
>> could still deal with a change in this binding at this stage.
>>
>> One way approach to handle this without changing the binding would be to
>> scan the DT for all iommu's up front and create a list of all nodes and
>> their iommu parent. The fact that the hierarchy is described in a way
>> that doesn't fit Linux well is really a Linux implementation detail.
>
> I may need some implementation to understand this further.
>
>> If changing the binding, a simple approach would be to allow
>> 'smmu-parent' to be a bus and/or device property and not just for
>> chained iommu's. This could be a global or bus property that is
>> inherited. Like interrupt-parent, you would have to deal with the parent
>> being itself. Also, perhaps iommu-parent would be a better name. In any
>> case, I'd like to see this all be a generic iommu binding.
>
> I guess that this would work.
>
> One concern to this may be that there might be the case where IOMMU
> hierarchy doesn't always follow the bus's one, I guess. I'm not so
> sure if it's an good example, but, platform_bus itself doesn't follow
> the actual bus hierarchy at all. But if we start to describe the bus
> hierarchy(adding a specific bus) we'll loose the benefit of
> platform_bus, where we can share almost all devices in a single
> driver.

It is not trying to follow the bus hierarchy. Inheritance is just a
convenience. Interrupts don't follow the bus hierarchy either.

Rob

>
> OTOH, do you see any limitations of point-to-point connections between
> a device and IOMMUs, described in the above "device_a" and "smmu_[a|b]"?
>
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006931.html
> [2] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> Hi Rob,
>
> Rob Herring <robherring2@gmail.com> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:
>
>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>> >> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>> >>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
>> >>>> 1, When a device is populated, it checks if that device is IOMMU'able
>> >>>>    or not. This is identified by "#stream-id-cells" in DT. If
>> >>>>    a device is normal(non IOMMU), a device is populated. If a device
>> >>>>    is IOMMU'able, it continues to be checked.
>> >
>> > [...]
>> >
>> >>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>> >>>> or not, but I haven't any better idea right now.
>> >>>
>> >>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>> >>> described more explicitly.
>>
>> I don't see how this can work. Typically you find a property and then
>> read the relevant #*-cells to determine the size. Having multiple cell
>> properties is asking for errors.
>
> The above was mentioned for PATCHv4 series[1], which used the arm,smmu
> DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
> iommu.
>
> In PATCHv5[2], we took the following DT binding where multiple cell
> properties seem to work ok.
>
>   smmu_a: iommu at xxxxxxxx {
>         #iommu-cells = <2>;
>         ....
>   };
>
>   smmu_b: iommu at xxxxxxxx {
>         #iommu-cells = <3>;
>         ....
>   };
>
>   device_a {
>          iommus = <&smmu_a param1 param2>,
>                   <&smmu_b param1 param2 param3>;
>   };
>
> This can describe the relation between a device and an iommu
> independently. The number of params needed for each IOMMU can be
> sepcified by #iommu-cells in its iommu entry.
>
>   device_a <-> smmu_a, needs 2 params for a device
>   device_a <-> smmu_b, needs 3 params for a device
>
> For example, "smmu_a" can be an bus level global IOMMU where all child
> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> only for "device_a".
>
> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>                                       |
>                                       |
>                                       +--"device_b"

I think the above binding would be the correct way to describe things
if you have 1 device connected to 2 IOMMUs (directly rather than
chained). IIUC, that is something you have on tegra?

For the topology above where you are chaining iommu's, I think
something like this is more accurately describing the hierarchy:

  smmu_b: iommu at xxxxxxxx {
        #iommu-cells = <3>;
         iommus = <&smmu_a param1 param2>;
       ....
  };
  device_a {
         iommus = <&smmu_b param1 param2 param3>;
  };

I remember discussing this with Will and seem to recall some issue
with describing things this way. But looking at it now, I don't see
what that was.

>> >> I think that Will Deacon can do better than I.
>> >
>> > I already commented briefly here:
>> >
>> >   http://www.spinics.net/lists/devicetree/msg11513.html
>> >
>> > basically deferring to DT people :)
>> >
>> > Anyway, I'm happy to tighten up the IOMMU requirement description but
>> > *not* at the expense of breaking what we currently have for the ARM SMMU,
>> > which is being used by Calxeda.
>> >
>> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>>
>> The above proposal would be an incompatible change. However, I think we
>> could still deal with a change in this binding at this stage.
>>
>> One way approach to handle this without changing the binding would be to
>> scan the DT for all iommu's up front and create a list of all nodes and
>> their iommu parent. The fact that the hierarchy is described in a way
>> that doesn't fit Linux well is really a Linux implementation detail.
>
> I may need some implementation to understand this further.
>
>> If changing the binding, a simple approach would be to allow
>> 'smmu-parent' to be a bus and/or device property and not just for
>> chained iommu's. This could be a global or bus property that is
>> inherited. Like interrupt-parent, you would have to deal with the parent
>> being itself. Also, perhaps iommu-parent would be a better name. In any
>> case, I'd like to see this all be a generic iommu binding.
>
> I guess that this would work.
>
> One concern to this may be that there might be the case where IOMMU
> hierarchy doesn't always follow the bus's one, I guess. I'm not so
> sure if it's an good example, but, platform_bus itself doesn't follow
> the actual bus hierarchy at all. But if we start to describe the bus
> hierarchy(adding a specific bus) we'll loose the benefit of
> platform_bus, where we can share almost all devices in a single
> driver.

It is not trying to follow the bus hierarchy. Inheritance is just a
convenience. Interrupts don't follow the bus hierarchy either.

Rob

>
> OTOH, do you see any limitations of point-to-point connections between
> a device and IOMMUs, described in the above "device_a" and "smmu_[a|b]"?
>
> [1] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/006931.html
> [2] http://lists.linuxfoundation.org/pipermail/iommu/2013-November/007004.html

Re: Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 4:31 AM, Will Deacon <will.deacon-5wv7dgnIgG8@public.gmane.org> wrote:
> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>>
>> The above proposal would be an incompatible change. However, I think we
>> could still deal with a change in this binding at this stage.
>>
>> One way approach to handle this without changing the binding would be to
>> scan the DT for all iommu's up front and create a list of all nodes and
>> their iommu parent. The fact that the hierarchy is described in a way
>> that doesn't fit Linux well is really a Linux implementation detail.
>>
>> If changing the binding, a simple approach would be to allow
>> 'smmu-parent' to be a bus and/or device property and not just for
>> chained iommu's. This could be a global or bus property that is
>> inherited. Like interrupt-parent, you would have to deal with the parent
>> being itself. Also, perhaps iommu-parent would be a better name. In any
>> case, I'd like to see this all be a generic iommu binding.
>
> I like that idea. I've recently been toying with removing the chained IOMMU
> support, since I don't think anybody is using it who is interested in
> mainline. However, making it more general sounds like a better idea.
>
> One potential issue is that I think the nvidia guys want to describe masters
> that master via multiple SMMUs (which I believe was the motivation for
> moving the stream-ids out into the master nodes, rather than keeping them in
> the SMMU). Again, that's not something we can easily add to the arm-smmu,
> because the incoming stream-ids are a property of the SMMU node.

For stream-ids, I think that is debatable. It is not a property of the
SMMU. The AXI ID is determined by the master. Yes, AXI ID is only part
of it and the SMMU determines the full stream-id, but that doesn't
really matter. Look at interrupts properties as a comparison. The
number of cells and the meaning of the cell values is determined by
the interrupt controller, not the node with the interrupts property.

Rob

> So the question is: do we actually need to describe masters that master
> through multiple SMMUs as a single node in the devicetree?
>
> Will

Report from 2013 ARM kernel summit

[Rob Herring]

On Wed, Nov 20, 2013 at 4:31 AM, Will Deacon <will.deacon@arm.com> wrote:
> On Tue, Nov 19, 2013 at 08:45:02PM +0000, Rob Herring wrote:
>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>> > Adding Andreas and Rob for input on potential binding additions to the SMMU.
>>
>> The above proposal would be an incompatible change. However, I think we
>> could still deal with a change in this binding at this stage.
>>
>> One way approach to handle this without changing the binding would be to
>> scan the DT for all iommu's up front and create a list of all nodes and
>> their iommu parent. The fact that the hierarchy is described in a way
>> that doesn't fit Linux well is really a Linux implementation detail.
>>
>> If changing the binding, a simple approach would be to allow
>> 'smmu-parent' to be a bus and/or device property and not just for
>> chained iommu's. This could be a global or bus property that is
>> inherited. Like interrupt-parent, you would have to deal with the parent
>> being itself. Also, perhaps iommu-parent would be a better name. In any
>> case, I'd like to see this all be a generic iommu binding.
>
> I like that idea. I've recently been toying with removing the chained IOMMU
> support, since I don't think anybody is using it who is interested in
> mainline. However, making it more general sounds like a better idea.
>
> One potential issue is that I think the nvidia guys want to describe masters
> that master via multiple SMMUs (which I believe was the motivation for
> moving the stream-ids out into the master nodes, rather than keeping them in
> the SMMU). Again, that's not something we can easily add to the arm-smmu,
> because the incoming stream-ids are a property of the SMMU node.

For stream-ids, I think that is debatable. It is not a property of the
SMMU. The AXI ID is determined by the master. Yes, AXI ID is only part
of it and the SMMU determines the full stream-id, but that doesn't
really matter. Look at interrupts properties as a comparison. The
number of cells and the meaning of the cell values is determined by
the interrupt controller, not the node with the interrupts property.

Rob

> So the question is: do we actually need to describe masters that master
> through multiple SMMUs as a single node in the devicetree?
>
> Will

Re: Report from 2013 ARM kernel summit

[Stephen Warren]

On 11/20/2013 01:02 PM, Rob Herring wrote:
> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>> Hi Rob,
>>
>> Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:
>>
>>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>>>> On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>>>>> Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>>>>>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>>>>>>> 1, When a device is populated, it checks if that device is IOMMU'able
>>>>>>>    or not. This is identified by "#stream-id-cells" in DT. If
>>>>>>>    a device is normal(non IOMMU), a device is populated. If a device
>>>>>>>    is IOMMU'able, it continues to be checked.
>>>>
>>>> [...]
>>>>
>>>>>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>>>>>>> or not, but I haven't any better idea right now.
>>>>>>
>>>>>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>>>>>> described more explicitly.
>>>
>>> I don't see how this can work. Typically you find a property and then
>>> read the relevant #*-cells to determine the size. Having multiple cell
>>> properties is asking for errors.
>>
>> The above was mentioned for PATCHv4 series[1], which used the arm,smmu
>> DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
>> iommu.
>>
>> In PATCHv5[2], we took the following DT binding where multiple cell
>> properties seem to work ok.
>>
>>   smmu_a: iommu@xxxxxxxx {
>>         #iommu-cells = <2>;
>>         ....
>>   };
>>
>>   smmu_b: iommu@xxxxxxxx {
>>         #iommu-cells = <3>;
>>         ....
>>   };
>>
>>   device_a {
>>          iommus = <&smmu_a param1 param2>,
>>                   <&smmu_b param1 param2 param3>;
>>   };
>>
>> This can describe the relation between a device and an iommu
>> independently. The number of params needed for each IOMMU can be
>> sepcified by #iommu-cells in its iommu entry.
>>
>>   device_a <-> smmu_a, needs 2 params for a device
>>   device_a <-> smmu_b, needs 3 params for a device
>>
>> For example, "smmu_a" can be an bus level global IOMMU where all child
>> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
>> only for "device_a".
>>
>> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>>                                       |
>>                                       |
>>                                       +--"device_b"
> 
> I think the above binding would be the correct way to describe things
> if you have 1 device connected to 2 IOMMUs (directly rather than
> chained). IIUC, that is something you have on tegra?

I'm not sure that we do, but perhaps; Hiroshi will have to answer.
Certainly whenever I personally mentioned chained IOMMUs it was simply
in the context of making sure the bindings allowed for any potential
arbitrary HW configuration, not because I had specific knowledge of one
that actually exists.

> For the topology above where you are chaining iommu's, I think
> something like this is more accurately describing the hierarchy:
> 
>   smmu_b: iommu@xxxxxxxx {
>         #iommu-cells = <3>;
>          iommus = <&smmu_a param1 param2>;
>        ....
>   };
>   device_a {
>          iommus = <&smmu_b param1 param2 param3>;
>   };
> 
> I remember discussing this with Will and seem to recall some issue
> with describing things this way. But looking at it now, I don't see
> what that was.

That's the DT content I would expect for that scenario.

The syntax:

>>          iommus = <&smmu_a param1 param2>,
>>                   <&smmu_b param1 param2 param3>;

... I would expect to be useful if a single device was a bus-master on
multiple buses, and each bus had a path to memory via a different IOMMU.
One example might be a DMA controller that bridges two buses. We
certainly have such a DMA controller on Tegra, although IIRC, the
upstream path to memory would pass through the same IOMMU from both
buses on current HW that I'm familiar with.

Report from 2013 ARM kernel summit

[Stephen Warren]

On 11/20/2013 01:02 PM, Rob Herring wrote:
> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
>> Hi Rob,
>>
>> Rob Herring <robherring2@gmail.com> wrote @ Tue, 19 Nov 2013 21:45:02 +0100:
>>
>>> On 11/19/2013 11:35 AM, Will Deacon wrote:
>>>> On Tue, Nov 19, 2013 at 09:40:54AM +0000, Hiroshi Doyu wrote:
>>>>> Grant Likely <grant.likely@secretlab.ca> wrote @ Fri, 15 Nov 2013 08:06:27 +0100:
>>>>>> On Mon, 11 Nov 2013 10:47:23 +0100, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
>>>>>>> 1, When a device is populated, it checks if that device is IOMMU'able
>>>>>>>    or not. This is identified by "#stream-id-cells" in DT. If
>>>>>>>    a device is normal(non IOMMU), a device is populated. If a device
>>>>>>>    is IOMMU'able, it continues to be checked.
>>>>
>>>> [...]
>>>>
>>>>>>> I'm not so sure if this dependecy on "#stream-id-cells" is acceptable
>>>>>>> or not, but I haven't any better idea right now.
>>>>>>
>>>>>> It seems a little fragile to me too. I'd rather the IOMMU requirement be
>>>>>> described more explicitly.
>>>
>>> I don't see how this can work. Typically you find a property and then
>>> read the relevant #*-cells to determine the size. Having multiple cell
>>> properties is asking for errors.
>>
>> The above was mentioned for PATCHv4 series[1], which used the arm,smmu
>> DT bindings,  "#stream-id-cells" in clients and "mmu-masters" in
>> iommu.
>>
>> In PATCHv5[2], we took the following DT binding where multiple cell
>> properties seem to work ok.
>>
>>   smmu_a: iommu at xxxxxxxx {
>>         #iommu-cells = <2>;
>>         ....
>>   };
>>
>>   smmu_b: iommu at xxxxxxxx {
>>         #iommu-cells = <3>;
>>         ....
>>   };
>>
>>   device_a {
>>          iommus = <&smmu_a param1 param2>,
>>                   <&smmu_b param1 param2 param3>;
>>   };
>>
>> This can describe the relation between a device and an iommu
>> independently. The number of params needed for each IOMMU can be
>> sepcified by #iommu-cells in its iommu entry.
>>
>>   device_a <-> smmu_a, needs 2 params for a device
>>   device_a <-> smmu_b, needs 3 params for a device
>>
>> For example, "smmu_a" can be an bus level global IOMMU where all child
>> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
>> only for "device_a".
>>
>> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>>                                       |
>>                                       |
>>                                       +--"device_b"
> 
> I think the above binding would be the correct way to describe things
> if you have 1 device connected to 2 IOMMUs (directly rather than
> chained). IIUC, that is something you have on tegra?

I'm not sure that we do, but perhaps; Hiroshi will have to answer.
Certainly whenever I personally mentioned chained IOMMUs it was simply
in the context of making sure the bindings allowed for any potential
arbitrary HW configuration, not because I had specific knowledge of one
that actually exists.

> For the topology above where you are chaining iommu's, I think
> something like this is more accurately describing the hierarchy:
> 
>   smmu_b: iommu at xxxxxxxx {
>         #iommu-cells = <3>;
>          iommus = <&smmu_a param1 param2>;
>        ....
>   };
>   device_a {
>          iommus = <&smmu_b param1 param2 param3>;
>   };
> 
> I remember discussing this with Will and seem to recall some issue
> with describing things this way. But looking at it now, I don't see
> what that was.

That's the DT content I would expect for that scenario.

The syntax:

>>          iommus = <&smmu_a param1 param2>,
>>                   <&smmu_b param1 param2 param3>;

... I would expect to be useful if a single device was a bus-master on
multiple buses, and each bus had a path to memory via a different IOMMU.
One example might be a DMA controller that bridges two buses. We
certainly have such a DMA controller on Tegra, although IIRC, the
upstream path to memory would pass through the same IOMMU from both
buses on current HW that I'm familiar with.

Re: Report from 2013 ARM kernel summit

[Hiroshi Doyu]

On Wed, 20 Nov 2013 21:22:39 +0100
Stephen Warren <swarren-3lzwWm7+Weoh9ZMKESR00Q@public.gmane.org> wrote:
...
> >> In PATCHv5[2], we took the following DT binding where multiple cell
> >> properties seem to work ok.
> >>
> >>   smmu_a: iommu@xxxxxxxx {
> >>         #iommu-cells = <2>;
> >>         ....
> >>   };
> >>
> >>   smmu_b: iommu@xxxxxxxx {
> >>         #iommu-cells = <3>;
> >>         ....
> >>   };
> >>
> >>   device_a {
> >>          iommus = <&smmu_a param1 param2>,
> >>                   <&smmu_b param1 param2 param3>;
> >>   };
> >>
> >> This can describe the relation between a device and an iommu
> >> independently. The number of params needed for each IOMMU can be
> >> sepcified by #iommu-cells in its iommu entry.
> >>
> >>   device_a <-> smmu_a, needs 2 params for a device
> >>   device_a <-> smmu_b, needs 3 params for a device
> >>
> >> For example, "smmu_a" can be an bus level global IOMMU where all child
> >> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> >> only for "device_a".
> >>
> >> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
> >>                                       |
> >>                                       |
> >>                                       +--"device_b"
> > 
> > I think the above binding would be the correct way to describe things
> > if you have 1 device connected to 2 IOMMUs (directly rather than
> > chained). IIUC, that is something you have on tegra?
> 
> I'm not sure that we do, but perhaps; Hiroshi will have to answer.
> Certainly whenever I personally mentioned chained IOMMUs it was simply
> in the context of making sure the bindings allowed for any potential
> arbitrary HW configuration, not because I had specific knowledge of one
> that actually exists.

In tegra, IOMMUs are chained in a way that:

- "device_a" run in "smmu_b"'s virtual address space.
- "device_a"'s requesting address is transted into "smmu_a"'s virtual
  address via "smmu_b".
- This "smmu_a"'s virtual address is translated into physical address
  via "smmu_a".

"smmu_a" and "smmu_b" are different driver although we haven't
converted "smmu_b" as the standard IOMMU API yet.

> > For the topology above where you are chaining iommu's, I think
> > something like this is more accurately describing the hierarchy:
> > 
> >   smmu_b: iommu@xxxxxxxx {
> >         #iommu-cells = <3>;
> >          iommus = <&smmu_a param1 param2>;
> >        ....
> >   };
> >   device_a {
> >          iommus = <&smmu_b param1 param2 param3>;
> >   };

This describes the dependency between IOMMUs in Tegra. We can populate
"smmu_a" always earlier than "smmu_b". Those dependency/hierarchy can
be stored in IOMMU class(global IOMMU list), which Thierry suggested.

Report from 2013 ARM kernel summit

[Hiroshi Doyu]

On Wed, 20 Nov 2013 21:22:39 +0100
Stephen Warren <swarren@wwwdotorg.org> wrote:
...
> >> In PATCHv5[2], we took the following DT binding where multiple cell
> >> properties seem to work ok.
> >>
> >>   smmu_a: iommu at xxxxxxxx {
> >>         #iommu-cells = <2>;
> >>         ....
> >>   };
> >>
> >>   smmu_b: iommu at xxxxxxxx {
> >>         #iommu-cells = <3>;
> >>         ....
> >>   };
> >>
> >>   device_a {
> >>          iommus = <&smmu_a param1 param2>,
> >>                   <&smmu_b param1 param2 param3>;
> >>   };
> >>
> >> This can describe the relation between a device and an iommu
> >> independently. The number of params needed for each IOMMU can be
> >> sepcified by #iommu-cells in its iommu entry.
> >>
> >>   device_a <-> smmu_a, needs 2 params for a device
> >>   device_a <-> smmu_b, needs 3 params for a device
> >>
> >> For example, "smmu_a" can be an bus level global IOMMU where all child
> >> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> >> only for "device_a".
> >>
> >> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
> >>                                       |
> >>                                       |
> >>                                       +--"device_b"
> > 
> > I think the above binding would be the correct way to describe things
> > if you have 1 device connected to 2 IOMMUs (directly rather than
> > chained). IIUC, that is something you have on tegra?
> 
> I'm not sure that we do, but perhaps; Hiroshi will have to answer.
> Certainly whenever I personally mentioned chained IOMMUs it was simply
> in the context of making sure the bindings allowed for any potential
> arbitrary HW configuration, not because I had specific knowledge of one
> that actually exists.

In tegra, IOMMUs are chained in a way that:

- "device_a" run in "smmu_b"'s virtual address space.
- "device_a"'s requesting address is transted into "smmu_a"'s virtual
  address via "smmu_b".
- This "smmu_a"'s virtual address is translated into physical address
  via "smmu_a".

"smmu_a" and "smmu_b" are different driver although we haven't
converted "smmu_b" as the standard IOMMU API yet.

> > For the topology above where you are chaining iommu's, I think
> > something like this is more accurately describing the hierarchy:
> > 
> >   smmu_b: iommu at xxxxxxxx {
> >         #iommu-cells = <3>;
> >          iommus = <&smmu_a param1 param2>;
> >        ....
> >   };
> >   device_a {
> >          iommus = <&smmu_b param1 param2 param3>;
> >   };

This describes the dependency between IOMMUs in Tegra. We can populate
"smmu_a" always earlier than "smmu_b". Those dependency/hierarchy can
be stored in IOMMU class(global IOMMU list), which Thierry suggested.

Re: Report from 2013 ARM kernel summit

[Will Deacon]

On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
> >   smmu_a: iommu@xxxxxxxx {
> >         #iommu-cells = <2>;
> >         ....
> >   };
> >
> >   smmu_b: iommu@xxxxxxxx {
> >         #iommu-cells = <3>;
> >         ....
> >   };
> >
> >   device_a {
> >          iommus = <&smmu_a param1 param2>,
> >                   <&smmu_b param1 param2 param3>;
> >   };
> >
> > This can describe the relation between a device and an iommu
> > independently. The number of params needed for each IOMMU can be
> > sepcified by #iommu-cells in its iommu entry.
> >
> >   device_a <-> smmu_a, needs 2 params for a device
> >   device_a <-> smmu_b, needs 3 params for a device
> >
> > For example, "smmu_a" can be an bus level global IOMMU where all child
> > devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> > only for "device_a".
> >
> > "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
> >                                       |
> >                                       |
> >                                       +--"device_b"
> 
> I think the above binding would be the correct way to describe things
> if you have 1 device connected to 2 IOMMUs (directly rather than
> chained). IIUC, that is something you have on tegra?
> 
> For the topology above where you are chaining iommu's, I think
> something like this is more accurately describing the hierarchy:
> 
>   smmu_b: iommu@xxxxxxxx {
>         #iommu-cells = <3>;
>          iommus = <&smmu_a param1 param2>;
>        ....
>   };
>   device_a {
>          iommus = <&smmu_b param1 param2 param3>;
>   };
> 
> I remember discussing this with Will and seem to recall some issue
> with describing things this way. But looking at it now, I don't see
> what that was.

I think it was the usual issue with StreamID remastering. For example, if
you had a device going into two chained IOMMUs, you need a way to relate the
StreamIDs of the device to the remastered IDs coming out of the first SMMU.

Will

Report from 2013 ARM kernel summit

[Will Deacon]

On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
> >   smmu_a: iommu at xxxxxxxx {
> >         #iommu-cells = <2>;
> >         ....
> >   };
> >
> >   smmu_b: iommu at xxxxxxxx {
> >         #iommu-cells = <3>;
> >         ....
> >   };
> >
> >   device_a {
> >          iommus = <&smmu_a param1 param2>,
> >                   <&smmu_b param1 param2 param3>;
> >   };
> >
> > This can describe the relation between a device and an iommu
> > independently. The number of params needed for each IOMMU can be
> > sepcified by #iommu-cells in its iommu entry.
> >
> >   device_a <-> smmu_a, needs 2 params for a device
> >   device_a <-> smmu_b, needs 3 params for a device
> >
> > For example, "smmu_a" can be an bus level global IOMMU where all child
> > devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
> > only for "device_a".
> >
> > "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
> >                                       |
> >                                       |
> >                                       +--"device_b"
> 
> I think the above binding would be the correct way to describe things
> if you have 1 device connected to 2 IOMMUs (directly rather than
> chained). IIUC, that is something you have on tegra?
> 
> For the topology above where you are chaining iommu's, I think
> something like this is more accurately describing the hierarchy:
> 
>   smmu_b: iommu at xxxxxxxx {
>         #iommu-cells = <3>;
>          iommus = <&smmu_a param1 param2>;
>        ....
>   };
>   device_a {
>          iommus = <&smmu_b param1 param2 param3>;
>   };
> 
> I remember discussing this with Will and seem to recall some issue
> with describing things this way. But looking at it now, I don't see
> what that was.

I think it was the usual issue with StreamID remastering. For example, if
you had a device going into two chained IOMMUs, you need a way to relate the
StreamIDs of the device to the remastered IDs coming out of the first SMMU.

Will

Re: Report from 2013 ARM kernel summit

[Stephen Warren]

On 11/21/2013 04:00 AM, Will Deacon wrote:
> On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
>> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> wrote:
>>>   smmu_a: iommu@xxxxxxxx {
>>>         #iommu-cells = <2>;
>>>         ....
>>>   };
>>>
>>>   smmu_b: iommu@xxxxxxxx {
>>>         #iommu-cells = <3>;
>>>         ....
>>>   };
>>>
>>>   device_a {
>>>          iommus = <&smmu_a param1 param2>,
>>>                   <&smmu_b param1 param2 param3>;
>>>   };
>>>
>>> This can describe the relation between a device and an iommu
>>> independently. The number of params needed for each IOMMU can be
>>> sepcified by #iommu-cells in its iommu entry.
>>>
>>>   device_a <-> smmu_a, needs 2 params for a device
>>>   device_a <-> smmu_b, needs 3 params for a device
>>>
>>> For example, "smmu_a" can be an bus level global IOMMU where all child
>>> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
>>> only for "device_a".
>>>
>>> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>>>                                       |
>>>                                       |
>>>                                       +--"device_b"
>>
>> I think the above binding would be the correct way to describe things
>> if you have 1 device connected to 2 IOMMUs (directly rather than
>> chained). IIUC, that is something you have on tegra?
>>
>> For the topology above where you are chaining iommu's, I think
>> something like this is more accurately describing the hierarchy:
>>
>>   smmu_b: iommu@xxxxxxxx {
>>         #iommu-cells = <3>;
>>          iommus = <&smmu_a param1 param2>;
>>        ....
>>   };
>>   device_a {
>>          iommus = <&smmu_b param1 param2 param3>;
>>   };
>>
>> I remember discussing this with Will and seem to recall some issue
>> with describing things this way. But looking at it now, I don't see
>> what that was.
> 
> I think it was the usual issue with StreamID remastering. For example, if
             ^^^ s/was/had/ I assume
> you had a device going into two chained IOMMUs, you need a way to relate the
> StreamIDs of the device to the remastered IDs coming out of the first SMMU.

That seems pretty easy to deal with. Am I missing something. Here's how
I assume that would work:

1)

If the mapping is static in HW, then you simply have a table that
describes the remapping. A table representation could be in the DT or
the driver itself. Either way, any static mapping is a property of the
HW itself, so shouldn't need to leak into the client DT nodes at all, right?

That might look like:

smmu_a: iommu@xxxxxxxx {
	#iommu-cells = <1>;
};
smmu_b: iommu@xxxxxxxx {
	#iommu-cells = <1>;

	// a) All stream IDs squashed down to 1 ID
	iommus = <&smmu_a SMMU_B's_STREAM_ID_IN_SMMU_A>;

	// OR b) stream IDs translated:
	iommus = <&smmu_a 0>,
		 <&smmu_a 1>,
		 <&smmu_a 2> ...;
	// this table could be in the driver instead
	iommu-stream-id-map =
		<DEV_A's_STREAM_ID_IN_SMMU_B 0>,
		<DEV_B's_STREAM_ID_IN_SMMU_B 1>,
		<DEV_C's_STREAM_ID_IN_SMMU_B 2>,
};
device_a {
	iommus = <&smmu_b DEV_A's_STREAM_ID_IN_SMMU_B>;
};

or 2)

If the remapping is entirely dynamic, then you'd either need a table in
the DT to describe how to set it up for the particular use-case (similar
to the above), or a function in the driver to manage the mapping at
run-time, probably hooked into the of_xlate() of the parent IOMMU, which
would call into some function of the source IOMMU.

In other words, of_xlate for smmu_a would do both:

a) Parse the SMMU specifier in the client's iommus property

b) If the parse specifier represents a "dynamic stream ID device/IOMMU",
call some function in the client driver to find out which stream ID(s)
the parse iommus entry (specifier) represents.

Report from 2013 ARM kernel summit

[Stephen Warren]

On 11/21/2013 04:00 AM, Will Deacon wrote:
> On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
>> On Wed, Nov 20, 2013 at 12:39 AM, Hiroshi Doyu <hdoyu@nvidia.com> wrote:
>>>   smmu_a: iommu at xxxxxxxx {
>>>         #iommu-cells = <2>;
>>>         ....
>>>   };
>>>
>>>   smmu_b: iommu at xxxxxxxx {
>>>         #iommu-cells = <3>;
>>>         ....
>>>   };
>>>
>>>   device_a {
>>>          iommus = <&smmu_a param1 param2>,
>>>                   <&smmu_b param1 param2 param3>;
>>>   };
>>>
>>> This can describe the relation between a device and an iommu
>>> independently. The number of params needed for each IOMMU can be
>>> sepcified by #iommu-cells in its iommu entry.
>>>
>>>   device_a <-> smmu_a, needs 2 params for a device
>>>   device_a <-> smmu_b, needs 3 params for a device
>>>
>>> For example, "smmu_a" can be an bus level global IOMMU where all child
>>> devices can be an master of "smmu_a", and "smmu_b" is a local IOMMU
>>> only for "device_a".
>>>
>>> "memory controller"---"smmu_a"---bus--+--"smmu_b"--"device_a"
>>>                                       |
>>>                                       |
>>>                                       +--"device_b"
>>
>> I think the above binding would be the correct way to describe things
>> if you have 1 device connected to 2 IOMMUs (directly rather than
>> chained). IIUC, that is something you have on tegra?
>>
>> For the topology above where you are chaining iommu's, I think
>> something like this is more accurately describing the hierarchy:
>>
>>   smmu_b: iommu at xxxxxxxx {
>>         #iommu-cells = <3>;
>>          iommus = <&smmu_a param1 param2>;
>>        ....
>>   };
>>   device_a {
>>          iommus = <&smmu_b param1 param2 param3>;
>>   };
>>
>> I remember discussing this with Will and seem to recall some issue
>> with describing things this way. But looking at it now, I don't see
>> what that was.
> 
> I think it was the usual issue with StreamID remastering. For example, if
             ^^^ s/was/had/ I assume
> you had a device going into two chained IOMMUs, you need a way to relate the
> StreamIDs of the device to the remastered IDs coming out of the first SMMU.

That seems pretty easy to deal with. Am I missing something. Here's how
I assume that would work:

1)

If the mapping is static in HW, then you simply have a table that
describes the remapping. A table representation could be in the DT or
the driver itself. Either way, any static mapping is a property of the
HW itself, so shouldn't need to leak into the client DT nodes at all, right?

That might look like:

smmu_a: iommu at xxxxxxxx {
	#iommu-cells = <1>;
};
smmu_b: iommu at xxxxxxxx {
	#iommu-cells = <1>;

	// a) All stream IDs squashed down to 1 ID
	iommus = <&smmu_a SMMU_B's_STREAM_ID_IN_SMMU_A>;

	// OR b) stream IDs translated:
	iommus = <&smmu_a 0>,
		 <&smmu_a 1>,
		 <&smmu_a 2> ...;
	// this table could be in the driver instead
	iommu-stream-id-map =
		<DEV_A's_STREAM_ID_IN_SMMU_B 0>,
		<DEV_B's_STREAM_ID_IN_SMMU_B 1>,
		<DEV_C's_STREAM_ID_IN_SMMU_B 2>,
};
device_a {
	iommus = <&smmu_b DEV_A's_STREAM_ID_IN_SMMU_B>;
};

or 2)

If the remapping is entirely dynamic, then you'd either need a table in
the DT to describe how to set it up for the particular use-case (similar
to the above), or a function in the driver to manage the mapping at
run-time, probably hooked into the of_xlate() of the parent IOMMU, which
would call into some function of the source IOMMU.

In other words, of_xlate for smmu_a would do both:

a) Parse the SMMU specifier in the client's iommus property

b) If the parse specifier represents a "dynamic stream ID device/IOMMU",
call some function in the client driver to find out which stream ID(s)
the parse iommus entry (specifier) represents.

Re: Report from 2013 ARM kernel summit

[Will Deacon]

Hi Stephen,

On Thu, Nov 21, 2013 at 06:40:28PM +0000, Stephen Warren wrote:
> On 11/21/2013 04:00 AM, Will Deacon wrote:
> > On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
> >> For the topology above where you are chaining iommu's, I think
> >> something like this is more accurately describing the hierarchy:
> >>
> >>   smmu_b: iommu@xxxxxxxx {
> >>         #iommu-cells = <3>;
> >>          iommus = <&smmu_a param1 param2>;
> >>        ....
> >>   };
> >>   device_a {
> >>          iommus = <&smmu_b param1 param2 param3>;
> >>   };
> >>
> >> I remember discussing this with Will and seem to recall some issue
> >> with describing things this way. But looking at it now, I don't see
> >> what that was.
> > 
> > I think it was the usual issue with StreamID remastering. For example, if
>              ^^^ s/was/had/ I assume

Take your pick!

> > you had a device going into two chained IOMMUs, you need a way to relate the
> > StreamIDs of the device to the remastered IDs coming out of the first SMMU.
> 
> That seems pretty easy to deal with. Am I missing something. Here's how
> I assume that would work:
> 
> 1)
> 
> If the mapping is static in HW, then you simply have a table that
> describes the remapping. A table representation could be in the DT or
> the driver itself. Either way, any static mapping is a property of the
> HW itself, so shouldn't need to leak into the client DT nodes at all, right?
> 
> That might look like:
> 
> smmu_a: iommu@xxxxxxxx {
> 	#iommu-cells = <1>;
> };
> smmu_b: iommu@xxxxxxxx {
> 	#iommu-cells = <1>;
> 
> 	// a) All stream IDs squashed down to 1 ID
> 	iommus = <&smmu_a SMMU_B's_STREAM_ID_IN_SMMU_A>;
> 
> 	// OR b) stream IDs translated:
> 	iommus = <&smmu_a 0>,
> 		 <&smmu_a 1>,
> 		 <&smmu_a 2> ...;
> 	// this table could be in the driver instead
> 	iommu-stream-id-map =
> 		<DEV_A's_STREAM_ID_IN_SMMU_B 0>,
> 		<DEV_B's_STREAM_ID_IN_SMMU_B 1>,
> 		<DEV_C's_STREAM_ID_IN_SMMU_B 2>,

Actually, something like this could work. I think we should also support the
case where a range of IDs are `windowed' (i.e. there is a constant offset
applied to incoming IDs to get the outgoing IDs) to avoid huge tables in
some cases.

> };
> device_a {
> 	iommus = <&smmu_b DEV_A's_STREAM_ID_IN_SMMU_B>;
> };
> 
> or 2)
> 
> If the remapping is entirely dynamic, then you'd either need a table in
> the DT to describe how to set it up for the particular use-case (similar
> to the above), or a function in the driver to manage the mapping at
> run-time, probably hooked into the of_xlate() of the parent IOMMU, which
> would call into some function of the source IOMMU.
> 
> In other words, of_xlate for smmu_a would do both:
> 
> a) Parse the SMMU specifier in the client's iommus property
> 
> b) If the parse specifier represents a "dynamic stream ID device/IOMMU",
> call some function in the client driver to find out which stream ID(s)
> the parse iommus entry (specifier) represents.

Sure. We just need to get the binding right so we're prepared when these
platforms turn up. There's also the usual problem of continuing to support
what we already do for existing SMMU bindings.

Will

Report from 2013 ARM kernel summit

[Will Deacon]

Hi Stephen,

On Thu, Nov 21, 2013 at 06:40:28PM +0000, Stephen Warren wrote:
> On 11/21/2013 04:00 AM, Will Deacon wrote:
> > On Wed, Nov 20, 2013 at 08:02:10PM +0000, Rob Herring wrote:
> >> For the topology above where you are chaining iommu's, I think
> >> something like this is more accurately describing the hierarchy:
> >>
> >>   smmu_b: iommu at xxxxxxxx {
> >>         #iommu-cells = <3>;
> >>          iommus = <&smmu_a param1 param2>;
> >>        ....
> >>   };
> >>   device_a {
> >>          iommus = <&smmu_b param1 param2 param3>;
> >>   };
> >>
> >> I remember discussing this with Will and seem to recall some issue
> >> with describing things this way. But looking at it now, I don't see
> >> what that was.
> > 
> > I think it was the usual issue with StreamID remastering. For example, if
>              ^^^ s/was/had/ I assume

Take your pick!

> > you had a device going into two chained IOMMUs, you need a way to relate the
> > StreamIDs of the device to the remastered IDs coming out of the first SMMU.
> 
> That seems pretty easy to deal with. Am I missing something. Here's how
> I assume that would work:
> 
> 1)
> 
> If the mapping is static in HW, then you simply have a table that
> describes the remapping. A table representation could be in the DT or
> the driver itself. Either way, any static mapping is a property of the
> HW itself, so shouldn't need to leak into the client DT nodes at all, right?
> 
> That might look like:
> 
> smmu_a: iommu at xxxxxxxx {
> 	#iommu-cells = <1>;
> };
> smmu_b: iommu at xxxxxxxx {
> 	#iommu-cells = <1>;
> 
> 	// a) All stream IDs squashed down to 1 ID
> 	iommus = <&smmu_a SMMU_B's_STREAM_ID_IN_SMMU_A>;
> 
> 	// OR b) stream IDs translated:
> 	iommus = <&smmu_a 0>,
> 		 <&smmu_a 1>,
> 		 <&smmu_a 2> ...;
> 	// this table could be in the driver instead
> 	iommu-stream-id-map =
> 		<DEV_A's_STREAM_ID_IN_SMMU_B 0>,
> 		<DEV_B's_STREAM_ID_IN_SMMU_B 1>,
> 		<DEV_C's_STREAM_ID_IN_SMMU_B 2>,

Actually, something like this could work. I think we should also support the
case where a range of IDs are `windowed' (i.e. there is a constant offset
applied to incoming IDs to get the outgoing IDs) to avoid huge tables in
some cases.

> };
> device_a {
> 	iommus = <&smmu_b DEV_A's_STREAM_ID_IN_SMMU_B>;
> };
> 
> or 2)
> 
> If the remapping is entirely dynamic, then you'd either need a table in
> the DT to describe how to set it up for the particular use-case (similar
> to the above), or a function in the driver to manage the mapping at
> run-time, probably hooked into the of_xlate() of the parent IOMMU, which
> would call into some function of the source IOMMU.
> 
> In other words, of_xlate for smmu_a would do both:
> 
> a) Parse the SMMU specifier in the client's iommus property
> 
> b) If the parse specifier represents a "dynamic stream ID device/IOMMU",
> call some function in the client driver to find out which stream ID(s)
> the parse iommus entry (specifier) represents.

Sure. We just need to get the binding right so we're prepared when these
platforms turn up. There's also the usual problem of continuing to support
what we already do for existing SMMU bindings.

Will

Report from 2013 ARM kernel summit

[Grant Likely]

Hi everyone,

Here is the report from the summit two weeks ago. I'm posting the
Google docs link for now because the plain text version just looks
plain ugly. I'll clean that up and post it tomorrow.

https://docs.google.com/document/d/1zQomH0H_wzb3tr3sDm1zNTyPWhMPkXalL7vmndF9D9U/edit?usp=sharing

Big thanks to Olof, Kevin and Mark for writing this report, and a huge
shout out to Paul Walmsley for taking incredible notes.

g.