Audio mem2mem devices aka asymmetric sample rate converters

Audio mem2mem devices aka asymmetric sample rate converters

[Sascha Hauer]

Hi All,

I am currently looking into getting the asymmetric sample rate converters
(ASRC) found on some i.MX SoCs to do something useful.

The ASRC units are completely independent units, i.e. independent of the rest
of the audio subsystem. They can read from memory using the SDMA engine,
convert sample rates and/or audio formats and write back to memory also using
the SDMA engine. The ASRC on the i.MX8MN has four contexts to convert up to
four streams simultanously. I am not aware of any other non i.MX SoCs having
such a unit, but I am pretty sure they exist on other SoCs as well.

There are likely two usecases for such a unit. First would be to offload sample
rate and format conversions to hardware. The other would be to synchronize
different audio sources/sinks with different master clocks to each other when
the master clocks drift away.

How would such units be integrated into ASoC? I can think of two ways. First
would be to create an separate audio card from them which records on one end
and plays back with a different sample rate / format on the other end, in the
v4l2 world that would be a classical mem2mem device. Is Alsa/ASoc prepared for
something like this? Would it be feasible to go into such a direction? I
haven't found any examples for this in the tree.

The other way is to attach the ASRC to an existing audio card. That is done
with the existing in-tree sound/soc/fsl/fsl_asrc.c and
sound/soc/fsl/fsl_easrc.c drivers.  This approach feels somehow limited as it's
not possible to just do conversions without playing/recording something. OTOH
userspace is unaffected which might be an advantage. What nags me with that
approach is that it's currently not integrated into the simple-audio-card or
audio-graph-card bindings. Currently the driver can only be used in conjunction
with the fsl,imx-audio-* card driver. It seems backward to integrate such a
generic ASRC unit into a special purpose audio card driver. The ASoC core is
fully unaware of the ASRC with this approach currently which also doesn't look
very appealing. OTOH I don't know if ASoC could handle this. Can ASoC handle
for example a chain of DAIs when there are different sample rates and formats
in that chain?

Currently I don't really know how to proceed. It would be great if you could
share some thoughts to this topic.

Thanks,
 Sascha

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Audio mem2mem devices aka asymmetric sample rate converters

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 3145 bytes --]

On Thu, Jun 02, 2022 at 01:21:06PM +0200, Sascha Hauer wrote:

> How would such units be integrated into ASoC? I can think of two ways. First
> would be to create an separate audio card from them which records on one end
> and plays back with a different sample rate / format on the other end, in the
> v4l2 world that would be a classical mem2mem device. Is Alsa/ASoc prepared for
> something like this? Would it be feasible to go into such a direction? I
> haven't found any examples for this in the tree.

You could certainly do that, though I'd expect userspace wouldn't
know what to do with it without specific configuration.  It also
feels like it's probably not what users really want - generally
the use case is for rewriting an audio stream without going back
to memory, going back to memory means chopping things up into
periods which would tend to introduce additional latency and/or
fragility which is undesirable even if the devices were DMAing
directly to memory.

> The other way is to attach the ASRC to an existing audio card. That is done
> with the existing in-tree sound/soc/fsl/fsl_asrc.c and
> sound/soc/fsl/fsl_easrc.c drivers.  This approach feels somehow limited as it's
> not possible to just do conversions without playing/recording something. OTOH
> userspace is unaffected which might be an advantage. What nags me with that
> approach is that it's currently not integrated into the simple-audio-card or
> audio-graph-card bindings. Currently the driver can only be used in conjunction
> with the fsl,imx-audio-* card driver. It seems backward to integrate such a
> generic ASRC unit into a special purpose audio card driver. The ASoC core is
> fully unaware of the ASRC with this approach currently which also doesn't look
> very appealing. OTOH I don't know if ASoC could handle this. Can ASoC handle
> for example a chain of DAIs when there are different sample rates and formats
> in that chain?

This is essentially the general problem with DPCM not really
scaling at all well, we need to rework the core so that it
understands tracking information about the digital parameters of
signals through the system like it tracks simple analog on/off
information.  At the minute the core doesn't really understand
what's going on with the digital routing within the SoC at all,
it's all done with manual fixups.

If you search for talks from Lars-Peter Clausen at ELC-E you
should find some good overviews of the general direction.  This
is broadly what all the stuff about converting everything to
components is going towards, we're removing the distinction
between CPU and CODEC components so that everything is
interchangable.  The problem is that someone (ideally people with
systems with this sort of hardware!) needs to do a bunch of heavy
lifting in the framework and nobody's had the time to work on the
main part of the problem yet.  Once it's done then things like
the audio-graph-card should be able to handle things easily.

In theory right now you should implement the ASRC as a component
driver.  You can then set it up as a standalone card if you want
to, or integrate into a custom card as you do now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Audio mem2mem devices aka asymmetric sample rate converters

[Sascha Hauer]

On Thu, Jun 02, 2022 at 02:32:23PM +0200, Mark Brown wrote:
> On Thu, Jun 02, 2022 at 01:21:06PM +0200, Sascha Hauer wrote:
> 
> > How would such units be integrated into ASoC? I can think of two ways. First
> > would be to create an separate audio card from them which records on one end
> > and plays back with a different sample rate / format on the other end, in the
> > v4l2 world that would be a classical mem2mem device. Is Alsa/ASoc prepared for
> > something like this? Would it be feasible to go into such a direction? I
> > haven't found any examples for this in the tree.
> 
> You could certainly do that, though I'd expect userspace wouldn't
> know what to do with it without specific configuration.  It also
> feels like it's probably not what users really want - generally
> the use case is for rewriting an audio stream without going back
> to memory, going back to memory means chopping things up into
> periods which would tend to introduce additional latency and/or
> fragility which is undesirable even if the devices were DMAing
> directly to memory.
> 
> > The other way is to attach the ASRC to an existing audio card. That is done
> > with the existing in-tree sound/soc/fsl/fsl_asrc.c and
> > sound/soc/fsl/fsl_easrc.c drivers.  This approach feels somehow limited as it's
> > not possible to just do conversions without playing/recording something. OTOH
> > userspace is unaffected which might be an advantage. What nags me with that
> > approach is that it's currently not integrated into the simple-audio-card or
> > audio-graph-card bindings. Currently the driver can only be used in conjunction
> > with the fsl,imx-audio-* card driver. It seems backward to integrate such a
> > generic ASRC unit into a special purpose audio card driver. The ASoC core is
> > fully unaware of the ASRC with this approach currently which also doesn't look
> > very appealing. OTOH I don't know if ASoC could handle this. Can ASoC handle
> > for example a chain of DAIs when there are different sample rates and formats
> > in that chain?
> 
> This is essentially the general problem with DPCM not really
> scaling at all well, we need to rework the core so that it
> understands tracking information about the digital parameters of
> signals through the system like it tracks simple analog on/off
> information.  At the minute the core doesn't really understand
> what's going on with the digital routing within the SoC at all,
> it's all done with manual fixups.
> 
> If you search for talks from Lars-Peter Clausen at ELC-E you
> should find some good overviews of the general direction.

You likely mean https://www.youtube.com/watch?v=6oQF2TzCYtQ. That indeed
gives a good overview where we are and where we want to go.

> This
> is broadly what all the stuff about converting everything to
> components is going towards, we're removing the distinction
> between CPU and CODEC components so that everything is
> interchangable.  The problem is that someone (ideally people with
> systems with this sort of hardware!) needs to do a bunch of heavy
> lifting in the framework and nobody's had the time to work on the
> main part of the problem yet.  Once it's done then things like
> the audio-graph-card should be able to handle things easily.
> 
> In theory right now you should implement the ASRC as a component
> driver.  You can then set it up as a standalone card if you want
> to, or integrate into a custom card as you do now.

Thanks for your input. I'll see how far I get.

Sascha


-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: Audio mem2mem devices aka asymmetric sample rate converters

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

On Wed, Jun 08, 2022 at 11:28:02AM +0200, Sascha Hauer wrote:
> On Thu, Jun 02, 2022 at 02:32:23PM +0200, Mark Brown wrote:

> > If you search for talks from Lars-Peter Clausen at ELC-E you
> > should find some good overviews of the general direction.

> You likely mean https://www.youtube.com/watch?v=6oQF2TzCYtQ. That indeed
> gives a good overview where we are and where we want to go.

Yup, that's the one.

> > In theory right now you should implement the ASRC as a component
> > driver.  You can then set it up as a standalone card if you want
> > to, or integrate into a custom card as you do now.

> Thanks for your input. I'll see how far I get.

Good luck!  :P

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]