VirtioSound device emulation implementation

VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 551 bytes --]

Hey everyone!

I want to work on implementing the emulation for the VritioSound device. I
contacted the mentor for the project, (Greg), who said it's fine and that I
should declare it on the mailing list in order to find out if someone else
is already working on this project. That is what this mail is for. I tried
searching for "Virtio", "Sound", and "VirtioSound" in the qemu-devel
mailing archives but couldn't find anything. If someone is already working
on it, please let me know, and if not then I'll start working on it right
away.

Shreyansh

[-- Attachment #2: Type: text/html, Size: 676 bytes --]

Re: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> Hey everyone!
>
> I want to work on implementing the emulation for the VritioSound device. I
> contacted the mentor for the project, (Greg), who said it's fine and that I
> should declare it on the mailing list in order to find out if someone else
> is already working on this project. That is what this mail is for. I tried
> searching for "Virtio", "Sound", and "VirtioSound" in the qemu-devel
> mailing archives but couldn't find anything. If someone is already working
> on it, please let me know, and if not then I'll start working on it right
> away.

I'm not working on it but please feel free to CC me on discussion and
patches. I have a general interest on Virtio IO implementations at the
moment.

Are you planning to make it an in-QEMU device or maybe a external
vhost-user daemon?

-- 
Alex Bennée

Re: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]

On Wed, 6 Jan 2021 at 17:12, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
>
> > Hey everyone!
> >
> > I want to work on implementing the emulation for the VritioSound device.
> I
> > contacted the mentor for the project, (Greg), who said it's fine and
> that I
> > should declare it on the mailing list in order to find out if someone
> else
> > is already working on this project. That is what this mail is for. I
> tried
> > searching for "Virtio", "Sound", and "VirtioSound" in the qemu-devel
> > mailing archives but couldn't find anything. If someone is already
> working
> > on it, please let me know, and if not then I'll start working on it right
> > away.
>
> I'm not working on it but please feel free to CC me on discussion and
> patches. I have a general interest on Virtio IO implementations at the
> moment.
>

Sure! (Though I should mention that I do not know a lot about the virtio
architecture yet. I am still reading up about it. So most of my mails might
be (silly) queries. Though I hope to learn about it soon and send in
some patches and tests :)


> Are you planning to make it an in-QEMU device or maybe a external
> vhost-user daemon?
>

The project page states that we need to use the QEMU audio subsystem
for playing and capturing audio samples. I am not entirely sure if this
implies
that the device should be an in-QEMU device or if it could be an external
daemon.
What do you suggest?

cc Gerd Hoffman




>
> --
> Alex Bennée
>

[-- Attachment #2: Type: text/html, Size: 2542 bytes --]

Re: VirtioSound device emulation implementation

[Gerd Hoffmann]

  Hi,

> > Are you planning to make it an in-QEMU device or maybe a external
> > vhost-user daemon?
> 
> The project page states that we need to use the QEMU audio subsystem
> for playing and capturing audio samples. I am not entirely sure if
> this implies that the device should be an in-QEMU device or if it
> could be an external daemon.  What do you suggest?

I think the easiest would be to make it an in-qemu device.  All the
infrastructure qemu has to play sound in various ways (alsa, pulse,
jack, oss, ...) can be used without much effort then.

That is not set in stone though.  It is certainly possible to implement
the device as separate process, using the vhost-user protocol for
communication with qemu.  On one side it is more effort as you have to
handle the audio streams and sound devices without qemu helping with
that.  On the other side vhost-user is also supported by other
hypervisors, so the implementation could be shared then.

HTH,
  Gerd

Re: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> On Wed, 6 Jan 2021 at 17:12, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>>
>> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
>>
>> > Hey everyone!
>> >
>> > I want to work on implementing the emulation for the VritioSound device.
>> I
>> > contacted the mentor for the project, (Greg), who said it's fine and
>> that I
>> > should declare it on the mailing list in order to find out if someone
>> else
>> > is already working on this project. That is what this mail is for. I
>> tried
>> > searching for "Virtio", "Sound", and "VirtioSound" in the qemu-devel
>> > mailing archives but couldn't find anything. If someone is already
>> working
>> > on it, please let me know, and if not then I'll start working on it right
>> > away.
>>
>> I'm not working on it but please feel free to CC me on discussion and
>> patches. I have a general interest on Virtio IO implementations at the
>> moment.
>>
>
> Sure! (Though I should mention that I do not know a lot about the virtio
> architecture yet. I am still reading up about it. So most of my mails might
> be (silly) queries. Though I hope to learn about it soon and send in
> some patches and tests :)
>
>
>> Are you planning to make it an in-QEMU device or maybe a external
>> vhost-user daemon?
>
> The project page states that we need to use the QEMU audio subsystem
> for playing and capturing audio samples.

Is this one of the QEMU internship projects?

> I am not entirely sure if this implies that the device should be an
> in-QEMU device or if it could be an external daemon.

I think it implies having the entire emulation in-QEMU.

> What do you suggest?
>
> cc Gerd Hoffman
>
>
>
>
>>
>> --
>> Alex Bennée
>>


-- 
Alex Bennée

Re: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

On Thu, 7 Jan 2021 at 22:49, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
>
> > On Wed, 6 Jan 2021 at 17:12, Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >>
> >> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
> >>
> >> > Hey everyone!
> >> >
> >> > I want to work on implementing the emulation for the VritioSound
> device.
> >> I
> >> > contacted the mentor for the project, (Greg), who said it's fine and
> >> that I
> >> > should declare it on the mailing list in order to find out if someone
> >> else
> >> > is already working on this project. That is what this mail is for. I
> >> tried
> >> > searching for "Virtio", "Sound", and "VirtioSound" in the qemu-devel
> >> > mailing archives but couldn't find anything. If someone is already
> >> working
> >> > on it, please let me know, and if not then I'll start working on it
> right
> >> > away.
> >>
> >> I'm not working on it but please feel free to CC me on discussion and
> >> patches. I have a general interest on Virtio IO implementations at the
> >> moment.
> >>
> >
> > Sure! (Though I should mention that I do not know a lot about the virtio
> > architecture yet. I am still reading up about it. So most of my mails
> might
> > be (silly) queries. Though I hope to learn about it soon and send in
> > some patches and tests :)
> >
> >
> >> Are you planning to make it an in-QEMU device or maybe a external
> >> vhost-user daemon?
> >
> > The project page states that we need to use the QEMU audio subsystem
> > for playing and capturing audio samples.
>
> Is this one of the QEMU internship projects?
>
Yes, it is one of the GSoC projects.

>
> > I am not entirely sure if this implies that the device should be an
> > in-QEMU device or if it could be an external daemon.
>
> I think it implies having the entire emulation in-QEMU.
>
I see, after reading yours and Gerd's response, I think I will study/think
about
the implementation details and then decide which will be the best route for
me.

>
> > What do you suggest?
> >
> > cc Gerd Hoffman
> >
> >
> >
> >
> >>
> >> --
> >> Alex Bennée
> >>
>
>
> --
> Alex Bennée
>

[-- Attachment #2: Type: text/html, Size: 3517 bytes --]

Re: VirtioSound device emulation implementation

[Gerd Hoffmann]

  Hi,

> >> Are you planning to make it an in-QEMU device or maybe a external
> >> vhost-user daemon?
> >
> > The project page states that we need to use the QEMU audio subsystem
> > for playing and capturing audio samples.
> 
> Is this one of the QEMU internship projects?

one of last years gsoc project ideas

take care,
  Gerd

Re: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2253 bytes --]

Hi,

I have been reading about the virtio and vhost specifications, however I
have a few doubts. I tried looking for them but I still
do not understand them clearly enough. From what I understand, there are
two protocols:

The virtio protocol: The one that specifies how we can have common
emulation for virtual devices. The front end drivers
interact with these devices, and these devices could then process the
information that they have received either in QEMU,
or somewhere else. From what I understand the front driver uses mmaps to
communicate with the virtio device.

The vhost protocol: The one that specifies how we can _offload_ the
processing from QEMU to a separate process. We
want to offload so that we do not have to stop the guest when we are
processing information passed to a virtio device. This
service could either be implemented in the host kernel or the host
userspace. Now when we offload the processing, we map the
memory of the device to this vhost service, so that this service has all
the information that it should process.
  Also, this process can generate the vCPU interrupts, and this process
responds to the ioeventfd notifications.

What I do not understand is, once we have this vhost service, either in
userspace or in kernel space, which does the information processing,
why do we need a virtio device still emulated in QEMU? Is it only to pass
on the configurations between the driver and the
vhost service? I know that the vhost service doesn't emulate anything, but
then what is the difference between "processing" the
information and "emulating" a device?

Also, from article[3], moving the vhost-net service to userspace was faster
somehow. I am assuming this was only the case for
networking devices, and would not be true in general. Since there would be
more context switches between user and kernel space?
(KVM receives the irq/ioevent notification and then transfers control back
to user space, as opposed to when vhost was in kernel
space.)

For context, I've been reading the following:
[1]
https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
[2] https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
[3] https://www.redhat.com/en/blog/journey-vhost-users-realm

[-- Attachment #2: Type: text/html, Size: 2897 bytes --]

Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 3685 bytes --]

---------- Forwarded message ---------
From: Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com>
Date: Mon, 11 Jan 2021 at 11:59
Subject: Re: VirtioSound device emulation implementation
To: Gerd Hoffmann <kraxel@redhat.com>




On Sun, 10 Jan 2021 at 13:55, Shreyansh Chouhan <
chouhan.shreyansh2702@gmail.com> wrote:

> Hi,
>
> I have been reading about the virtio and vhost specifications, however I
> have a few doubts. I tried looking for them but I still
> do not understand them clearly enough. From what I understand, there are
> two protocols:
>
> The virtio protocol: The one that specifies how we can have common
> emulation for virtual devices. The front end drivers
> interact with these devices, and these devices could then process the
> information that they have received either in QEMU,
> or somewhere else. From what I understand the front driver uses mmaps to
> communicate with the virtio device.
>
> The vhost protocol: The one that specifies how we can _offload_ the
> processing from QEMU to a separate process. We
> want to offload so that we do not have to stop the guest when we are
> processing information passed to a virtio device. This
> service could either be implemented in the host kernel or the host
> userspace. Now when we offload the processing, we map the
> memory of the device to this vhost service, so that this service has all
> the information that it should process.
>   Also, this process can generate the vCPU interrupts, and this process
> responds to the ioeventfd notifications.
>
> What I do not understand is, once we have this vhost service, either in
> userspace or in kernel space, which does the information processing,
> why do we need a virtio device still emulated in QEMU? Is it only to pass
> on the configurations between the driver and the
> vhost service? I know that the vhost service doesn't emulate anything, but
> then what is the difference between "processing" the
> information and "emulating" a device?
>
> Also, from article[3], moving the vhost-net service to userspace was
> faster somehow. I am assuming this was only the case for
> networking devices, and would not be true in general. Since there would be
> more context switches between user and kernel space?
> (KVM receives the irq/ioevent notification and then transfers control back
> to user space, as opposed to when vhost was in kernel
> space.)
>
> For context, I've been reading the following:
> [1]
> https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
> [2]
> https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
> [3] https://www.redhat.com/en/blog/journey-vhost-users-realm
>
>
Found the answers in this blog:
http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
In short, yes, the configuration plane still remains with QEMU. The
frontend driver interacts with the PCI
adapter emulated in QEMU, for configurations and memory map setup. Only the
data plane is forwarded
to the vhost service. This makes sense since we would only want to
configure the device once, and hence
having that emulated in QEMU is not a performance issue, as much as having
the data plane was.

There is still a little confusion in my mind regarding a few things, but I
think looking at the source code
of the already implemented drivers will clear that up for me. So that is
what I will be doing next.

I will start looking at the source code for in-QEMU and vhost
implementations of other virtio drivers, and then decide which one I'd like
to
go with. I will probably follow that decision with an implementation
plan/timeline so that everyone can follow the progress on the
development of this project.

[-- Attachment #2: Type: text/html, Size: 5170 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> ---------- Forwarded message ---------
> From: Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com>
> Date: Mon, 11 Jan 2021 at 11:59
> Subject: Re: VirtioSound device emulation implementation
> To: Gerd Hoffmann <kraxel@redhat.com>
>
>
>
>
> On Sun, 10 Jan 2021 at 13:55, Shreyansh Chouhan <
> chouhan.shreyansh2702@gmail.com> wrote:
>
>> Hi,
>>
>> I have been reading about the virtio and vhost specifications, however I
>> have a few doubts. I tried looking for them but I still
>> do not understand them clearly enough. From what I understand, there are
>> two protocols:
>>
>> The virtio protocol: The one that specifies how we can have common
>> emulation for virtual devices. The front end drivers
>> interact with these devices, and these devices could then process the
>> information that they have received either in QEMU,
>> or somewhere else. From what I understand the front driver uses mmaps to
>> communicate with the virtio device.
>>
>> The vhost protocol: The one that specifies how we can _offload_ the
>> processing from QEMU to a separate process. We
>> want to offload so that we do not have to stop the guest when we are
>> processing information passed to a virtio device. This
>> service could either be implemented in the host kernel or the host
>> userspace. Now when we offload the processing, we map the
>> memory of the device to this vhost service, so that this service has all
>> the information that it should process.
>>   Also, this process can generate the vCPU interrupts, and this process
>> responds to the ioeventfd notifications.
>>
>> What I do not understand is, once we have this vhost service, either in
>> userspace or in kernel space, which does the information processing,
>> why do we need a virtio device still emulated in QEMU? Is it only to pass
>> on the configurations between the driver and the
>> vhost service? I know that the vhost service doesn't emulate anything, but
>> then what is the difference between "processing" the
>> information and "emulating" a device?
>>
>> Also, from article[3], moving the vhost-net service to userspace was
>> faster somehow. I am assuming this was only the case for
>> networking devices, and would not be true in general. Since there would be
>> more context switches between user and kernel space?
>> (KVM receives the irq/ioevent notification and then transfers control back
>> to user space, as opposed to when vhost was in kernel
>> space.)
>>
>> For context, I've been reading the following:
>> [1]
>> https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
>> [2]
>> https://www.redhat.com/en/blog/deep-dive-virtio-networking-and-vhost-net
>> [3] https://www.redhat.com/en/blog/journey-vhost-users-realm
>>
>>
> Found the answers in this blog:
> http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
> In short, yes, the configuration plane still remains with QEMU. The
> frontend driver interacts with the PCI
> adapter emulated in QEMU, for configurations and memory map setup. Only the
> data plane is forwarded
> to the vhost service. This makes sense since we would only want to
> configure the device once, and hence
> having that emulated in QEMU is not a performance issue, as much as having
> the data plane was.

Also if you are running a pure TCG emulation QEMU can pass along the
signalled events from the guest to the vhost-user daemon as well.

> There is still a little confusion in my mind regarding a few things, but I
> think looking at the source code
> of the already implemented drivers will clear that up for me. So that is
> what I will be doing next.
>
> I will start looking at the source code for in-QEMU and vhost
> implementations of other virtio drivers, and then decide which one I'd like
> to
> go with. I will probably follow that decision with an implementation
> plan/timeline so that everyone can follow the progress on the
> development of this project.


-- 
Alex Bennée

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

Just an update:

I've studied the virtio specification along with the source code and I now
understand what the device implementation is
going to look like. Also I understand the source code a lot better. I am
now reading about the qemu vhost-user protocol.

Although I haven't read about the vhost-user daemon in detail, from what
little I have read, I would say that the daemon
would get the virtqueues from the virtio device and forward it to the sound
device of the host. (This is the hard part
I think, since an in QEMU device would use code already written for
processing these queues.) I think only the tx and rx
queues would be shared, and although I do not know exactly how the sharing
will be implemented, I think the memory
will be shared to the vhost-user daemon too? So now the virtqueue memory is
shared between the virtio driver in guest
OS, the virtio device in QEMU, and the vhost-user daemon running in the
host userspace.

As for the configuration part, the driver will negotiate features with the
virtio device in QEMU, which in turn will communicate
with the vhost-user daemon (via sockets) to get the features supported I
think.

This is what I think it will roughly look like. (Of course modulo the
implementation details.) I do not yet understand how
much more difficult will implementing the vhost-user daemon be, and since I
was already
warned about the difficulty, I will not risk making any hasty decisions
that later hinder the project. I will read up
about the vhost-user daemon and how it's implemented to get a better idea,
and then make the final call.

Anyways I am super excited about the project. I got to learn about some
really cool things in the past couple of days,
and I can not wait to implement it. :)

[-- Attachment #2: Type: text/html, Size: 2062 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> Just an update:
>
> I've studied the virtio specification along with the source code and I now
> understand what the device implementation is
> going to look like. Also I understand the source code a lot better. I am
> now reading about the qemu vhost-user protocol.
>
> Although I haven't read about the vhost-user daemon in detail, from what
> little I have read, I would say that the daemon
> would get the virtqueues from the virtio device and forward it to the sound
> device of the host. (This is the hard part
> I think, since an in QEMU device would use code already written for
> processing these queues.)

I can't comment on the difficulty there but this does point more towards
using the in-QEMU approach given we have a bunch of utility functions already.

> I think only the tx and rx
> queues would be shared, and although I do not know exactly how the sharing
> will be implemented, I think the memory
> will be shared to the vhost-user daemon too? So now the virtqueue memory is
> shared between the virtio driver in guest
> OS, the virtio device in QEMU, and the vhost-user daemon running in the
> host userspace.

QEMU uses a memfd file descriptor to share the guests entire memory map
with the daemon.

> As for the configuration part, the driver will negotiate features with the
> virtio device in QEMU, which in turn will communicate
> with the vhost-user daemon (via sockets) to get the features supported I
> think.
>
> This is what I think it will roughly look like. (Of course modulo the
> implementation details.) I do not yet understand how
> much more difficult will implementing the vhost-user daemon be, and since I
> was already
> warned about the difficulty, I will not risk making any hasty decisions
> that later hinder the project. I will read up
> about the vhost-user daemon and how it's implemented to get a better idea,
> and then make the final call.

If you want to see an example of a branch new vhost-user daemon being
built up from scratch see my recent virtio-rpmb series. The first few
patches of in-QEMU code will be the same boilerplate either way I think:

  https://patchew.org/QEMU/20200925125147.26943-1-alex.bennee@linaro.org/

> Anyways I am super excited about the project. I got to learn about some
> really cool things in the past couple of days,
> and I can not wait to implement it. :)


-- 
Alex Bennée

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2793 bytes --]

On Thu, 14 Jan 2021 at 23:17, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
>
> > Just an update:
> >
> > I've studied the virtio specification along with the source code and I
> now
> > understand what the device implementation is
> > going to look like. Also I understand the source code a lot better. I am
> > now reading about the qemu vhost-user protocol.
> >
> > Although I haven't read about the vhost-user daemon in detail, from what
> > little I have read, I would say that the daemon
> > would get the virtqueues from the virtio device and forward it to the
> sound
> > device of the host. (This is the hard part
> > I think, since an in QEMU device would use code already written for
> > processing these queues.)
>
> I can't comment on the difficulty there but this does point more towards
> using the in-QEMU approach given we have a bunch of utility functions
> already.
>
> > I think only the tx and rx
> > queues would be shared, and although I do not know exactly how the
> sharing
> > will be implemented, I think the memory
> > will be shared to the vhost-user daemon too? So now the virtqueue memory
> is
> > shared between the virtio driver in guest
> > OS, the virtio device in QEMU, and the vhost-user daemon running in the
> > host userspace.
>
> QEMU uses a memfd file descriptor to share the guests entire memory map
> with the daemon.
>
Oh I see.

>
> > As for the configuration part, the driver will negotiate features with
> the
> > virtio device in QEMU, which in turn will communicate
> > with the vhost-user daemon (via sockets) to get the features supported I
> > think.
> >
> > This is what I think it will roughly look like. (Of course modulo the
> > implementation details.) I do not yet understand how
> > much more difficult will implementing the vhost-user daemon be, and
> since I
> > was already
> > warned about the difficulty, I will not risk making any hasty decisions
> > that later hinder the project. I will read up
> > about the vhost-user daemon and how it's implemented to get a better
> idea,
> > and then make the final call.
>
> If you want to see an example of a branch new vhost-user daemon being
> built up from scratch see my recent virtio-rpmb series. The first few
> patches of in-QEMU code will be the same boilerplate either way I think:
>
>   https://patchew.org/QEMU/20200925125147.26943-1-alex.bennee@linaro.org/

This looks super helpful! Thanks a lot for this.

>
>
> > Anyways I am super excited about the project. I got to learn about some
> > really cool things in the past couple of days,
> > and I can not wait to implement it. :)
>
>
> --
> Alex Bennée
>


- Shreyansh Chouhan

[-- Attachment #2: Type: text/html, Size: 3841 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

>
> If you want to see an example of a branch new vhost-user daemon being
>> built up from scratch see my recent virtio-rpmb series. The first few
>> patches of in-QEMU code will be the same boilerplate either way I think:
>>
>>   https://patchew.org/QEMU/20200925125147.26943-1-alex.bennee@linaro.org/
>
> This really clears everything about vhost-user daemons. Thanks a lot.

I am currently looking at how the sound subsystem is implemented in QEMU.
If I can recreate something similar for the vhost-user daemon,
I'll make it a vhost-user daemon. Else I will implement
it as an in-QEMU device. Either ways, I'll provide an implementation
plan with a loose timeline tomorrow.
I will start writing the code for the device from tomorrow.

[-- Attachment #2: Type: text/html, Size: 1342 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

I will make it an in-QEMU pci device. After it is done and the device is
working, I can
write a vhost-user daemon and move the device out of QEMU should it be
needed.
This way I'd already have the virtio-pci device tested out by the time I
get to the vhost-user
daemon. Also I'll be a lot more familiar to sound and virtio devices by
then hopefully. I hope
this is fine?

As for the implementation plan, I will start out by writing the structure
for the device, `VirtIOSnd`
or `VirtIOSound`. Since QEMU already has a base `VirtIODevice` with common
implementation
along with the `VirtQueue`s, this shouldn't be very difficult. Following
this I will start implementing
the methods mentioned in the `VirtIODeviceClass`. This would include
creating a separate
`virtio-sound-pci.c` file with the `realize`, `unrealize` and various init
functions.

After that, I will start writing the `get`, `set` and validate functions
mentioned in the `VirtioDeviceClass`.
QEMU already has an implementation for `start_ioeventfd` and
`stop_ioeventfd`.
This should get us done with the configuration plane. I am thinking of
writing
a few tests at this point. I will have to read about the qtest framework
first though.
This should take at most around a month. I am hoping to get it done earlier
than that
time (~3 weeks) if things go well. From this most of the time goes to
tests. I am taking
a bit of a margin to err on the side of caution.

This now brings us to the dataplane and processing of the data. I think
once the data plane
is successfully implemented, processing shouldn't take a lot of time. Once
the control
queue, event queue, tx queue and rx queues are implemented, I can start
implementing
the Jack and PCM control requests. I can write tests after completing each
of these
phases. As of right now, I am not awake enough to give
a clear plan on the data plane implementation. I will clear the plan out
tomorrow. The
dataplane along with the processing should at most take a little over a
month to implement I'd
guess.

I will make sure I write a follow up mail tomorrow with a clearer and
complete plan. Apologies for
any statements that weren't very clear or mistakes in the language that I
used.

- Shreyansh Chouhan

[-- Attachment #2: Type: text/html, Size: 2656 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

I wasn't really able to clearly understand how I will use QEMU VIrtqueues
exactly.
So, I decided that I will start implementing the part that I already
understand.
I think once I have something implemented I'll have a better chance at
understanding this.

I am currently writing the boilerplate code for the device, using
`virtio-net` as a reference.

[-- Attachment #2: Type: text/html, Size: 442 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

Just a follow up.
Still working on device initialization. I am trying to understand the
relation between virtio-net and net device in qemu
in hopes of recreating something similar with the audio device. Once I have
the device initialization part
ready I will send in a couple of patches for review.

[-- Attachment #2: Type: text/html, Size: 372 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> Just a follow up.
> Still working on device initialization. I am trying to understand the
> relation between virtio-net and net device in qemu
> in hopes of recreating something similar with the audio device.

All QEMU devices have two parts, a frontend (which the guest sees) and a
backend (which is how the data gets to somewhere in the host). Some of
the command line options in QEMU elide the details for convenience (-nic
and -drive are examples). The -netdev device is all about configuring
the backend of the network part for a paired -device front end. There is
a similar setup for audio (-audiodev) although I'll defer to Gerd's
expertise on how the two interact.

> Once I have
> the device initialization part
> ready I will send in a couple of patches for review.

Good stuff.

-- 
Alex Bennée

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 1499 bytes --]

Thanks a lot Alex!

> All QEMU devices have two parts, a frontend (which the guest sees) and a
> backend (which is how the data gets to somewhere in the host). Some of
> the command line options in QEMU elide the details for convenience (-nic
> and -drive are examples). The -netdev device is all about configuring
> the backend of the network part for a paired -device front end. There is
> a similar setup for audio (-audiodev) although I'll defer to Gerd's
> expertise on how the two interact.

This helped me understand a bit more about how the devices work. In the
source
code, I found the `virtio-net-pci.c` and `virtio-net.c` files, I think the
pci device is what is visible to the guest.
So `virtio-net-pci.c` should be the front end?

For the realize function, I saw that the `virtio_net_device_realize`
function initializes
the `net_conf` for the device. So I am guessing that the
`virtio_snd_device_realize` function
should initialize the number of jacks and streams a device has, along with
the configuration
for all these jacks and streams?

The thing is I do not understand `net` devices all that well so I get a bit
confused with
what is specific to a net device and what should I actually be worried
about :)

The device initalization step mentions that the jack and streams should be
read and
a query should be made for the config of all jacks and streams. What should
be the
default values of these configurations? I think the realize function is
responsible
for setting these up.

[-- Attachment #2: Type: text/html, Size: 1798 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Alex Bennée]

Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:

> Thanks a lot Alex!
>
>> All QEMU devices have two parts, a frontend (which the guest sees) and a
>> backend (which is how the data gets to somewhere in the host). Some of
>> the command line options in QEMU elide the details for convenience (-nic
>> and -drive are examples). The -netdev device is all about configuring
>> the backend of the network part for a paired -device front end. There is
>> a similar setup for audio (-audiodev) although I'll defer to Gerd's
>> expertise on how the two interact.
>
> This helped me understand a bit more about how the devices work. In the
> source
> code, I found the `virtio-net-pci.c` and `virtio-net.c` files, I think the
> pci device is what is visible to the guest.
> So `virtio-net-pci.c` should be the front end?

No - they are both front-ends. In VirtIO you have two transports. The
first is virtio-mmio which is a simple set of memory mapped IO
registers. There is no discovery (well there is some but the guest needs
to find where in memory the mmio bus is via some method like hardcoded
address or DTB).

The second is virtio-pci. Here the virtio devices are encapsulated in a
PCI bus so you get the benefits of discovery and bus enumeration.

> For the realize function, I saw that the `virtio_net_device_realize`
> function initializes
> the `net_conf` for the device. So I am guessing that the
> `virtio_snd_device_realize` function
> should initialize the number of jacks and streams a device has, along with
> the configuration
> for all these jacks and streams?

No those are all front-end configuration knobs. 

> The thing is I do not understand `net` devices all that well so I get a bit
> confused with
> what is specific to a net device and what should I actually be worried
> about :)
>
> The device initalization step mentions that the jack and streams should be
> read and
> a query should be made for the config of all jacks and streams. What should
> be the
> default values of these configurations? I think the realize function is
> responsible
> for setting these up.

Gerd?



-- 
Alex Bennée

Re: Fwd: VirtioSound device emulation implementation

[Gerd Hoffmann]

On Thu, Jan 28, 2021 at 09:58:23AM +0530, Shreyansh Chouhan wrote:
> Thanks a lot Alex!
> 
> > All QEMU devices have two parts, a frontend (which the guest sees) and a
> > backend (which is how the data gets to somewhere in the host). Some of
> > the command line options in QEMU elide the details for convenience (-nic
> > and -drive are examples). The -netdev device is all about configuring
> > the backend of the network part for a paired -device front end. There is
> > a similar setup for audio (-audiodev) although I'll defer to Gerd's
> > expertise on how the two interact.
> 
> This helped me understand a bit more about how the devices work. In the
> source
> code, I found the `virtio-net-pci.c` and `virtio-net.c` files, I think the
> pci device is what is visible to the guest.
> So `virtio-net-pci.c` should be the front end?

Well, virtio can have different transports, thats why the separation.
pci is the most common one, but there are also mmio (supported by x86
microvm and arm/aarch64 virt) and ccw (used on s390x).

virtio-net.c is the actual virtio device, virtio-net-pci.c has mostly
glue code to make the virtio device visible on the pci bus.

> For the realize function, I saw that the `virtio_net_device_realize`
> function initializes
> the `net_conf` for the device. So I am guessing that the
> `virtio_snd_device_realize` function
> should initialize the number of jacks and streams a device has, along with
> the configuration
> for all these jacks and streams?

Yes.  net_conf is common config (backend, mac address, maybe more) for
network devices.  For sound devices that would audiodev (link the device
to a backend which then handles alsa/pulse/jack/oss/sdl/whatever).

Configuration can be done either using device properties, or by having
different devices.  Using device properties is probably the easier way
to go.  An example for the latter are the qemu hda codecs.  We have
three different devices:

 (1) hda-output: playback-only, offering a virtual line-out.
 (2) hda-duplex: record/playback, offering line-in and line-out.
 (3) hda-micro:  record/playback, offering microphone and speaker.

The difference between (2) and (3) is just the jack labeling, otherwise
the behavior is 100% identical (and they share the emulation code of
course).  But some windows VoIP apps didn't like the line-in and
complained about a missing microphone, so we added that ...

> The thing is I do not understand `net` devices all that well so I get a bit
> confused with
> what is specific to a net device and what should I actually be worried
> about :)

The only thing really required is the audiodev property.  Everything
else can be hard-coded initially, and once the basics are working
refined (like adding properties for jack labels, ...).

> The device initalization step mentions that the jack and streams
> should be read and a query should be made for the config of all jacks
> and streams. What should be the default values of these
> configurations? I think the realize function is responsible for
> setting these up.

I'd start with offering a single 48000, 16bit, stereo, line-out pcm
playback channel.

HTH,
  Gerd

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]

On Thu, 28 Jan 2021 at 16:54, Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Shreyansh Chouhan <chouhan.shreyansh2702@gmail.com> writes:
>
> > Thanks a lot Alex!
> >
> >> All QEMU devices have two parts, a frontend (which the guest sees) and a
> >> backend (which is how the data gets to somewhere in the host). Some of
> >> the command line options in QEMU elide the details for convenience (-nic
> >> and -drive are examples). The -netdev device is all about configuring
> >> the backend of the network part for a paired -device front end. There is
> >> a similar setup for audio (-audiodev) although I'll defer to Gerd's
> >> expertise on how the two interact.
> >
> > This helped me understand a bit more about how the devices work. In the
> > source
> > code, I found the `virtio-net-pci.c` and `virtio-net.c` files, I think
> the
> > pci device is what is visible to the guest.
> > So `virtio-net-pci.c` should be the front end?
>
> No - they are both front-ends. In VirtIO you have two transports. The
> first is virtio-mmio which is a simple set of memory mapped IO
> registers. There is no discovery (well there is some but the guest needs
> to find where in memory the mmio bus is via some method like hardcoded
> address or DTB).
>
> The second is virtio-pci. Here the virtio devices are encapsulated in a
> PCI bus so you get the benefits of discovery and bus enumeration.
>
So I am a bit confused here. First of there are two parts to a virtio
device in general. One is the
front end that the guest sees, and one is where the processing happens,
which could be either in QEMU
or as a separate daemon. In case of an in QEMU virtio device, the QEMU
front end is replaced by the
virtio-front end. Is that true? That's what I think should happen if both
of those files are front ends.

When you said earlier that all QEMU devices have two parts, I somehow
thought that
the in-QEMU backend for the audio device will itself have two parts.

>
> > For the realize function, I saw that the `virtio_net_device_realize`
> > function initializes
> > the `net_conf` for the device. So I am guessing that the
> > `virtio_snd_device_realize` function
> > should initialize the number of jacks and streams a device has, along
> with
> > the configuration
> > for all these jacks and streams?
>
> No those are all front-end configuration knobs.
>
I am assuming these are set from the command-line or the configuration
files.

>
> > The thing is I do not understand `net` devices all that well so I get a
> bit
> > confused with
> > what is specific to a net device and what should I actually be worried
> > about :)
> >
> > The device initalization step mentions that the jack and streams should
> be
> > read and
> > a query should be made for the config of all jacks and streams. What
> should
> > be the
> > default values of these configurations? I think the realize function is
> > responsible
> > for setting these up.
>
> Gerd?
>
>
>
> --
> Alex Bennée
>
--
Shreyansh

[-- Attachment #2: Type: text/html, Size: 4049 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2318 bytes --]

(Gerd, I wasn't able to get gmail to quote your email, so I have just copy
pasted the relavant parts.)

> Yes.  net_conf is common config (backend, mac address, maybe more) for
> network devices.  For sound devices that would audiodev (link the device
> to a backend which then handles alsa/pulse/jack/oss/sdl/whatever).

Ah I see, so the net_conf corresponds to audiodev? I thought it would
correspond
to `virtio_snd_conf`. So as alex said, `virtio_snd_conf` is the front end
configuration.

> Configuration can be done either using device properties, or by having
> different devices.  Using device properties is probably the easier way
> to go.  An example for the latter are the qemu hda codecs.  We have
> three different devices:

>  (1) hda-output: playback-only, offering a virtual line-out.
>  (2) hda-duplex: record/playback, offering line-in and line-out.
>  (3) hda-micro:  record/playback, offering microphone and speaker.

> The difference between (2) and (3) is just the jack labeling, otherwise
> the behavior is 100% identical (and they share the emulation code of
> course).  But some windows VoIP apps didn't like the line-in and
> complained about a missing microphone, so we added that ...

>> The thing is I do not understand `net` devices all that well so I get a
bit
>> confused with
>> what is specific to a net device and what should I actually be worried
>> about :)

> The only thing really required is the audiodev property.  Everything
> else can be hard-coded initially, and once the basics are working
> refined (like adding properties for jack labels, ...).

So in the realize function I set up the audiodev, right? Also in that case
why the difference between the net and sound devices? In the net
device we set up the common config in net_conf. Does the net_conf
somehow later sets up NetDev too?

And what is a NetClientState? Is the NetClient the emulated guest? This
confuses me a lot. I can't understand what will be the parellel audio device
property.

Also I can't seem to find where we parse the command line options passed to
qemu.
The code structure is a bit different from what I had expected. In
virtio_net_device_realize
we set duplex to half or full depending on the value of the
net_conf.duplex_str. But I
couldn't find where we actually set it.

cc Alex

--
Shreyansh

[-- Attachment #2: Type: text/html, Size: 3962 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Gerd Hoffmann]

On Thu, Jan 28, 2021 at 09:20:31PM +0530, Shreyansh Chouhan wrote:
> (Gerd, I wasn't able to get gmail to quote your email, so I have just copy
> pasted the relavant parts.)
> 
> > Yes.  net_conf is common config (backend, mac address, maybe more) for
> > network devices.  For sound devices that would audiodev (link the device
> > to a backend which then handles alsa/pulse/jack/oss/sdl/whatever).
> 
> Ah I see, so the net_conf corresponds to audiodev?

Oops.  Confused that.  nic_conf (struct NICConf) is the common config
for all network devices, not net_conf.

See DEFINE_NIC_PROPERTIES() in include/net/net.h

NICConf.peers (exposed as "netdev" property) links the emulated device
(frontend) to a netdev (backend) which actually processes the packets
sent by the guest.

In a simliar way the audiodev property links the emulated audio device
to a backend which handles the host-side audio playback using alsa,
pulseaudio or something else.

> I thought it would correspond to `virtio_snd_conf`. So as alex said,
> `virtio_snd_conf` is the front end configuration.

Correct.

The backend is configured separately, i.e.

  -netdev user,id=net0,$moreargs

or

  -audiodev alsa,id=snd0,$moreargs

Then the two are linked by id, i.e.

  -device virtio-net-pci,netdev=net0

or

  -device virtio-sound-pci,audiodev=snd0

> > The only thing really required is the audiodev property.  Everything
> > else can be hard-coded initially, and once the basics are working
> > refined (like adding properties for jack labels, ...).
> 
> So in the realize function I set up the audiodev, right? Also in that case
> why the difference between the net and sound devices? In the net
> device we set up the common config in net_conf. Does the net_conf
> somehow later sets up NetDev too?
> 
> And what is a NetClientState? Is the NetClient the emulated guest? This
> confuses me a lot. I can't understand what will be the parellel audio device
> property.

Not fully sure what NetClientState is, I guess it is shared struct for
both frontend and backend to manage the connection state.

The audio subsystem has simliar structs, SWVoiceIn and SWVoiceOut for
example.  There also is QEMUSoundCard.  I'd suggest to check out the
source code of other audio devices for code examples.

> Also I can't seem to find where we parse the command line options
> passed to qemu.  The code structure is a bit different from what I had
> expected. In virtio_net_device_realize we set duplex to half or full
> depending on the value of the net_conf.duplex_str. But I couldn't find
> where we actually set it.

See virtio_net_properties[].  There is a line in the array:

    DEFINE_PROP_STRING("duplex", VirtIONet, net_conf.duplex_str),

And the whole array is registered using:

   device_class_set_props(dc, virtio_net_properties);

That is enough to make those properties work, the qemu core handles
everything for you.  See hw/core/qdev-properties.c if you are curious,
but you can also just consider that a black box at service for you ;)

take care,
  Gerd

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 3758 bytes --]

On Thu, 28 Jan 2021 at 22:00, Gerd Hoffmann <kraxel@redhat.com> wrote:

> On Thu, Jan 28, 2021 at 09:20:31PM +0530, Shreyansh Chouhan wrote:
> > (Gerd, I wasn't able to get gmail to quote your email, so I have just
> copy
> > pasted the relavant parts.)
> >
> > > Yes.  net_conf is common config (backend, mac address, maybe more) for
> > > network devices.  For sound devices that would audiodev (link the
> device
> > > to a backend which then handles alsa/pulse/jack/oss/sdl/whatever).
> >
> > Ah I see, so the net_conf corresponds to audiodev?
>
> Oops.  Confused that.  nic_conf (struct NICConf) is the common config
> for all network devices, not net_conf.
>
> See DEFINE_NIC_PROPERTIES() in include/net/net.h
>
> NICConf.peers (exposed as "netdev" property) links the emulated device
> (frontend) to a netdev (backend) which actually processes the packets
> sent by the guest.
>
> In a simliar way the audiodev property links the emulated audio device
> to a backend which handles the host-side audio playback using alsa,
> pulseaudio or something else.
>
> > I thought it would correspond to `virtio_snd_conf`. So as alex said,
> > `virtio_snd_conf` is the front end configuration.
>
> Correct.
>
> The backend is configured separately, i.e.
>
>   -netdev user,id=net0,$moreargs
>
> or
>
>   -audiodev alsa,id=snd0,$moreargs
>
> Then the two are linked by id, i.e.
>
>   -device virtio-net-pci,netdev=net0
>
> or
>
>   -device virtio-sound-pci,audiodev=snd0
>

Ah ha! So `virtio-snd-conf` corresponds to the `-device` configuration
and `audiodev` to the backend configuration. I think the audio code
now makes more sense to me. I will give it another read.

> > The only thing really required is the audiodev property.  Everything
> > > else can be hard-coded initially, and once the basics are working
> > > refined (like adding properties for jack labels, ...).
> >
> > So in the realize function I set up the audiodev, right? Also in that
> case
> > why the difference between the net and sound devices? In the net
> > device we set up the common config in net_conf. Does the net_conf
> > somehow later sets up NetDev too?
> >
> > And what is a NetClientState? Is the NetClient the emulated guest? This
> > confuses me a lot. I can't understand what will be the parellel audio
> device
> > property.
>
> Not fully sure what NetClientState is, I guess it is shared struct for
> both frontend and backend to manage the connection state.
>
> The audio subsystem has simliar structs, SWVoiceIn and SWVoiceOut for
> example.  There also is QEMUSoundCard.  I'd suggest to check out the
> source code of other audio devices for code examples.
>
I will read it and revert back if I have any queries.

>
> > Also I can't seem to find where we parse the command line options
> > passed to qemu.  The code structure is a bit different from what I had
> > expected. In virtio_net_device_realize we set duplex to half or full
> > depending on the value of the net_conf.duplex_str. But I couldn't find
> > where we actually set it.
>
> See virtio_net_properties[].  There is a line in the array:
>
>     DEFINE_PROP_STRING("duplex", VirtIONet, net_conf.duplex_str),
>
I thought this just declared a property, and didn't set it. But now that in
retrospect
we already declared the variable when we defined the struct so that doesn't
make
sense.

>
> And the whole array is registered using:
>
>    device_class_set_props(dc, virtio_net_properties);
>
> That is enough to make those properties work, the qemu core handles
> everything for you.  See hw/core/qdev-properties.c if you are curious,
> but you can also just consider that a black box at service for you ;)
>
I think I will give it a quick look :P

>
> take care,
>   Gerd
>
> Thanks a lot!
--
Shreyansh

[-- Attachment #2: Type: text/html, Size: 5193 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 3808 bytes --]

On Thu, 28 Jan 2021 at 23:04, Shreyansh Chouhan <
chouhan.shreyansh2702@gmail.com> wrote:

> I think I will give it a quick look :P
>
This certainly wasn't quick I admit.

> Thanks a lot!
> --
> Shreyansh
>
> Hey! I hope you people are doing fine. :)
So colleges reopened and I was a bit busy for the past 2 weeks. Although I
did get to work on this
on weekends.

So about the audiodev setup, I checked the source code on how it is setup,
and I found out I
did not really have to do a lot for that. QEMU already registers the
audiodev passed on command
line and later makes it to an audiostate, and adds it to the list of
audiostates. And when the device
option is parsed, all its properties are set via the visitors iirc. So all
I had to do was make a new device
type, define a QEMUSoundCard property and register it, and this should be
enough to set the audio state for the QEMUSoundCard in this
new device. Do correct me if I am wrong here. (audiostates in turn contain
the audiodev)

I have some code written, but it is a bit incomplete and I would like to
complete at least the initialization
part before sending in patches for review. I wanted to ask some questions
(again :P) regarding a few things.

I read the source code for the `gus` sound device. (gus.c) And set up the
audiosettings and SWVoiceOut
from there. Here is my first question, I feel as if SWVoiceOut should be
available per stream. So the
`VirtIOSound` device would have a list of `SWVoiceOut`?

Secondly I saw in the `ac97.c` source, (which is a PCI sound device,) a lot
of PCIDev related set up in
the realize function, but they were not present in the `virtio-net.c`
source. Do I need them? (`ac97.c` set
up PCI_COMMAND, PCI_STATUS, PCI_BASE_ADDRESS and similar things in PCIDev.
For now the pci
setup in `virtio-snd-pci.c` basically mimics `virtio-net-pci.c` which uses
a `VirtIOPCIProxy` obj.)

Thirdly, the properties are registered at two different places, once in the
`virtio-net.c` source and once
in the `virtio-net-pci.c` source. I understand the the ones in
`virito-net.c`/`virtio-snd.c` they are the device
properties, as in the configurations we can set for the device and other,
well, properties. But what
exactly are the properties defined in the `virtio-net-pci.c` source file? I
have a vague idea of
what they are, but I can't exactly put my finger on it. It's almost as if
`virtio-net` and `virito-net-pci`
are two different devices each with thier own properties, and the
virtio-net-pci helps communication
between virtio-net and QEMU guest. It registers using the same `type_init`
macro, and that macro
registers modules which can later be initialized by QEMU if I am not wrong.
Since I didn't get a lot
of time I was not able to really dig into the PCI code, and taking a look
at it might make things clearer,
but once I started writing this mail, I thought I could ask this too.

Since there can be more than on streams and more than one jacks, we need to
have a list of configurations
for them, and since they should be index adressable, should I use an array
for them? When I was reading the
source I did not come across a QEMU list structure with indexed adressing.
If there is one please let me know.

Finally, I do understand what the pcm streams are, and I have been able to
set them up with hardcoded
initial configs (modulo the hda part). But I do not understand what exactly
are `jacks` and what should I do to set them up.
Which source file should I look at for this? I came across a few jack
related structs, but didn't see
a device using them. (I did not search for it as vigorously, I only tried
grepping in `hw/audio` and the
only results were from `intel-hda-defs`. They were comments on enums. Again
since I was writing
this mail I thought I should ask it here too.)

Regards,
Shreyansh Chouhan

[-- Attachment #2: Type: text/html, Size: 4990 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Gerd Hoffmann]

  Hi,

> I read the source code for the `gus` sound device. (gus.c) And set up the
> audiosettings and SWVoiceOut
> from there. Here is my first question, I feel as if SWVoiceOut should be
> available per stream.

Correct.

> So the `VirtIOSound` device would have a list of `SWVoiceOut`?

In the simplest case just one, most devices have two (one 'out' for
playback, one 'in' for recording).  Can be more.

> Secondly I saw in the `ac97.c` source, (which is a PCI sound device,) a lot
> of PCIDev related set up in
> the realize function, but they were not present in the `virtio-net.c`
> source. Do I need them? (`ac97.c` set
> up PCI_COMMAND, PCI_STATUS, PCI_BASE_ADDRESS and similar things in PCIDev.
> For now the pci
> setup in `virtio-snd-pci.c` basically mimics `virtio-net-pci.c` which uses
> a `VirtIOPCIProxy` obj.)

In case of a virtio-pci device VirtIOPCIProxy will deal with all the pci
details, no need for you to set this up.

> Thirdly, the properties are registered at two different places, once in the
> `virtio-net.c` source and once
> in the `virtio-net-pci.c` source. I understand the the ones in
> `virito-net.c`/`virtio-snd.c` they are the device
> properties, as in the configurations we can set for the device and other,
> well, properties. But what
> exactly are the properties defined in the `virtio-net-pci.c` source file? I
> have a vague idea of
> what they are, but I can't exactly put my finger on it. It's almost as if
> `virtio-net` and `virito-net-pci`
> are two different devices each with thier own properties, and the
> virtio-net-pci helps communication
> between virtio-net and QEMU guest.

virtio-net-pci.c is a small wrapper which combines virtio-net.c and
VirtIOPCIProxy.  Most properties virtio-net-pci.c has are just aliases
and come from either VirtIOPCIProxy or virtio-net.c

> Since there can be more than on streams and more than one jacks, we
> need to have a list of configurations for them, and since they should
> be index adressable, should I use an array for them? When I was
> reading the source I did not come across a QEMU list structure with
> indexed adressing.  If there is one please let me know.

I'd suggest to simply use an array.

> Finally, I do understand what the pcm streams are, and I have been able to
> set them up with hardcoded
> initial configs (modulo the hda part). But I do not understand what exactly
> are `jacks` and what should I do to set them up.
> Which source file should I look at for this? I came across a few jack
> related structs, but didn't see
> a device using them. (I did not search for it as vigorously, I only tried
> grepping in `hw/audio` and the
> only results were from `intel-hda-defs`. They were comments on enums. Again
> since I was writing
> this mail I thought I should ask it here too.)

Jacks are modeled after hda specification.  See also hda-codec-common.h
They basically describe the inputs and outputs, along the lines of "this
output is a green line-out jack / blue line-out jack / speaker / ...".

In case of qemu it is all virtual anyway and doesn't matter that much.
The guest might show this in the audio config tools though, so it
should be something commonly used.  hda-duplex for example uses "red
line-in" for recording and "green line-out" for playback because those
are the colors typically used for these jacks on your laptop.

HTH,
  Gerd

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 4778 bytes --]

Hey everyone!
I wrote code for PCM streams, which I'd like to discuss here. But since it
isn't
really complete yet, I thought I'd ask if I can copy paste the functions
that I want to ask
about and clear things that way.

For now I will just write my queries here and try to explain them as best
as I can.

Starting off with the get config function for pcm streams. Initially I
thought I
was supposed to store those configs in the VirtIOSound struct but then I
realized
these configurations should be queried from the backend and then be
presented to the driver/guest.

Now, the virtio sound card supports multiple formats and bit rates. If we
have fixed settings
turned on in the audiodev, the virtio sound card should support only a
single freq and frame rate
depending upon what was passed to the command line which we can get from
audio_get_pdo_out.
Is this correct?

Secondly if fixed settings was not set, what should the get config query
return with for supported
formats and bitrates? For now I am returning the formats defined in the
enum for the qemu
audio subsystem. I read in the man pages that if the mixing engine is off,
qemu assumes that
the backend supports the the multiple streams and channels that are
supported by the
virtual card. So should I return everything that the virito sound card can
support?

Thirdly, for the set params function, how do I change the params for an
SWVoiceOut stream? I could
not find a function for changing the audsettings for a stream. Should I
close the stream and
reopen it? Should I directly change the values in the pcm info struct
inside the stream?

I learned that the callback passed in AUD_open_out, (lets call it the write
audio callback,)  is supposed to mix and write the
buffers to HWVoiceOut. I have written that, the basic algorithm being:

1. Pop element from tx virtqueue.
2. Get the xfer header from the elem->out_sg (iov_to_buf(elem->out_sg, 1,
0, &hdr, sizeof(hdr)))
3. Get the buffer from elem->out_sg (iov_to_buf(elem->out_sg, 1,
sizeof(hdr), &mixbuf, period_bytes))
4. AUD_write the buffer
5. Initialize an VIRTIO_SND_S_OK response.
6. Write the response to elem->in_sg (iov_from_buf(elem->in_sg,
elem->in_num, 0, &resp, sizeof(resp)))
7. If tx queue is not empty go back to step 1.

I think I can send a period elapsed notification too after reading
period_bytes from the tx_virtqueue.

Will this be enough? From other sound card implementations I found out
about a lot of pointers which
I think were read and write pointers for the buffer. But since we are doing
this via a virtqueue, I don't
feel as if those pointers will be necessary.

Also I do not understand what the tx virtqueue handler is supposed to do. I
have written a handler
for the control queue. But I don't know what to do about the tx queue for
now. I thought it would be
something similar to what the callback does, it wouldn't play the audio
though.

Also since the callback does so many things, I do not understand how I can
implement the
pcm stream prepare, start, stop and release functions. The prepare function
is supposed to
allocate resources for the stream, but we already do that in the realize_fn
for the device
(AUD_open_out). So should I move that part out of the realize function and
into the prepare
stream function? I can then have the write audio callback called in the
start stream function.
The release function would just g_free the stream. This version would have
set_params store
the params of the streams in the device itself, or in a new device state
structure. Then when
prepare is called for a stream, we use the stored audsettings for
AUD_open_out. The start
function would simply call the virtio_snd_callback and write the audio
using AUD_write. I still
do not know what I would have to write in the stop stream function.

Another thing that I wanted to ask was about the hda codec. The
specification mentions that
the virtio sound card has a single codec device in it. I saw a lot of codec
device related code
in hda-codec.c which I think can be re-used for this. But there were no
headers that exposed
the code elsewhere. After reading through the hda specification I realized
that these function
group nids all come under the codec, so the jacks will be pin widgets
attached to this codec.
And the streams will be the streams associated with this codec. But I do
not understand how
I should go about implementing the codec, or if I need to implement it
considering the
already existing source from intel-hda and hda-codec.c.

Also sorry for the late response, I had fallen ill. Also I had to move
thrice in the past
month, so I couldn't really work on this a lot, and I didn't want to write
a mail without
having any work to show to you guys. Thanks a lot for being patient with
me. :)

Regards,
Shreyansh

[-- Attachment #2: Type: text/html, Size: 5722 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Gerd Hoffmann]

  Hi,

> Starting off with the get config function for pcm streams. Initially I
> thought I was supposed to store those configs in the VirtIOSound
> struct but then I realized these configurations should be queried from
> the backend and then be presented to the driver/guest.

No.  The device can completely ignore the backend capabilities.

We have mixing-engine and fixed-settings.

With mixing-engine=on and fixed-settings=on (default) qemu will mix all
streams, resample if needed, and pass a single stream to the backend.

mixing-engine=off will turn disable the qemu mixer, the streams are
passed as-is to the backend then.  This requires fixed-settings=off too
(see documentation comments in qapi/audio.json).  The backend must
handle the settings the device asks for, either by configuring the audio
hardware accordingly (oss/alsa), or by passing on the settings to the
sound daemon (pulseaudio).

> Now, the virtio sound card supports multiple formats and bit rates. If
> we have fixed settings turned on in the audiodev, the virtio sound
> card should support only a single freq and frame rate depending upon
> what was passed to the command line which we can get from
> audio_get_pdo_out.  Is this correct?

No, see above, qemu will resample if needed.

> Secondly if fixed settings was not set, what should the get config
> query return with for supported formats and bitrates? For now I am
> returning the formats defined in the enum for the qemu audio
> subsystem.

The device can support every format and sample rate which is supported
by both qemu and the virtio spec.  Whenever it actually makes sense to
support outdated formats like S8 is questionable.  On the other hand not
supporting it will not simplify the code much.  Your choice.

> Thirdly, for the set params function, how do I change the params for
> an SWVoiceOut stream?  I could not find a function for changing the
> audsettings for a stream. Should I close the stream and reopen it?

Call AUD_open_out() again with the new settings.

> I learned that the callback passed in AUD_open_out, (lets call it the write
> audio callback,)  is supposed to mix and write the
> buffers to HWVoiceOut. I have written that, the basic algorithm being:
> 
> 1. Pop element from tx virtqueue.
> 2. Get the xfer header from the elem->out_sg (iov_to_buf(elem->out_sg, 1,
> 0, &hdr, sizeof(hdr)))
> 3. Get the buffer from elem->out_sg (iov_to_buf(elem->out_sg, 1,
> sizeof(hdr), &mixbuf, period_bytes))
> 4. AUD_write the buffer

AUD_write returns the number of bytes actually accepted.

In case the audio backend consumed the complete buffer you can go ahead
as described.  Otherwise stop here and resume (try AUD_write() the
remaining data) when the callback is called again.

> Also I do not understand what the tx virtqueue handler is supposed to
> do. I have written a handler for the control queue. But I don't know
> what to do about the tx queue for now. I thought it would be something
> similar to what the callback does, it wouldn't play the audio though.

The tx handler probably doesn't need to do much if anything in case you
do the virtqueue processing in the audio callback as described above.

> Also since the callback does so many things, I do not understand how I
> can implement the pcm stream prepare, start, stop and release
> functions. The prepare function is supposed to allocate resources for
> the stream, but we already do that in the realize_fn for the device
> (AUD_open_out). So should I move that part out of the realize function
> and into the prepare stream function?

start/stop maps to AUD_set_active_out(card, true/false);
You can probably just ignore prepare + release.

> Another thing that I wanted to ask was about the hda codec. The
> specification mentions that the virtio sound card has a single codec
> device in it. I saw a lot of codec device related code in hda-codec.c
> which I think can be re-used for this. But there were no headers that
> exposed the code elsewhere. After reading through the hda
> specification I realized that these function group nids all come under
> the codec, so the jacks will be pin widgets attached to this codec.
> And the streams will be the streams associated with this codec. But I
> do not understand how I should go about implementing the codec, or if
> I need to implement it considering the already existing source from
> intel-hda and hda-codec.c.

I don't think you can reuse much code, if any.

The AC_* #defines in intel-hda-defs.h should be useful though (jack
colors etc).  Move them to a separate header file is probably a good
idea.

> Also sorry for the late response, I had fallen ill. Also I had to move
> thrice in the past month, so I couldn't really work on this a lot, and
> I didn't want to write a mail without having any work to show to you
> guys. Thanks a lot for being patient with me. :)

No problem.  I'm likewise busy or on (easter) vacation at times and fail
to send timely answers (sorry for that).

HTH & take care,
  Gerd

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

On Fri, 16 Apr 2021 at 17:02, Gerd Hoffmann <kraxel@redhat.com> wrote:

>   Hi,
>
> > I learned that the callback passed in AUD_open_out, (lets call it the
> write
> > audio callback,)  is supposed to mix and write the
> > buffers to HWVoiceOut. I have written that, the basic algorithm being:
> >
> > 1. Pop element from tx virtqueue.
> > 2. Get the xfer header from the elem->out_sg (iov_to_buf(elem->out_sg, 1,
> > 0, &hdr, sizeof(hdr)))
> > 3. Get the buffer from elem->out_sg (iov_to_buf(elem->out_sg, 1,
> > sizeof(hdr), &mixbuf, period_bytes))
> > 4. AUD_write the buffer
>
> AUD_write returns the number of bytes actually accepted.
>
> In case the audio backend consumed the complete buffer you can go ahead
> as described.  Otherwise stop here and resume (try AUD_write() the
> remaining data) when the callback is called again.
>
The callback that is supposed to write the buffers to HWVoiceOut has to
do it such that it only writes the buffers for the streams that it was
called by. The tx
virtqueue is going to have buffers for all the streams. How do I handle
this situation?

Say, I have 3 output streams 0, 1 and 2, the callback was called on stream
0, how do
I only play buffers corresponding to this stream? Is there a way by which I
could iterate over
the virtqeueue?

If not then I should probably store the VirtQueueElements corresponding to
a specific stream
in the stream itself. I have to store the elements because I'd have to
write to the response and
mark these elements as used, (via virtqueue_push i think.)

Also, is virtqueue a 'queue'? I remember reading in the specs that the
device can consume
the buffers in any order it prefers unless a feature bit is negotiated. So
does virtqueue_pop
always return the first element that entered the queue? (FIFO?)

If it is FIFO, then I think the only way would be to iterate over the
virtqueue. However if `virtqueu_pop`
does return the last inserted element, I can use `handle_tx


> No problem.  I'm likewise busy or on (easter) vacation at times and fail
> to send timely answers (sorry for that).
>
> HTH & take care,
>   Gerd
>
>

[-- Attachment #2: Type: text/html, Size: 2894 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 2614 bytes --]

On Mon, 19 Apr 2021 at 18:00, Shreyansh Chouhan <
chouhan.shreyansh2702@gmail.com> wrote:

>
> On Fri, 16 Apr 2021 at 17:02, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>>   Hi,
>>
>> > I learned that the callback passed in AUD_open_out, (lets call it the
>> write
>> > audio callback,)  is supposed to mix and write the
>> > buffers to HWVoiceOut. I have written that, the basic algorithm being:
>> >
>> > 1. Pop element from tx virtqueue.
>> > 2. Get the xfer header from the elem->out_sg (iov_to_buf(elem->out_sg,
>> 1,
>> > 0, &hdr, sizeof(hdr)))
>> > 3. Get the buffer from elem->out_sg (iov_to_buf(elem->out_sg, 1,
>> > sizeof(hdr), &mixbuf, period_bytes))
>> > 4. AUD_write the buffer
>>
>> AUD_write returns the number of bytes actually accepted.
>>
>> In case the audio backend consumed the complete buffer you can go ahead
>> as described.  Otherwise stop here and resume (try AUD_write() the
>> remaining data) when the callback is called again.
>>
> The callback that is supposed to write the buffers to HWVoiceOut has to
> do it such that it only writes the buffers for the streams that it was
> called by. The tx
> virtqueue is going to have buffers for all the streams. How do I handle
> this situation?
>
> Say, I have 3 output streams 0, 1 and 2, the callback was called on stream
> 0, how do
> I only play buffers corresponding to this stream? Is there a way by which
> I could iterate over
> the virtqeueue?
>
> If not then I should probably store the VirtQueueElements corresponding to
> a specific stream
> in the stream itself. I have to store the elements because I'd have to
> write to the response and
> mark these elements as used, (via virtqueue_push i think.)
>
> Also, is virtqueue a 'queue'? I remember reading in the specs that the
> device can consume
> the buffers in any order it prefers unless a feature bit is negotiated. So
> does virtqueue_pop
> always return the first element that entered the queue? (FIFO?)
>
> If it is FIFO, then I think the only way would be to iterate over the
> virtqueue. However if `virtqueu_pop`
> does return the last inserted element, I can use `handle_tx`
>
to get the last element inserted and then add it to the corresponding
stream.

Sorry for the incomplete mail. I don't know how I hit send.

>
>
>> No problem.  I'm likewise busy or on (easter) vacation at times and fail
>> to send timely answers (sorry for that).
>>
> Also that's fine, I just got a bit worried :)

I think once this `tx` part is done I'll have the pcm streams ready, and I
can send in patches by tonight
or tomorrow evening latest.

>
>> HTH & take care,
>>   Gerd
>>
>>

[-- Attachment #2: Type: text/html, Size: 4255 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Gerd Hoffmann]

> > AUD_write returns the number of bytes actually accepted.
> >
> > In case the audio backend consumed the complete buffer you can go ahead
> > as described.  Otherwise stop here and resume (try AUD_write() the
> > remaining data) when the callback is called again.
> >
> The callback that is supposed to write the buffers to HWVoiceOut has to
> do it such that it only writes the buffers for the streams that it was
> called by. The tx
> virtqueue is going to have buffers for all the streams. How do I handle
> this situation?

Well, maybe splitting the streams is a good job for the tx handler, i.e.
take the queue element out of the queue, check stream, stick into a
stream-specific buffer list.

The have the AUD_write callback check the stream buffer list.
instead of the virt queue.

Make sure you only complete the virtqueue elements after the buffers
have been passed to the qemu audiodev completely.

Yes, it is perfectly fine to process multiple virtqueue elements at the
same time, and out-of-order completion is fine too.

HTH,
  Gerd

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

Hi,

I am sorry for the absence of activity on this. A couple of people very
close to me died, and I also got busy
with the linux kernel mentorship program for a while. It was a weird year.
But I am back on this now.

I have the basic functionality of the sound card working. I tested it on an
ubuntu vm and it plays audio just fine. I'd like to test
other things like changing stream parameters and channel maps. Also the
VIRTIO_SND_PCM_F_MSG_POLLING,
VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS, VIRTIO_SND_PCM_F_EVT_XRUNS features
aren't implemented yet.
Implementing VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS hopefully shouldn't be much
of a challenge. We can send a notification
when we have processed period bytes (if I correctly understand this).

I will send another rfc patch series once I have channel maps and jacks
ready. And one more when I have these PCM features
ready.

[-- Attachment #2: Type: text/html, Size: 1047 bytes --]

Re: Fwd: VirtioSound device emulation implementation

[Laurent Vivier]

Le 29/12/2021 à 06:52, Shreyansh Chouhan a écrit :
> Hi,
> 
> I am sorry for the absence of activity on this. A couple of people very close to me died, and I also 
> got busy
> with the linux kernel mentorship program for a while. It was a weird year. But I am back on this now.
> 
> I have the basic functionality of the sound card working. I tested it on an ubuntu vm and it plays 
> audio just fine. I'd like to test
> other things like changing stream parameters and channel maps. Also the VIRTIO_SND_PCM_F_MSG_POLLING,
> VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS, VIRTIO_SND_PCM_F_EVT_XRUNS features aren't implemented yet.
> Implementing VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS hopefully shouldn't be much of a challenge. We can 
> send a notification
> when we have processed period bytes (if I correctly understand this).
> 
> I will send another rfc patch series once I have channel maps and jacks ready. And one more when I 
> have these PCM features
> ready.

Hi,

I'm trying to test your RFC series but I have a core dump (using alsa-utils speaker-test), do you 
have an updated branch I could pull?

Thanks,
Laurent

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

On Thu, 20 Jan 2022 at 16:09, Laurent Vivier <laurent@vivier.eu> wrote:
>
> Le 29/12/2021 à 06:52, Shreyansh Chouhan a écrit :
> > Hi,
> >
> > I am sorry for the absence of activity on this. A couple of people very close to me died, and I also
> > got busy
> > with the linux kernel mentorship program for a while. It was a weird year. But I am back on this now.
> >
> > I have the basic functionality of the sound card working. I tested it on an ubuntu vm and it plays
> > audio just fine. I'd like to test
> > other things like changing stream parameters and channel maps. Also the VIRTIO_SND_PCM_F_MSG_POLLING,
> > VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS, VIRTIO_SND_PCM_F_EVT_XRUNS features aren't implemented yet.
> > Implementing VIRTIO_SND_PCM_F_EVT_SHMEM_PERIODS hopefully shouldn't be much of a challenge. We can
> > send a notification
> > when we have processed period bytes (if I correctly understand this).
> >
> > I will send another rfc patch series once I have channel maps and jacks ready. And one more when I
> > have these PCM features
> > ready.
>
Hi,
>
> Hi,
>
> I'm trying to test your RFC series but I have a core dump (using alsa-utils speaker-test), do you
> have an updated branch I could pull?

Yes, I will send the updated RFC by tomorrow (Tuesday).

Thanks,
Shreyansh Chouhan

>
>
> Thanks,
> Laurent

Re: Fwd: VirtioSound device emulation implementation

[Shreyansh Chouhan]

On Mon, 7 Feb 2022 at 09:36, Shreyansh Chouhan
<chouhan.shreyansh2702@gmail.com> wrote:
>
> On Thu, 20 Jan 2022 at 16:09, Laurent Vivier <laurent@vivier.eu> wrote:
> >
> >
> > Hi,
> >
> > I'm trying to test your RFC series but I have a core dump (using alsa-utils speaker-test), do you
> > have an updated branch I could pull?
>
> Yes, I will send the updated RFC by tomorrow (Tuesday).
>

Sorry I won't be able to send it in today. I have the sound card ready
but I have yet to fix some things
from the comments on the previous RFC. I joined at a new position
today and was expecting to be able
to work on this after work ended. I still haven't been able to
complete the patches. I will send the updated
patch series latest by Friday.

Thanks,
Shreyansh Chouhan

> Thanks,
> Shreyansh Chouhan
>
> >
> >
> > Thanks,
> > Laurent