Ability to customize the prserver connection?

Ability to customize the prserver connection?

[Rusty Howell]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

Hello, is there a way to (or is there interest from others) in being able to customize the connection to the prserver?   We have a need to send an extra parameter to the prserver (which we have also customized).  We were able to do this by monkeypatching the prserv client during the do_packagedata task. (Rule #5 Never modify Poky)  We had a custom TaskStarted handler which would monkeypatch the getPR() method and add the extra parameter we wanted.  We recently upgraded to honister which broke our method of monkeypatching.

I know most Yocto users focus on images and don't bother with package feeds. But for those who do, I wanted to work on a way to allow one to create a custom prserver connection rather than being forced to use the client in bb/lib/prserv/client.py and hack at it via inherits and task handlers.

[-- Attachment #2: Type: text/html, Size: 859 bytes --]

Re: [poky] Ability to customize the prserver connection?

[Richard Purdie]

On Wed, 2023-05-03 at 10:03 -0700, Rusty Howell wrote:
> Hello, is there a way to (or is there interest from others) in being
> able to customize the connection to the prserver?   We have a need to
> send an extra parameter to the prserver (which we have also
> customized).  We were able to do this by monkeypatching the prserv
> client during the do_packagedata task. (Rule #5 Never modify Poky) 
> We had a custom TaskStarted handler which would monkeypatch the
> getPR() method and add the extra parameter we wanted.  We recently
> upgraded to honister which broke our method of monkeypatching.
> 
> I know most Yocto users focus on images and don't bother with package
> feeds. But for those who do, I wanted to work on a way to allow one
> to create a custom prserver connection rather than being forced to
> use the client in bb/lib/prserv/client.py and hack at it via inherits
> and task handlers.

I guess we've assumed you might just have multiple PR servers for this
situation and switch to the right one for what you want. I'm not aware
of a way to specifically do what you're asking for here at present.

The PR Server code and approach is all a bit old and creaky. I think
adding some kind of customisable field/data would likely be a
reasonable extension. I know the hash equivalence server handles
situations like this in a much better way.

So I'm open to proposals for this (and to clean up/improve PR serv in
general). I would like to understand more about what kind of data this
is if possible so we can better understand the use scenarios for PR
serv and hopefully ensure it better suits real world usage in future.

Cheers,

Richard

Re: [poky] Ability to customize the prserver connection?

[Rusty Howell]

[-- Attachment #1: Type: text/plain, Size: 3480 bytes --]

Hi Richard,

My company has need of proper package feeds that support multiple active OS
releases. That is, we need to have the ability to work on OSv1.1 and OSv2
simultaneously. The problem we are having is that PRAUTO responses from the
prserver do not have a concept of OS release. The later query gets a higher
number. So if we build OSv1.1 after OSv2, that means that OSv1.1 has a
package with a greater PRAUTO value than it's counterpart in OSv2.  In that
scenario, our OS dist-upgrade will be incomplete. I solved this initially
by hacking the PRserver to include an int os_version, based on a variable
in our site.conf. And then I changed the prserver to support this extra
param with another column in the database.  Now queries to the prserver
consist of a tuple like ("base-files-1.0-r0", "aarch64", "12345678978",
"11")  where "11" is the new os_version param, and the OSv2 counterpart
("base-files-1.0-r0", "aarch64", "12345678978", "20") where "20" is the new
param.  The SELECT queries to the database find the max value for that
package/arch and os_version <=X.

In this manner, any increase in the OSv1.1 package, will cause the OSv2
package to get bumped on the next build automatically, regardless of a
checksum match or not.
Recently with our Honister upgrade, we had the added complexity of changes
to the transport protocol (xmlrpc to socket+json) as well as simple changes
to package.bbclass has caused us redo some things. And hence the idea of
being able to provide our own custom prserver client for bitbake to consume.
I hope this explains our needs well enough.

Rusty


On Wed, May 3, 2023 at 12:03 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2023-05-03 at 10:03 -0700, Rusty Howell wrote:
> > Hello, is there a way to (or is there interest from others) in being
> > able to customize the connection to the prserver?   We have a need to
> > send an extra parameter to the prserver (which we have also
> > customized).  We were able to do this by monkeypatching the prserv
> > client during the do_packagedata task. (Rule #5 Never modify Poky)
> > We had a custom TaskStarted handler which would monkeypatch the
> > getPR() method and add the extra parameter we wanted.  We recently
> > upgraded to honister which broke our method of monkeypatching.
> >
> > I know most Yocto users focus on images and don't bother with package
> > feeds. But for those who do, I wanted to work on a way to allow one
> > to create a custom prserver connection rather than being forced to
> > use the client in bb/lib/prserv/client.py and hack at it via inherits
> > and task handlers.
>
> I guess we've assumed you might just have multiple PR servers for this
> situation and switch to the right one for what you want. I'm not aware
> of a way to specifically do what you're asking for here at present.
>
> The PR Server code and approach is all a bit old and creaky. I think
> adding some kind of customisable field/data would likely be a
> reasonable extension. I know the hash equivalence server handles
> situations like this in a much better way.
>
> So I'm open to proposals for this (and to clean up/improve PR serv in
> general). I would like to understand more about what kind of data this
> is if possible so we can better understand the use scenarios for PR
> serv and hopefully ensure it better suits real world usage in future.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 4112 bytes --]

Re: [poky] Ability to customize the prserver connection?

[Richard Purdie]

On Wed, 2023-05-03 at 13:06 -0600, Rusty Howell wrote:
> My company has need of proper package feeds that support multiple
> active OS releases. That is, we need to have the ability to work on
> OSv1.1 and OSv2 simultaneously. The problem we are having is that
> PRAUTO responses from the prserver do not have a concept of OS
> release. The later query gets a higher number. So if we build OSv1.1
> after OSv2, that means that OSv1.1 has a package with a greater
> PRAUTO value than it's counterpart in OSv2.  In that scenario, our OS
> dist-upgrade will be incomplete. I solved this initially by hacking
> the PRserver to include an int os_version, based on a variable in our
> site.conf. And then I changed the prserver to support this extra
> param with another column in the database.  Now queries to the
> prserver consist of a tuple like ("base-files-1.0-r0", "aarch64",
> "12345678978", "11")  where "11" is the new os_version param, and the
> OSv2 counterpart ("base-files-1.0-r0", "aarch64", "12345678978",
> "20") where "20" is the new param.  The SELECT queries to the
> database find the max value for that package/arch and os_version <=X.
> 
> In this manner, any increase in the OSv1.1 package, will cause the
> OSv2 package to get bumped on the next build automatically,
> regardless of a checksum match or not.
> Recently with our Honister upgrade, we had the added complexity of
> changes to the transport protocol (xmlrpc to socket+json) as well as
> simple changes to package.bbclass has caused us redo some things. And
> hence the idea of being able to provide our own custom prserver
> client for bitbake to consume.
> I hope this explains our needs well enough.

That makes sense. It could be done with alternate ports but I'd take a
new field which is used as an ID to make this possible with one port.
Patches welcome.

Cheers,

Richard