[Printing-architecture] Use of filter functions in CUPS 2.x and 3.x

[Printing-architecture] Use of filter functions in CUPS 2.x and 3.x

[Till Kamppeter]

Hi,

with the background of CUPS 3.x, planned to get released 2 years from 
now, not supporting PPD files any more and only considering driverless 
IPP printers I have changed the architecture of cups-filters, converting 
filter executables (CUPS filters) into filter functions (library 
functions doing a filter's task, with standardized call 
scheme/interface) so that the code of the filters is preserved but they 
get more universally usable.

https://openprinting.github.io/OpenPrinting-News-October-2021/#cups

In addition I have created some auxiliary functions (partially filter 
functions by themselves) to call filter functions in a chain, feed data 
into filter through a pipe, call classic CUPS external filter/backend 
executable wrapped into a filter function, save data stream between 
chained filter function calls into a file for debugging, ...

https://openprinting.github.io/news/
https://github.com/OpenPrinting/cups-filters

With this I was especially able to create Printer Applications 
retro-fitting classic CUPS printer drivers, an important step to be able 
to switch to a PPD-less and driverless CUPS without dropping support for 
legacy printers.

https://snapcraft.io/search?q=OpenPrinting
https://github.com/OpenPrinting/pappl-retrofit

Now I am also thinking about making use of the filter functions somehow 
in CUPS, both to get improvement in the upcoming 2 years where CUPS 
still supports PPD files and classic drivers and also in PPD-less CUPS 
3.x, where data-format conversions are still needed, as for example 
print jobs usually come in PDF but it is not required for a driverless 
IPP printer to support PDF. Also implementation of functionality like 
N-up or flattening filled PDF forms (both tasks currently done by 
pdftopdf) is needed in CUPS 3.x.

We should do our best to avoid duplicate code in OpenPrinting and not 
re-invent the wheel.


"universal" filter for CUPS
---------------------------

https://gist.github.com/pranshukharkwal/9413499a6744049ef549159948392023

As a first approach to improve PPDish CUPS 2.4.x and 2.5.x I have run 
the GSoC project of a universal CUPS filter, where one single CUPS 
filter executable does all the filtering for a job by calling filter 
functions, in a chain if needed. This reduces the number of external 
executable calls by CUPS vastly, as normally CUPS calls for each filter 
to run first the cups-exec helper program and cups-exec then calls the 
filter executable, making up 2 external executable calls per filter.

The filter function chaining in the universal CUPS filter still does a 
fork for each filter function though and pipes the print data from 
filter function to filter function. Also filters of printer drivers and 
CUPS backends are not part of the universal filter and need to get 
called separately by CUPS.

The disk space saved is low, as with cups-filters 2.x each filter 
executable in /usr/lib/cups/filters/ is only a little code stub calling 
the actual filter function in libcupsfilters,

Also the universal filter still needs a fix to work correctly with PPD 
files which use "cupsFilter2" instead of "cupsFilter" lines.

So it is a little bit of a question whether it is really worth the 
effort to replace the individual filters called by CUPS 2.4.x and 2.5.x 
by this universal filter, especially also that we have only more 2 years 
where CUPS uses PPD files.

So before I complete this (if not, the universal filter will at least be 
used to make cups-browsed's "implicitclass" backend use filter functions 
instead of external filters) I want to hear some opinions about this, 
especially also whether actually saving external executable calls is 
more resource-saving than for example saving forked parallel tasks and 
pipes between them.


Forking and piping vs. one filter after the other
-------------------------------------------------

In general if forking and piping for a filter chain consumes much more 
than calling each filter function one after the other directly and let 
them write their output into temporary files for the next filter 
functions in the chain reading from the previous filter function's temp 
file I am also thinking about adding a second mode to filterChain() to 
let it operate this way. WDYT?


Using filter functions directly in CUPS
---------------------------------------

And, finally, should we make the filter functions be directly used by 
CUPS, either that in CUPS 2.5.x in scheduler/job.c we call filter 
functions instead of external filter executables (at least for standard, 
non-driver filters), or that for CUPS 3.x we do the needed file format 
conversions by filter function calls (there are no driver filters)? 
Backends could also be converted to filter functions and be directly 
called. WDYT?

Also, do we need extra functionality in the filter function concept for 
using it directly in CUPS? For example add a field to the records of the 
filter functions called by filterChain() to tell as which user each 
individual filter function should get run?

Also, if we want CUPS to use filter functions, we need to do 
re-structuring to avoid circular dependencies, as libcupsfilters uses 
libcups and if a function in libcups would call a filter function of 
libcupsfilters, libcups is using libcupsfilters, ... The planned 
splitting of libcups, local CUPS daemon, sharing CUPS daemon of CUPS 3.x 
could help here as if the daemons call filter functions but not libcups, 
we will not get a circular dependency. WDYT?


It would be great if we could discuss these topics before finalizing 
cups-filters 2.x

    Till

Re: [Printing-architecture] Use of filter functions in CUPS 2.x and 3.x

[Zdenek Dohnal]

[-- Attachment #1: Type: text/plain, Size: 7306 bytes --]

Hi Till,

I'm sorry for answering this late - HDD got wasted, Christmas, all new year
responsibilities and I'm not particularly strong on my feet regarding
filters, so I'm not sure whether my feedback will relevant, but so there it
is:

On Mon, Dec 13, 2021 at 10:07 PM Till Kamppeter <till.kamppeter@gmail.com>
wrote:

> Hi,
>
> with the background of CUPS 3.x, planned to get released 2 years from
> now, not supporting PPD files any more and only considering driverless
> IPP printers I have changed the architecture of cups-filters, converting
> filter executables (CUPS filters) into filter functions (library
> functions doing a filter's task, with standardized call
> scheme/interface) so that the code of the filters is preserved but they
> get more universally usable.
>

Kudos for doing that! I would propose to leave each filter binaries for
check/test target and use them in the test suite. IMO it would be great to
know that at least common use cases work the same way as with the previous
filter binaries.


>
> Now I am also thinking about making use of the filter functions somehow
> in CUPS, both to get improvement in the upcoming 2 years where CUPS
> still supports PPD files and classic drivers and also in PPD-less CUPS
> 3.x, where data-format conversions are still needed, as for example
> print jobs usually come in PDF but it is not required for a driverless
> IPP printer to support PDF. Also implementation of functionality like
> N-up or flattening filled PDF forms (both tasks currently done by
> pdftopdf) is needed in CUPS 3.x.
>

Agreed - IMO it is sensible to spread new features in the timeframe instead
of putting all changes into work at once and got hit by numerous possible
issues.


>
> We should do our best to avoid duplicate code in OpenPrinting and not
> re-invent the wheel.
>
+1

>
>
> "universal" filter for CUPS
> ---------------------------
>
> https://gist.github.com/pranshukharkwal/9413499a6744049ef549159948392023
>
> As a first approach to improve PPDish CUPS 2.4.x and 2.5.x I have run
> the GSoC project of a universal CUPS filter, where one single CUPS
> filter executable does all the filtering for a job by calling filter
> functions, in a chain if needed. This reduces the number of external
> executable calls by CUPS vastly, as normally CUPS calls for each filter
> to run first the cups-exec helper program and cups-exec then calls the
> filter executable, making up 2 external executable calls per filter.
>
> The filter function chaining in the universal CUPS filter still does a
> fork for each filter function though and pipes the print data from
> filter function to filter function. Also filters of printer drivers and
> CUPS backends are not part of the universal filter and need to get
> called separately by CUPS.
>
> The disk space saved is low, as with cups-filters 2.x each filter
> executable in /usr/lib/cups/filters/ is only a little code stub calling
> the actual filter function in libcupsfilters,
>
> Also the universal filter still needs a fix to work correctly with PPD
> files which use "cupsFilter2" instead of "cupsFilter" lines.
>
> So it is a little bit of a question whether it is really worth the
> effort to replace the individual filters called by CUPS 2.4.x and 2.5.x
> by this universal filter, especially also that we have only more 2 years
> where CUPS uses PPD files.
>
> So before I complete this (if not, the universal filter will at least be
> used to make cups-browsed's "implicitclass" backend use filter functions
> instead of external filters) I want to hear some opinions about this,
> especially also whether actually saving external executable calls is
> more resource-saving than for example saving forked parallel tasks and
> pipes between them.
>

IMO it would be great to have this before we fully switch to non-PPD
printing stack - this way we could test the filtering functions on more
places than just in printer applications, which still aren't widely used
for now.


>
>
> Forking and piping vs. one filter after the other
> -------------------------------------------------
>
> In general if forking and piping for a filter chain consumes much more
> than calling each filter function one after the other directly and let
> them write their output into temporary files for the next filter
> functions in the chain reading from the previous filter function's temp
> file I am also thinking about adding a second mode to filterChain() to
> let it operate this way. WDYT?
>

To be honest forking and piping sounds better for me - a file can stay on
system if something goes wrong, taking the resources, additionally
protective systems can complain about accessing/creating files in
unexpected location. AFAIK forking and piping sounds cleaner to me...


>
>
> Using filter functions directly in CUPS
> ---------------------------------------
>
> And, finally, should we make the filter functions be directly used by
> CUPS, either that in CUPS 2.5.x in scheduler/job.c we call filter
> functions instead of external filter executables (at least for standard,
> non-driver filters), or that for CUPS 3.x we do the needed file format
> conversions by filter function calls (there are no driver filters)?
> Backends could also be converted to filter functions and be directly
> called. WDYT?
>

I would prefer universal filter binary for the current CUPS 2.x, because,
as you wrote below, filter functions from libcupsfilters would cause
circular deps and the CUPS splitting is best done in CUPS 3.0.


>
> Also, do we need extra functionality in the filter function concept for
> using it directly in CUPS? For example add a field to the records of the
> filter functions called by filterChain() to tell as which user each
> individual filter function should get run?
>

I'm not sure if I am able to tell something relevant about this - how I
imagine the filter function is a 'in' pointer, 'out' pointer, ipp
attributes of the destination and array of CUPS options related to the job
(basically similar what current filter binaries take as parameters and
produce, with exception that instead of PPD we have IPP attributes). Maybe
to have some 'void' pointer for additional data, if some filter function
will need more info?


>
> Also, if we want CUPS to use filter functions, we need to do
> re-structuring to avoid circular dependencies, as libcupsfilters uses
> libcups and if a function in libcups would call a filter function of
> libcupsfilters, libcups is using libcupsfilters, ... The planned
> splitting of libcups, local CUPS daemon, sharing CUPS daemon of CUPS 3.x
> could help here as if the daemons call filter functions but not libcups,
> we will not get a circular dependency. WDYT?
>

As I wrote above, I would prefer an universal driver binary because of this
for now to prevent the circular dependency and pasting cups-filters code
into CUPS.



>
>
> It would be great if we could discuss these topics before finalizing
> cups-filters 2.x
>

Looking forward to cups-filters-2.0 :)

Zdenek


>
>     Till
> _______________________________________________
> Printing-architecture mailing list
> Printing-architecture@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/printing-architecture
>
>

-- 
Zdenek Dohnal
Software Engineer
Red Hat, BRQ-TPBC

[-- Attachment #2: Type: text/html, Size: 10125 bytes --]

Re: [Printing-architecture] Use of filter functions in CUPS 2.x and 3.x

[Till Kamppeter]

On 20/01/2022 10:53, Zdenek Dohnal wrote:
> Hi Till,
> 
> I'm sorry for answering this late - HDD got wasted, Christmas, all new 
> year responsibilities and I'm not particularly strong on my feet 
> regarding filters, so I'm not sure whether my feedback will relevant, 
> but so there it is:
> 
> On Mon, Dec 13, 2021 at 10:07 PM Till Kamppeter 
> <till.kamppeter@gmail.com <mailto:till.kamppeter@gmail.com>> wrote:
> 
>     Hi,
> 
>     with the background of CUPS 3.x, planned to get released 2 years from
>     now, not supporting PPD files any more and only considering driverless
>     IPP printers I have changed the architecture of cups-filters,
>     converting
>     filter executables (CUPS filters) into filter functions (library
>     functions doing a filter's task, with standardized call
>     scheme/interface) so that the code of the filters is preserved but they
>     get more universally usable.
> 
> 
> Kudos for doing that! I would propose to leave each filter binaries for 
> check/test target and use them in the test suite. IMO it would be great 
> to know that at least common use cases work the same way as with the 
> previous filter binaries.
> 

There are still filter binaries for calling the filters the classic way. 
To not duplicate code they are stubs which do not more than calling the 
corresponding filter function. The binaries are in filter/ (as the 
executables before) and the filter functions in libcupsfilter/. So 
filters can be called in 3 ways:

1. As individual executables
2. Filter function chains called by the "universal" filter
3. Filter functions directly

>     So before I complete this (if not, the universal filter will at
>     least be
>     used to make cups-browsed's "implicitclass" backend use filter
>     functions
>     instead of external filters) I want to hear some opinions about this,
>     especially also whether actually saving external executable calls is
>     more resource-saving than for example saving forked parallel tasks and
>     pipes between them.
> 
> 
> IMO it would be great to have this before we fully switch to non-PPD 
> printing stack - this way we could test the filtering functions on more 
> places than just in printer applications, which still aren't widely used 
> for now.
> 

The simple switch-over to cups-filters 2.x already makes the filter 
functions being used, called through the individual wrappers (see (1) 
above). By simply replacing a config file one can switch the universal 
wrapper calling filter function chains. So a quick switch-over to 
cups-filters 2.x will give us already a lot of testing before we get 
CUPS 3.x.

Also more use of the Printer Applications (for example in distros with 
snapped/sandboxed CUPS) will provide more testing.

> 
> 
>     Forking and piping vs. one filter after the other
>     -------------------------------------------------
> 
>     In general if forking and piping for a filter chain consumes much more
>     than calling each filter function one after the other directly and let
>     them write their output into temporary files for the next filter
>     functions in the chain reading from the previous filter function's temp
>     file I am also thinking about adding a second mode to filterChain() to
>     let it operate this way. WDYT?
> 
> 
> To be honest forking and piping sounds better for me - a file can stay 
> on system if something goes wrong, taking the resources, additionally 
> protective systems can complain about accessing/creating files in 
> unexpected location. AFAIK forking and piping sounds cleaner to me...
> 

OK.

> 
> 
>     Using filter functions directly in CUPS
>     ---------------------------------------
> 
>     And, finally, should we make the filter functions be directly used by
>     CUPS, either that in CUPS 2.5.x in scheduler/job.c we call filter
>     functions instead of external filter executables (at least for
>     standard,
>     non-driver filters), or that for CUPS 3.x we do the needed file format
>     conversions by filter function calls (there are no driver filters)?
>     Backends could also be converted to filter functions and be directly
>     called. WDYT?
> 
> 
> I would prefer universal filter binary for the current CUPS 2.x, 
> because, as you wrote below, filter functions from libcupsfilters would 
> cause circular deps and the CUPS splitting is best done in CUPS 3.0.
> 

OK.

> 
>     Also, do we need extra functionality in the filter function concept for
>     using it directly in CUPS? For example add a field to the records of
>     the
>     filter functions called by filterChain() to tell as which user each
>     individual filter function should get run?
> 
> 
> I'm not sure if I am able to tell something relevant about this - how I 
> imagine the filter function is a 'in' pointer, 'out' pointer, ipp 
> attributes of the destination and array of CUPS options related to the 
> job (basically similar what current filter binaries take as parameters 
> and produce, with exception that instead of PPD we have IPP attributes). 
> Maybe to have some 'void' pointer for additional data, if some filter 
> function will need more info?
> 

The call scheme of filter functions already has this void pointer, the 
"parameters" pointer. See use of filter functions in the source code of 
the universal() filter function and also in libpappl-retrofit, the 
job-processing part especially.

> 
>     Also, if we want CUPS to use filter functions, we need to do
>     re-structuring to avoid circular dependencies, as libcupsfilters uses
>     libcups and if a function in libcups would call a filter function of
>     libcupsfilters, libcups is using libcupsfilters, ... The planned
>     splitting of libcups, local CUPS daemon, sharing CUPS daemon of CUPS
>     3.x
>     could help here as if the daemons call filter functions but not
>     libcups,
>     we will not get a circular dependency. WDYT?
> 
> 
> As I wrote above, I would prefer an universal driver binary because of 
> this for now to prevent the circular dependency and pasting cups-filters 
> code into CUPS.
> 

OK.

    Till