The transcoding test above now makes similar size to cbr.

There is a new issue though. The GOP patch (2) is problematic.

It doesn't show on the transcoding test, but after reading the patch comment
about gop affecting rate control I tested with gstreamer default (30) and max =

keyframe-period=300

With vbr plus high bitrates this makes a large difference in file size.

This still happens if I disable dual instance in radeon_vce.c.

CBR seems to be unaffected, though dual instance encoding seems to come out a
bit lower than single instance.

The vbr test =

gst-launch-1.0 -f filesrc location=/mnt/ramdisk/trees-1440p50.nv12
blocksize=5529600 !
video/x-raw,format=NV12,width=2560,height=1440,framerate=50/1 ! queue !
vaapih264enc rate-control=vbr bitrate=50000 keyframe-period=300 ! video/x-h264,
profile=baseline ! filesink location=/mnt/ramdisk/out.264

Result = 9.4M file (source is 500 frames)

above with keyframe-period=30 = 42M which is close to expected result.