The thing that seems to trigger the sRGB correction is the fact that the
destination texture has an sRGB internal. If I change it to GL_RGBA8 I get the
expected result.

The format of the source RB doesn't seem to matter here.