Re: [PATCH v2 1/4] drm/komeda: Add a new helper drm_color_ctm_s31_32_to_qm_n()

From: Ilia Mirkin <imirkin@alum.mit.edu>
To: "james qian wang (Arm Technology China)" <james.qian.wang@arm.com>
Cc: Liviu Dudau <Liviu.Dudau@arm.com>,
	"airlied@linux.ie" <airlied@linux.ie>,
	Brian Starkey <Brian.Starkey@arm.com>,
	"maarten.lankhorst@linux.intel.com" 
	<maarten.lankhorst@linux.intel.com>,
	"sean@poorly.run" <sean@poorly.run>,
	"Jonathan Chai (Arm Technology China)" <Jonathan.Chai@arm.com>,
	"Julien Yin (Arm Technology China)" <Julien.Yin@arm.com>,
	"Thomas Sun (Arm Technology China)" <thomas.Sun@arm.com>,
	"Lowry Li (Arm Technology China)" <Lowry.Li@arm.com>,
	Ayan Halder <Ayan.Halder@arm.com>,
	"Tiannan Zhu (Arm Technology China)" <Tiannan.Zhu@arm.com>,
	"Yiqi Kang (Arm Technology China)" <Yiqi.Kang@arm.com>,
	nd <nd@arm.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"dri-devel@lists.freedesktop.org"
	<dri-devel@lists.freedesktop.org>, Ben Davis <Ben.Davis@arm.com>,
	"Oscar Zhang (Arm Technology China)" <Oscar.Zhang@arm.com>,
	"Channing Chen (Arm Technology China)" <Channing.Chen@arm.com>,
	Mihail Atanassov <Mihail.Atanassov@arm.com>
Subject: Re: [PATCH v2 1/4] drm/komeda: Add a new helper drm_color_ctm_s31_32_to_qm_n()
Date: Mon, 14 Oct 2019 23:48:38 -0400	[thread overview]
Message-ID: <CAKb7Uvh7y20oikYR+UpabgXLHJM2i+2DPVyYSwE37d=NpheUGg@mail.gmail.com> (raw)
In-Reply-To: <20191015011604.GA26941@jamwan02-TSP300>

On Mon, Oct 14, 2019 at 9:16 PM james qian wang (Arm Technology China)
<james.qian.wang@arm.com> wrote:
> On Mon, Oct 14, 2019 at 11:58:48AM -0400, Ilia Mirkin wrote:
> > On Fri, Oct 11, 2019 at 1:43 AM james qian wang (Arm Technology China)
> > <james.qian.wang@arm.com> wrote:
> > > + *
> > > + * Convert and clamp S31.32 sign-magnitude to Qm.n 2's complement.
> > > + */
> > > +uint64_t drm_color_ctm_s31_32_to_qm_n(uint64_t user_input,
> > > +                                     uint32_t m, uint32_t n)
> > > +{
> > > +       u64 mag = (user_input & ~BIT_ULL(63)) >> (32 - n);
> > > +       bool negative = !!(user_input & BIT_ULL(63));
> > > +       s64 val;
> > > +
> > > +       /* the range of signed 2s complement is [-2^n+m, 2^n+m - 1] */
> >
> > This implies that n = 32, m = 0 would actually yield a 33-bit 2's
> > complement number. Is that what you meant?
>
> Yes, since m doesn't include sign-bit So a Q0.32 is a 33bit value.

This goes counter to what the wikipedia page says [
https://en.wikipedia.org/wiki/Q_(number_format) ]:

(reformatted slightly for text-only consumption):

"""
For example, a Q15.1 format number:

- requires 15+1 = 16 bits
- its range is [-2^14, 2^14 - 2^-1] = [-16384.0, +16383.5] = [0x8000,
0x8001 ... 0xFFFF, 0x0000, 0x0001 ... 0x7FFE, 0x7FFF]
- its resolution is 2^-1 = 0.5
"""

This suggests that the proper way to represent a standard 32-bit 2's
complement integer would be Q32.0.

  -ilia