Question about domain_init (v5.3-v5.7)

Question about domain_init (v5.3-v5.7)

[Jerry Snitselaar]

Is there a reason we check the requested guest address width against the
iommu's mgaw, instead of the agaw that we already know for the iommu?
I've run into a case with a new system where the mgaw reported is 57,
but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
the highest supported agaw is 48 and the domain_init code fails here. In
other places like prepare_domain_attach_device, the dmar domain agaw
gets adjusted down to the iommu agaw. The agaw of the iommu gets
determined based off what is reported for sagaw. I'm wondering if it
can't instead do:

---
 drivers/iommu/intel-iommu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 6ca5c92ef2e5..a8e41ec36d9e 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
 	domain_reserve_special_ranges(domain);

 	/* calculate AGAW */
-	if (guest_width > cap_mgaw(iommu->cap))
-	        guest_width = cap_mgaw(iommu->cap);
+	if (guest_width > agaw_to_width(iommu->agaw))
+	        guest_width = agaw_to_width(iommu->agaw);
 	domain->gaw = guest_width;
 	adjust_width = guestwidth_to_adjustwidth(guest_width);
 	agaw = width_to_agaw(adjust_width);
--
2.27.0


Thoughts? With the former code the ehci device for the ilo fails when
trying to get a private domain.

Thanks,
Jerry

Re: Question about domain_init (v5.3-v5.7)

[Lu Baolu]

Hi Jerry,

On 2020/11/26 4:27, Jerry Snitselaar wrote:
> 
> Is there a reason we check the requested guest address width against the
> iommu's mgaw, instead of the agaw that we already know for the iommu?
> I've run into a case with a new system where the mgaw reported is 57,
> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
> the highest supported agaw is 48 and the domain_init code fails here. In

Isn't this a platform bug? If it's too late to fix it in the BIOS, you
maybe have to add a platform specific quirk to set mgaw to the highest
supported agaw?

Best regards,
baolu

> other places like prepare_domain_attach_device, the dmar domain agaw
> gets adjusted down to the iommu agaw. The agaw of the iommu gets
> determined based off what is reported for sagaw. I'm wondering if it
> can't instead do:
> 
> ---
>   drivers/iommu/intel-iommu.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index 6ca5c92ef2e5..a8e41ec36d9e 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>   	domain_reserve_special_ranges(domain);
> 
>   	/* calculate AGAW */
> -	if (guest_width > cap_mgaw(iommu->cap))
> -	        guest_width = cap_mgaw(iommu->cap);
> +	if (guest_width > agaw_to_width(iommu->agaw))
> +	        guest_width = agaw_to_width(iommu->agaw);
>   	domain->gaw = guest_width;
>   	adjust_width = guestwidth_to_adjustwidth(guest_width);
>   	agaw = width_to_agaw(adjust_width);
> --
> 2.27.0
> 
> 
> Thoughts? With the former code the ehci device for the ilo fails when
> trying to get a private domain.
> 
> Thanks,
> Jerry
> 

Re: Question about domain_init (v5.3-v5.7)

[Jerry Snitselaar]

Lu Baolu @ 2020-11-26 04:01 MST:

> Hi Jerry,
>
> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>> Is there a reason we check the requested guest address width against
>> the
>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>> I've run into a case with a new system where the mgaw reported is 57,
>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>> the highest supported agaw is 48 and the domain_init code fails here. In
>
> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
> maybe have to add a platform specific quirk to set mgaw to the highest
> supported agaw?
>
> Best regards,
> baolu

Is there somewhere you can point me to that discusses how they should be
setting the mgaw? I misunderstood when I previously asked you about
whether the mgaw could be a value that was greater than any of sagaw.
If it is a bios issue, then they should fix it there.

>
>> other places like prepare_domain_attach_device, the dmar domain agaw
>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>> determined based off what is reported for sagaw. I'm wondering if it
>> can't instead do:
>> ---
>>   drivers/iommu/intel-iommu.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>> diff --git a/drivers/iommu/intel-iommu.c
>> b/drivers/iommu/intel-iommu.c
>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>> --- a/drivers/iommu/intel-iommu.c
>> +++ b/drivers/iommu/intel-iommu.c
>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>   	domain_reserve_special_ranges(domain);
>>   	/* calculate AGAW */
>> -	if (guest_width > cap_mgaw(iommu->cap))
>> -	        guest_width = cap_mgaw(iommu->cap);
>> +	if (guest_width > agaw_to_width(iommu->agaw))
>> +	        guest_width = agaw_to_width(iommu->agaw);
>>   	domain->gaw = guest_width;
>>   	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>   	agaw = width_to_agaw(adjust_width);
>> --
>> 2.27.0
>> 
>> Thoughts? With the former code the ehci device for the ilo fails when
>> trying to get a private domain.
>> Thanks,
>> Jerry
>> 

Re: Question about domain_init (v5.3-v5.7)

[Lu Baolu]

Hi Jerry,

On 11/27/20 5:35 AM, Jerry Snitselaar wrote:
> 
> Lu Baolu @ 2020-11-26 04:01 MST:
> 
>> Hi Jerry,
>>
>> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>>> Is there a reason we check the requested guest address width against
>>> the
>>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>>> I've run into a case with a new system where the mgaw reported is 57,
>>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>>> the highest supported agaw is 48 and the domain_init code fails here. In
>>
>> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
>> maybe have to add a platform specific quirk to set mgaw to the highest
>> supported agaw?
>>
>> Best regards,
>> baolu
> 
> Is there somewhere you can point me to that discusses how they should be
> setting the mgaw? I misunderstood when I previously asked you about
> whether the mgaw could be a value that was greater than any of sagaw.
> If it is a bios issue, then they should fix it there.

MGAW indicates the max gpa width supported by 2nd translation. The VT-d
spec requires that this value must be at least equal to the host
physical addressibility. According to this, BIOS is good, right?

For this failure case, domain_init() just wants to find a suitable agaw
for the private domain. I think it makes sense to check against
iommu->agaw instead of cap_mgaw.

Best regards,
baolu

> 
>>
>>> other places like prepare_domain_attach_device, the dmar domain agaw
>>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>>> determined based off what is reported for sagaw. I'm wondering if it
>>> can't instead do:
>>> ---
>>>    drivers/iommu/intel-iommu.c | 4 ++--
>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>> diff --git a/drivers/iommu/intel-iommu.c
>>> b/drivers/iommu/intel-iommu.c
>>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>>> --- a/drivers/iommu/intel-iommu.c
>>> +++ b/drivers/iommu/intel-iommu.c
>>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>>    	domain_reserve_special_ranges(domain);
>>>    	/* calculate AGAW */
>>> -	if (guest_width > cap_mgaw(iommu->cap))
>>> -	        guest_width = cap_mgaw(iommu->cap);
>>> +	if (guest_width > agaw_to_width(iommu->agaw))
>>> +	        guest_width = agaw_to_width(iommu->agaw);
>>>    	domain->gaw = guest_width;
>>>    	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>>    	agaw = width_to_agaw(adjust_width);
>>> --
>>> 2.27.0
>>>
>>> Thoughts? With the former code the ehci device for the ilo fails when
>>> trying to get a private domain.
>>> Thanks,
>>> Jerry
>>>
> 

Re: Question about domain_init (v5.3-v5.7)

[Jerry Snitselaar]

Lu Baolu @ 2020-11-26 19:12 MST:

> Hi Jerry,
>
> On 11/27/20 5:35 AM, Jerry Snitselaar wrote:
>> Lu Baolu @ 2020-11-26 04:01 MST:
>> 
>>> Hi Jerry,
>>>
>>> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>>>> Is there a reason we check the requested guest address width against
>>>> the
>>>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>>>> I've run into a case with a new system where the mgaw reported is 57,
>>>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>>>> the highest supported agaw is 48 and the domain_init code fails here. In
>>>
>>> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
>>> maybe have to add a platform specific quirk to set mgaw to the highest
>>> supported agaw?
>>>
>>> Best regards,
>>> baolu
>> Is there somewhere you can point me to that discusses how they
>> should be
>> setting the mgaw? I misunderstood when I previously asked you about
>> whether the mgaw could be a value that was greater than any of sagaw.
>> If it is a bios issue, then they should fix it there.
>
> MGAW indicates the max gpa width supported by 2nd translation. The VT-d
> spec requires that this value must be at least equal to the host
> physical addressibility. According to this, BIOS is good, right?
>

Yes, the host address width is 46. MGAW reports 57 (56+1), and highest
sagaw bit is for 48.


> For this failure case, domain_init() just wants to find a suitable agaw
> for the private domain. I think it makes sense to check against
> iommu->agaw instead of cap_mgaw.
>
> Best regards,
> baolu
>
>> 
>>>
>>>> other places like prepare_domain_attach_device, the dmar domain agaw
>>>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>>>> determined based off what is reported for sagaw. I'm wondering if it
>>>> can't instead do:
>>>> ---
>>>>    drivers/iommu/intel-iommu.c | 4 ++--
>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>> diff --git a/drivers/iommu/intel-iommu.c
>>>> b/drivers/iommu/intel-iommu.c
>>>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>>>> --- a/drivers/iommu/intel-iommu.c
>>>> +++ b/drivers/iommu/intel-iommu.c
>>>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>>>    	domain_reserve_special_ranges(domain);
>>>>    	/* calculate AGAW */
>>>> -	if (guest_width > cap_mgaw(iommu->cap))
>>>> -	        guest_width = cap_mgaw(iommu->cap);
>>>> +	if (guest_width > agaw_to_width(iommu->agaw))
>>>> +	        guest_width = agaw_to_width(iommu->agaw);
>>>>    	domain->gaw = guest_width;
>>>>    	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>>>    	agaw = width_to_agaw(adjust_width);
>>>> --
>>>> 2.27.0
>>>>
>>>> Thoughts? With the former code the ehci device for the ilo fails when
>>>> trying to get a private domain.
>>>> Thanks,
>>>> Jerry
>>>>
>> 

Re: Question about domain_init (v5.3-v5.7)

[Jerry Snitselaar]

Lu Baolu @ 2020-11-26 19:12 MST:

> Hi Jerry,
>
> On 11/27/20 5:35 AM, Jerry Snitselaar wrote:
>> Lu Baolu @ 2020-11-26 04:01 MST:
>> 
>>> Hi Jerry,
>>>
>>> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>>>> Is there a reason we check the requested guest address width against
>>>> the
>>>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>>>> I've run into a case with a new system where the mgaw reported is 57,
>>>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>>>> the highest supported agaw is 48 and the domain_init code fails here. In
>>>
>>> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
>>> maybe have to add a platform specific quirk to set mgaw to the highest
>>> supported agaw?
>>>
>>> Best regards,
>>> baolu
>> Is there somewhere you can point me to that discusses how they
>> should be
>> setting the mgaw? I misunderstood when I previously asked you about
>> whether the mgaw could be a value that was greater than any of sagaw.
>> If it is a bios issue, then they should fix it there.
>
> MGAW indicates the max gpa width supported by 2nd translation. The VT-d
> spec requires that this value must be at least equal to the host
> physical addressibility. According to this, BIOS is good, right?
>
> For this failure case, domain_init() just wants to find a suitable agaw
> for the private domain. I think it makes sense to check against
> iommu->agaw instead of cap_mgaw.
>
> Best regards,
> baolu
>

From this bit in the spec about MGAW:

    Guest addressability for a given DMA request is limited to the
    minimum of the value reported through this field and the adjusted
    guest address width of the corresponding page-table structure.
    (Adjusted guest address widths supported by hardware are reported
    through the SAGAW field).

That does suggest it should be adjusted down to the sagaw value in this case, yes?
Just want to make sure I'm understanding it correctly.

>> 
>>>
>>>> other places like prepare_domain_attach_device, the dmar domain agaw
>>>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>>>> determined based off what is reported for sagaw. I'm wondering if it
>>>> can't instead do:
>>>> ---
>>>>    drivers/iommu/intel-iommu.c | 4 ++--
>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>> diff --git a/drivers/iommu/intel-iommu.c
>>>> b/drivers/iommu/intel-iommu.c
>>>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>>>> --- a/drivers/iommu/intel-iommu.c
>>>> +++ b/drivers/iommu/intel-iommu.c
>>>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>>>    	domain_reserve_special_ranges(domain);
>>>>    	/* calculate AGAW */
>>>> -	if (guest_width > cap_mgaw(iommu->cap))
>>>> -	        guest_width = cap_mgaw(iommu->cap);
>>>> +	if (guest_width > agaw_to_width(iommu->agaw))
>>>> +	        guest_width = agaw_to_width(iommu->agaw);
>>>>    	domain->gaw = guest_width;
>>>>    	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>>>    	agaw = width_to_agaw(adjust_width);
>>>> --
>>>> 2.27.0
>>>>
>>>> Thoughts? With the former code the ehci device for the ilo fails when
>>>> trying to get a private domain.
>>>> Thanks,
>>>> Jerry
>>>>
>> 

Re: Question about domain_init (v5.3-v5.7)

[Jerry Snitselaar]

Jerry Snitselaar @ 2020-11-30 10:50 MST:

> Lu Baolu @ 2020-11-26 19:12 MST:
>
>> Hi Jerry,
>>
>> On 11/27/20 5:35 AM, Jerry Snitselaar wrote:
>>> Lu Baolu @ 2020-11-26 04:01 MST:
>>> 
>>>> Hi Jerry,
>>>>
>>>> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>>>>> Is there a reason we check the requested guest address width against
>>>>> the
>>>>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>>>>> I've run into a case with a new system where the mgaw reported is 57,
>>>>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>>>>> the highest supported agaw is 48 and the domain_init code fails here. In
>>>>
>>>> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
>>>> maybe have to add a platform specific quirk to set mgaw to the highest
>>>> supported agaw?
>>>>
>>>> Best regards,
>>>> baolu
>>> Is there somewhere you can point me to that discusses how they
>>> should be
>>> setting the mgaw? I misunderstood when I previously asked you about
>>> whether the mgaw could be a value that was greater than any of sagaw.
>>> If it is a bios issue, then they should fix it there.
>>
>> MGAW indicates the max gpa width supported by 2nd translation. The VT-d
>> spec requires that this value must be at least equal to the host
>> physical addressibility. According to this, BIOS is good, right?
>>
>> For this failure case, domain_init() just wants to find a suitable agaw
>> for the private domain. I think it makes sense to check against
>> iommu->agaw instead of cap_mgaw.
>>
>> Best regards,
>> baolu
>>
>
> From this bit in the spec about MGAW:
>
>     Guest addressability for a given DMA request is limited to the
>     minimum of the value reported through this field and the adjusted
>     guest address width of the corresponding page-table structure.
>     (Adjusted guest address widths supported by hardware are reported
>     through the SAGAW field).
>
> That does suggest it should be adjusted down to the sagaw value in this case, yes?
> Just want to make sure I'm understanding it correctly.

Or I guess that is really talking about if you had an mgaw lower than the the
sagaw, the dma request would be limited to that lower mgaw value.

>
>>> 
>>>>
>>>>> other places like prepare_domain_attach_device, the dmar domain agaw
>>>>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>>>>> determined based off what is reported for sagaw. I'm wondering if it
>>>>> can't instead do:
>>>>> ---
>>>>>    drivers/iommu/intel-iommu.c | 4 ++--
>>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>> diff --git a/drivers/iommu/intel-iommu.c
>>>>> b/drivers/iommu/intel-iommu.c
>>>>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>>>>> --- a/drivers/iommu/intel-iommu.c
>>>>> +++ b/drivers/iommu/intel-iommu.c
>>>>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>>>>    	domain_reserve_special_ranges(domain);
>>>>>    	/* calculate AGAW */
>>>>> -	if (guest_width > cap_mgaw(iommu->cap))
>>>>> -	        guest_width = cap_mgaw(iommu->cap);
>>>>> +	if (guest_width > agaw_to_width(iommu->agaw))
>>>>> +	        guest_width = agaw_to_width(iommu->agaw);
>>>>>    	domain->gaw = guest_width;
>>>>>    	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>>>>    	agaw = width_to_agaw(adjust_width);
>>>>> --
>>>>> 2.27.0
>>>>>
>>>>> Thoughts? With the former code the ehci device for the ilo fails when
>>>>> trying to get a private domain.
>>>>> Thanks,
>>>>> Jerry
>>>>>
>>> 

Re: Question about domain_init (v5.3-v5.7)

[Lu Baolu]

Hi Jerry,

On 12/1/20 1:50 AM, Jerry Snitselaar wrote:
> 
> Lu Baolu @ 2020-11-26 19:12 MST:
> 
>> Hi Jerry,
>>
>> On 11/27/20 5:35 AM, Jerry Snitselaar wrote:
>>> Lu Baolu @ 2020-11-26 04:01 MST:
>>>
>>>> Hi Jerry,
>>>>
>>>> On 2020/11/26 4:27, Jerry Snitselaar wrote:
>>>>> Is there a reason we check the requested guest address width against
>>>>> the
>>>>> iommu's mgaw, instead of the agaw that we already know for the iommu?
>>>>> I've run into a case with a new system where the mgaw reported is 57,
>>>>> but if they set PAE to 46 instead of 52 in the bios, then sagaw reports
>>>>> the highest supported agaw is 48 and the domain_init code fails here. In
>>>>
>>>> Isn't this a platform bug? If it's too late to fix it in the BIOS, you
>>>> maybe have to add a platform specific quirk to set mgaw to the highest
>>>> supported agaw?
>>>>
>>>> Best regards,
>>>> baolu
>>> Is there somewhere you can point me to that discusses how they
>>> should be
>>> setting the mgaw? I misunderstood when I previously asked you about
>>> whether the mgaw could be a value that was greater than any of sagaw.
>>> If it is a bios issue, then they should fix it there.
>>
>> MGAW indicates the max gpa width supported by 2nd translation. The VT-d
>> spec requires that this value must be at least equal to the host
>> physical addressibility. According to this, BIOS is good, right?
>>
>> For this failure case, domain_init() just wants to find a suitable agaw
>> for the private domain. I think it makes sense to check against
>> iommu->agaw instead of cap_mgaw.
>>
>> Best regards,
>> baolu
>>
> 
>  From this bit in the spec about MGAW:
> 
>      Guest addressability for a given DMA request is limited to the
>      minimum of the value reported through this field and the adjusted
>      guest address width of the corresponding page-table structure.
>      (Adjusted guest address widths supported by hardware are reported
>      through the SAGAW field).
> 
> That does suggest it should be adjusted down to the sagaw value in this case, yes?
> Just want to make sure I'm understanding it correctly.

Yes. I think so.

Best regards,
baolu

> 
>>>
>>>>
>>>>> other places like prepare_domain_attach_device, the dmar domain agaw
>>>>> gets adjusted down to the iommu agaw. The agaw of the iommu gets
>>>>> determined based off what is reported for sagaw. I'm wondering if it
>>>>> can't instead do:
>>>>> ---
>>>>>     drivers/iommu/intel-iommu.c | 4 ++--
>>>>>     1 file changed, 2 insertions(+), 2 deletions(-)
>>>>> diff --git a/drivers/iommu/intel-iommu.c
>>>>> b/drivers/iommu/intel-iommu.c
>>>>> index 6ca5c92ef2e5..a8e41ec36d9e 100644
>>>>> --- a/drivers/iommu/intel-iommu.c
>>>>> +++ b/drivers/iommu/intel-iommu.c
>>>>> @@ -1862,8 +1862,8 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu,
>>>>>     	domain_reserve_special_ranges(domain);
>>>>>     	/* calculate AGAW */
>>>>> -	if (guest_width > cap_mgaw(iommu->cap))
>>>>> -	        guest_width = cap_mgaw(iommu->cap);
>>>>> +	if (guest_width > agaw_to_width(iommu->agaw))
>>>>> +	        guest_width = agaw_to_width(iommu->agaw);
>>>>>     	domain->gaw = guest_width;
>>>>>     	adjust_width = guestwidth_to_adjustwidth(guest_width);
>>>>>     	agaw = width_to_agaw(adjust_width);
>>>>> --
>>>>> 2.27.0
>>>>>
>>>>> Thoughts? With the former code the ehci device for the ilo fails when
>>>>> trying to get a private domain.
>>>>> Thanks,
>>>>> Jerry
>>>>>
>>>
>