Re: [PATCH v4 7/9] powerpc/pseries/vas: Reopen windows with DLPAR core add

[Nicholas Piggin <npiggin@gmail.com>]

Excerpts from Haren Myneni's message of February 20, 2022 6:01 am:
> 
> VAS windows can be closed in the hypervisor due to lost credits
> when the core is removed and the kernel gets fault for NX
> requests on these in-active windows. If these credits are
> available later for core add, reopen these windows and set them
> active. When the OS sees page faults on these active windows,
> it creates mapping on the new paste address. Then the user space
> can continue to use these windows and send HW compression
> requests to NX successfully.

Just for my own ignorance, what happens if userspace does not get
another page fault on that window? Presumably when it gets a page
fault it changes to an available window and doesn't just keep
re-trying. So in what situation does it attempt to re-access a
faulting window?

Thanks,
Nick

> 
> Signed-off-by: Haren Myneni <haren@linux.ibm.com>
> ---
>  arch/powerpc/platforms/pseries/vas.c | 91 +++++++++++++++++++++++++++-
>  1 file changed, 90 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c
> index a297720bcdae..96178dd58adf 100644
> --- a/arch/powerpc/platforms/pseries/vas.c
> +++ b/arch/powerpc/platforms/pseries/vas.c
> @@ -565,6 +565,88 @@ static int __init get_vas_capabilities(u8 feat, enum vas_cop_feat_type type,
>  	return 0;
>  }
>  
> +/*
> + * VAS windows can be closed due to lost credits when the core is
> + * removed. So reopen them if credits are available due to DLPAR
> + * core add and set the window active status. When NX sees the page
> + * fault on the unmapped paste address, the kernel handles the fault
> + * by setting the remapping to new paste address if the window is
> + * active.
> + */
> +static int reconfig_open_windows(struct vas_caps *vcaps, int creds)
> +{
> +	long domain[PLPAR_HCALL9_BUFSIZE] = {VAS_DEFAULT_DOMAIN_ID};
> +	struct vas_cop_feat_caps *caps = &vcaps->caps;
> +	struct pseries_vas_window *win = NULL, *tmp;
> +	int rc, mv_ents = 0;
> +
> +	/*
> +	 * Nothing to do if there are no closed windows.
> +	 */
> +	if (!vcaps->nr_close_wins)
> +		return 0;
> +
> +	/*
> +	 * For the core removal, the hypervisor reduces the credits
> +	 * assigned to the LPAR and the kernel closes VAS windows
> +	 * in the hypervisor depends on reduced credits. The kernel
> +	 * uses LIFO (the last windows that are opened will be closed
> +	 * first) and expects to open in the same order when credits
> +	 * are available.
> +	 * For example, 40 windows are closed when the LPAR lost 2 cores
> +	 * (dedicated). If 1 core is added, this LPAR can have 20 more
> +	 * credits. It means the kernel can reopen 20 windows. So move
> +	 * 20 entries in the VAS windows lost and reopen next 20 windows.
> +	 */
> +	if (vcaps->nr_close_wins > creds)
> +		mv_ents = vcaps->nr_close_wins - creds;
> +
> +	list_for_each_entry_safe(win, tmp, &vcaps->list, win_list) {
> +		if (!mv_ents)
> +			break;
> +
> +		mv_ents--;
> +	}
> +
> +	list_for_each_entry_safe_from(win, tmp, &vcaps->list, win_list) {
> +		/*
> +		 * Nothing to do on this window if it is not closed
> +		 * with VAS_WIN_NO_CRED_CLOSE
> +		 */
> +		if (!(win->vas_win.status & VAS_WIN_NO_CRED_CLOSE))
> +			continue;
> +
> +		rc = allocate_setup_window(win, (u64 *)&domain[0],
> +					   caps->win_type);
> +		if (rc)
> +			return rc;
> +
> +		rc = h_modify_vas_window(win);
> +		if (rc)
> +			goto out;
> +
> +		mutex_lock(&win->vas_win.task_ref.mmap_mutex);
> +		/*
> +		 * Set window status to active
> +		 */
> +		win->vas_win.status &= ~VAS_WIN_NO_CRED_CLOSE;
> +		mutex_unlock(&win->vas_win.task_ref.mmap_mutex);
> +		win->win_type = caps->win_type;
> +		if (!--vcaps->nr_close_wins)
> +			break;
> +	}
> +
> +	return 0;
> +out:
> +	/*
> +	 * Window modify HCALL failed. So close the window to the
> +	 * hypervisor and return.
> +	 */
> +	free_irq_setup(win);
> +	h_deallocate_vas_window(win->vas_win.winid);
> +	return rc;
> +}
> +
>  /*
>   * The hypervisor reduces the available credits if the LPAR lost core. It
>   * means the excessive windows should not be active and the user space
> @@ -673,7 +755,14 @@ static int vas_reconfig_capabilties(u8 type)
>  	 * closed / reopened. Hold the vas_pseries_mutex so that the
>  	 * the user space can not open new windows.
>  	 */
> -	if (old_nr_creds >  new_nr_creds) {
> +	if (old_nr_creds <  new_nr_creds) {
> +		/*
> +		 * If the existing target credits is less than the new
> +		 * target, reopen windows if they are closed due to
> +		 * the previous DLPAR (core removal).
> +		 */
> +		rc = reconfig_open_windows(vcaps, new_nr_creds - old_nr_creds);
> +	} else {
>  		/*
>  		 * # active windows is more than new LPAR available
>  		 * credits. So close the excessive windows.
> -- 
> 2.27.0
> 
> 
>