[PATCH] tty: vt: Add checks after calling kzalloc

[PATCH] tty: vt: Add checks after calling kzalloc

[Jiasheng Jiang]

As the potential failure of the memory allocation,
it should be better to check the return value after
calling kzalloc and return error if fails.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Signed-off-by: Jiasheng Jiang <jiasheng@iscas.ac.cn>
---
 drivers/tty/vt/vt.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index ae9c926acd6f..a4d59f3a3ce0 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -3519,11 +3519,19 @@ static int __init con_init(void)
 
 	for (currcons = 0; currcons < MIN_NR_CONSOLES; currcons++) {
 		vc_cons[currcons].d = vc = kzalloc(sizeof(struct vc_data), GFP_NOWAIT);
+		if (!vc) {
+			console_unlock();
+			return -ENOMEM;
+		}
 		INIT_WORK(&vc_cons[currcons].SAK_work, vc_SAK);
 		tty_port_init(&vc->port);
 		visual_init(vc, currcons, 1);
 		/* Assuming vc->vc_{cols,rows,screenbuf_size} are sane here. */
 		vc->vc_screenbuf = kzalloc(vc->vc_screenbuf_size, GFP_NOWAIT);
+		if (!vc->vc_screenbuf) {
+			console_unlock();
+			return -ENOMEM;
+		}
 		vc_init(vc, vc->vc_rows, vc->vc_cols,
 			currcons || !vc->vc_sw->con_save_screen);
 	}
-- 
2.25.1

Re: [PATCH] tty: vt: Add checks after calling kzalloc

[Greg KH]

On Wed, Aug 31, 2022 at 03:57:42PM +0800, Jiasheng Jiang wrote:
> As the potential failure of the memory allocation,
> it should be better to check the return value after
> calling kzalloc and return error if fails.
> 
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Signed-off-by: Jiasheng Jiang <jiasheng@iscas.ac.cn>
> ---
>  drivers/tty/vt/vt.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
> index ae9c926acd6f..a4d59f3a3ce0 100644
> --- a/drivers/tty/vt/vt.c
> +++ b/drivers/tty/vt/vt.c
> @@ -3519,11 +3519,19 @@ static int __init con_init(void)
>  
>  	for (currcons = 0; currcons < MIN_NR_CONSOLES; currcons++) {
>  		vc_cons[currcons].d = vc = kzalloc(sizeof(struct vc_data), GFP_NOWAIT);
> +		if (!vc) {
> +			console_unlock();
> +			return -ENOMEM;
> +		}
>  		INIT_WORK(&vc_cons[currcons].SAK_work, vc_SAK);
>  		tty_port_init(&vc->port);
>  		visual_init(vc, currcons, 1);
>  		/* Assuming vc->vc_{cols,rows,screenbuf_size} are sane here. */
>  		vc->vc_screenbuf = kzalloc(vc->vc_screenbuf_size, GFP_NOWAIT);
> +		if (!vc->vc_screenbuf) {
> +			console_unlock();
> +			return -ENOMEM;
> +		}
>  		vc_init(vc, vc->vc_rows, vc->vc_cols,
>  			currcons || !vc->vc_sw->con_save_screen);
>  	}
> -- 
> 2.25.1
> 

This has been attempted many times in the past, sorry.  Unless you can
prove that this can actually happen in real life, we are going to leave
these as-is.

Please do not just do random changes like this without actually testing
to see if it is possible to happen.

thanks,

greg k-h

Re: [PATCH] tty: vt: Add checks after calling kzalloc

[Tetsuo Handa]

Please check if your system can survive, by doing

On 2022/08/31 18:08, Jiasheng Jiang wrote:
> On Wed, Aug 31, 2022 at 03:57:42PM +0800, Jiasheng Jiang wrote:
>>>  	for (currcons = 0; currcons < MIN_NR_CONSOLES; currcons++) {
>>>  		vc_cons[currcons].d = vc = kzalloc(sizeof(struct vc_data), GFP_NOWAIT);

-		vc_cons[currcons].d = vc = kzalloc(sizeof(struct vc_data), GFP_NOWAIT);
+		vc_cons[currcons].d = vc = NULL;

>>> +		if (!vc) {
>>> +			console_unlock();
>>> +			return -ENOMEM;
>>> +		}
>>>  		INIT_WORK(&vc_cons[currcons].SAK_work, vc_SAK);
>>>  		tty_port_init(&vc->port);
>>>  		visual_init(vc, currcons, 1);
>>>  		/* Assuming vc->vc_{cols,rows,screenbuf_size} are sane here. */
>>>  		vc->vc_screenbuf = kzalloc(vc->vc_screenbuf_size, GFP_NOWAIT);

and/or

-		vc->vc_screenbuf = kzalloc(vc->vc_screenbuf_size, GFP_NOWAIT);
+		vc->vc_screenbuf = NULL;

>>> +		if (!vc->vc_screenbuf) {
>>> +			console_unlock();
>>> +			return -ENOMEM;
>>> +		}

in addition to your patch.

This is __init function of built-in module, isn't it?
Who can recover from this allocation failure? ;-)