[PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
various runtime allocation failures. The series enables daily usage of devices
without exausting limited resources like CMA or DMM space if GPU rendering is
needed.

The first patch doesn't bring any functional changes, it just moves some
TILER/DMM related code to a separate function, to simplify the review of the
next two patches.

The second patch allows off-CPU rendering to non-scanout buffers. Without that
patch, it is basically impossible to use the driver allocated GEM buffers on
OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
CMA soon gets fragmented and we start seeing allocation failures. Such failres
in Xorg cannot be handeled gracefully, so the system is basically unusable.

Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
fragmented with time, leading to allocation failures.

Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
PVR EXA from https://github.com/maemo-leste/xf86-video-omap

Ivaylo Dimitrov (3):
  drm: omapdrm: simplify omap_gem_pin
  drm: omapdrm: Support exporting of non-contiguous GEM BOs
  drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

 drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
 drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
 drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
 3 files changed, 116 insertions(+), 90 deletions(-)

-- 
1.9.1

[PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
various runtime allocation failures. The series enables daily usage of devices
without exausting limited resources like CMA or DMM space if GPU rendering is
needed.

The first patch doesn't bring any functional changes, it just moves some
TILER/DMM related code to a separate function, to simplify the review of the
next two patches.

The second patch allows off-CPU rendering to non-scanout buffers. Without that
patch, it is basically impossible to use the driver allocated GEM buffers on
OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
CMA soon gets fragmented and we start seeing allocation failures. Such failres
in Xorg cannot be handeled gracefully, so the system is basically unusable.

Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
fragmented with time, leading to allocation failures.

Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
PVR EXA from https://github.com/maemo-leste/xf86-video-omap

Ivaylo Dimitrov (3):
  drm: omapdrm: simplify omap_gem_pin
  drm: omapdrm: Support exporting of non-contiguous GEM BOs
  drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

 drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
 drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
 drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
 3 files changed, 116 insertions(+), 90 deletions(-)

-- 
1.9.1

[PATCH 1/3] drm: omapdrm: simplify omap_gem_pin

[Ivaylo Dimitrov]

Move tiler related code to its own function.

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c | 75 +++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 33 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index b0fa174..bb12cb4 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -750,6 +750,46 @@ void omap_gem_dma_sync_buffer(struct drm_gem_object *obj,
 	}
 }
 
+static int omap_gem_pin_tiler(struct drm_gem_object *obj)
+{
+	struct omap_gem_object *omap_obj = to_omap_bo(obj);
+	u32 npages = obj->size >> PAGE_SHIFT;
+	enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
+	struct tiler_block *block;
+	int ret;
+
+	BUG_ON(omap_obj->block);
+
+	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
+		block = tiler_reserve_2d(fmt, omap_obj->width, omap_obj->height,
+					 PAGE_SIZE);
+	} else {
+		block = tiler_reserve_1d(obj->size);
+	}
+
+	if (IS_ERR(block)) {
+		ret = PTR_ERR(block);
+		dev_err(obj->dev->dev, "could not remap: %d (%d)\n", ret, fmt);
+		goto fail;
+	}
+
+	/* TODO: enable async refill.. */
+	ret = tiler_pin(block, omap_obj->pages, npages, omap_obj->roll, true);
+	if (ret) {
+		tiler_release(block);
+		dev_err(obj->dev->dev, "could not pin: %d\n", ret);
+		goto fail;
+	}
+
+	omap_obj->dma_addr = tiler_ssptr(block);
+	omap_obj->block = block;
+
+	DBG("got dma address: %pad", &omap_obj->dma_addr);
+
+fail:
+	return ret;
+}
+
 /**
  * omap_gem_pin() - Pin a GEM object in memory
  * @obj: the GEM object
@@ -774,11 +814,6 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 
 	if (!omap_gem_is_contiguous(omap_obj) && priv->has_dmm) {
 		if (refcount_read(&omap_obj->dma_addr_cnt) == 0) {
-			u32 npages = obj->size >> PAGE_SHIFT;
-			enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
-			struct tiler_block *block;
-
-			BUG_ON(omap_obj->block);
 
 			refcount_set(&omap_obj->dma_addr_cnt, 1);
 
@@ -786,35 +821,9 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 			if (ret)
 				goto fail;
 
-			if (omap_obj->flags & OMAP_BO_TILED_MASK) {
-				block = tiler_reserve_2d(fmt,
-						omap_obj->width,
-						omap_obj->height, PAGE_SIZE);
-			} else {
-				block = tiler_reserve_1d(obj->size);
-			}
-
-			if (IS_ERR(block)) {
-				ret = PTR_ERR(block);
-				dev_err(obj->dev->dev,
-					"could not remap: %d (%d)\n", ret, fmt);
-				goto fail;
-			}
-
-			/* TODO: enable async refill.. */
-			ret = tiler_pin(block, omap_obj->pages, npages,
-					omap_obj->roll, true);
-			if (ret) {
-				tiler_release(block);
-				dev_err(obj->dev->dev,
-						"could not pin: %d\n", ret);
+			ret = omap_gem_pin_tiler(obj);
+			if (ret)
 				goto fail;
-			}
-
-			omap_obj->dma_addr = tiler_ssptr(block);
-			omap_obj->block = block;
-
-			DBG("got dma address: %pad", &omap_obj->dma_addr);
 		} else {
 			refcount_inc(&omap_obj->dma_addr_cnt);
 		}
-- 
1.9.1

[PATCH 1/3] drm: omapdrm: simplify omap_gem_pin

[Ivaylo Dimitrov]

Move tiler related code to its own function.

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c | 75 +++++++++++++++++++++-----------------
 1 file changed, 42 insertions(+), 33 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index b0fa174..bb12cb4 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -750,6 +750,46 @@ void omap_gem_dma_sync_buffer(struct drm_gem_object *obj,
 	}
 }
 
+static int omap_gem_pin_tiler(struct drm_gem_object *obj)
+{
+	struct omap_gem_object *omap_obj = to_omap_bo(obj);
+	u32 npages = obj->size >> PAGE_SHIFT;
+	enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
+	struct tiler_block *block;
+	int ret;
+
+	BUG_ON(omap_obj->block);
+
+	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
+		block = tiler_reserve_2d(fmt, omap_obj->width, omap_obj->height,
+					 PAGE_SIZE);
+	} else {
+		block = tiler_reserve_1d(obj->size);
+	}
+
+	if (IS_ERR(block)) {
+		ret = PTR_ERR(block);
+		dev_err(obj->dev->dev, "could not remap: %d (%d)\n", ret, fmt);
+		goto fail;
+	}
+
+	/* TODO: enable async refill.. */
+	ret = tiler_pin(block, omap_obj->pages, npages, omap_obj->roll, true);
+	if (ret) {
+		tiler_release(block);
+		dev_err(obj->dev->dev, "could not pin: %d\n", ret);
+		goto fail;
+	}
+
+	omap_obj->dma_addr = tiler_ssptr(block);
+	omap_obj->block = block;
+
+	DBG("got dma address: %pad", &omap_obj->dma_addr);
+
+fail:
+	return ret;
+}
+
 /**
  * omap_gem_pin() - Pin a GEM object in memory
  * @obj: the GEM object
@@ -774,11 +814,6 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 
 	if (!omap_gem_is_contiguous(omap_obj) && priv->has_dmm) {
 		if (refcount_read(&omap_obj->dma_addr_cnt) == 0) {
-			u32 npages = obj->size >> PAGE_SHIFT;
-			enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
-			struct tiler_block *block;
-
-			BUG_ON(omap_obj->block);
 
 			refcount_set(&omap_obj->dma_addr_cnt, 1);
 
@@ -786,35 +821,9 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 			if (ret)
 				goto fail;
 
-			if (omap_obj->flags & OMAP_BO_TILED_MASK) {
-				block = tiler_reserve_2d(fmt,
-						omap_obj->width,
-						omap_obj->height, PAGE_SIZE);
-			} else {
-				block = tiler_reserve_1d(obj->size);
-			}
-
-			if (IS_ERR(block)) {
-				ret = PTR_ERR(block);
-				dev_err(obj->dev->dev,
-					"could not remap: %d (%d)\n", ret, fmt);
-				goto fail;
-			}
-
-			/* TODO: enable async refill.. */
-			ret = tiler_pin(block, omap_obj->pages, npages,
-					omap_obj->roll, true);
-			if (ret) {
-				tiler_release(block);
-				dev_err(obj->dev->dev,
-						"could not pin: %d\n", ret);
+			ret = omap_gem_pin_tiler(obj);
+			if (ret)
 				goto fail;
-			}
-
-			omap_obj->dma_addr = tiler_ssptr(block);
-			omap_obj->block = block;
-
-			DBG("got dma address: %pad", &omap_obj->dma_addr);
 		} else {
 			refcount_inc(&omap_obj->dma_addr_cnt);
 		}
-- 
1.9.1

[PATCH 2/3] drm: omapdrm: Support exporting of non-contiguous GEM BOs

[Ivaylo Dimitrov]

Currently code allocates non-scanout BOs from SHMEM and those objects are
accessible to userspace by mmap(). However, on devices with no DMM (like
OMAP3), the same objects are not accessible by kernel drivers that want to
render to them as code refuses to export them. In turn this means that on
devices with no DMM, all buffers must be allocated as scanout, otherwise
only CPU can access them. On those devices, scanout buffers are allocated
from CMA, making those allocations highly unreliable.

Fix that by implementing functionality to export SHMEM backed buffers on
devices with no DMM. This makes CMA memory only being used when needed,
instead for every buffer that has to be off-CPU rendered.

Tested on Motorola Droid4 and Nokia N900

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c        | 125 +++++++++++++++++-------------
 drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
 drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
 3 files changed, 73 insertions(+), 60 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index bb12cb4..41c1a6d 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -38,7 +38,7 @@ struct omap_gem_object {
 	/** roll applied when mapping to DMM */
 	u32 roll;
 
-	/** protects dma_addr_cnt, block, pages, dma_addrs and vaddr */
+	/** protects pin_cnt, block, pages, dma_addrs and vaddr */
 	struct mutex lock;
 
 	/**
@@ -50,24 +50,24 @@ struct omap_gem_object {
 	 * - buffers imported from dmabuf (with the OMAP_BO_MEM_DMABUF flag set)
 	 *   if they are physically contiguous (when sgt->orig_nents == 1)
 	 *
-	 * - buffers mapped through the TILER when dma_addr_cnt is not zero, in
-	 *   which case the DMA address points to the TILER aperture
+	 * - buffers mapped through the TILER when pin_cnt is not zero, in which
+	 *   case the DMA address points to the TILER aperture
 	 *
 	 * Physically contiguous buffers have their DMA address equal to the
 	 * physical address as we don't remap those buffers through the TILER.
 	 *
 	 * Buffers mapped to the TILER have their DMA address pointing to the
-	 * TILER aperture. As TILER mappings are refcounted (through
-	 * dma_addr_cnt) the DMA address must be accessed through omap_gem_pin()
-	 * to ensure that the mapping won't disappear unexpectedly. References
-	 * must be released with omap_gem_unpin().
+	 * TILER aperture. As TILER mappings are refcounted (through pin_cnt)
+	 * the DMA address must be accessed through omap_gem_pin() to ensure
+	 * that the mapping won't disappear unexpectedly. References must be
+	 * released with omap_gem_unpin().
 	 */
 	dma_addr_t dma_addr;
 
 	/**
-	 * # of users of dma_addr
+	 * # of users
 	 */
-	refcount_t dma_addr_cnt;
+	refcount_t pin_cnt;
 
 	/**
 	 * If the buffer has been imported from a dmabuf the OMAP_DB_DMABUF flag
@@ -812,32 +812,28 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 
 	mutex_lock(&omap_obj->lock);
 
-	if (!omap_gem_is_contiguous(omap_obj) && priv->has_dmm) {
-		if (refcount_read(&omap_obj->dma_addr_cnt) == 0) {
+	if (!omap_gem_is_contiguous(omap_obj)) {
+		if (refcount_read(&omap_obj->pin_cnt) == 0) {
 
-			refcount_set(&omap_obj->dma_addr_cnt, 1);
+			refcount_set(&omap_obj->pin_cnt, 1);
 
 			ret = omap_gem_attach_pages(obj);
 			if (ret)
 				goto fail;
 
-			ret = omap_gem_pin_tiler(obj);
-			if (ret)
-				goto fail;
+			if (priv->has_dmm) {
+				ret = omap_gem_pin_tiler(obj);
+				if (ret)
+					goto fail;
+			}
 		} else {
-			refcount_inc(&omap_obj->dma_addr_cnt);
+			refcount_inc(&omap_obj->pin_cnt);
 		}
-
-		if (dma_addr)
-			*dma_addr = omap_obj->dma_addr;
-	} else if (omap_gem_is_contiguous(omap_obj)) {
-		if (dma_addr)
-			*dma_addr = omap_obj->dma_addr;
-	} else {
-		ret = -EINVAL;
-		goto fail;
 	}
 
+	if (dma_addr)
+		*dma_addr = omap_obj->dma_addr;
+
 fail:
 	mutex_unlock(&omap_obj->lock);
 
@@ -856,27 +852,29 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
 	struct omap_gem_object *omap_obj = to_omap_bo(obj);
 	int ret;
 
-	if (omap_gem_is_contiguous(omap_obj) || !priv->has_dmm)
+	if (omap_gem_is_contiguous(omap_obj))
 		return;
 
-	if (refcount_dec_and_test(&omap_obj->dma_addr_cnt)) {
+	if (refcount_dec_and_test(&omap_obj->pin_cnt)) {
 		if (omap_obj->sgt) {
 			sg_free_table(omap_obj->sgt);
 			kfree(omap_obj->sgt);
 			omap_obj->sgt = NULL;
 		}
-		ret = tiler_unpin(omap_obj->block);
-		if (ret) {
-			dev_err(obj->dev->dev,
-				"could not unpin pages: %d\n", ret);
-		}
-		ret = tiler_release(omap_obj->block);
-		if (ret) {
-			dev_err(obj->dev->dev,
-				"could not release unmap: %d\n", ret);
+		if (priv->has_dmm) {
+			ret = tiler_unpin(omap_obj->block);
+			if (ret) {
+				dev_err(obj->dev->dev,
+					"could not unpin pages: %d\n", ret);
+			}
+			ret = tiler_release(omap_obj->block);
+			if (ret) {
+				dev_err(obj->dev->dev,
+					"could not release unmap: %d\n", ret);
+			}
+			omap_obj->dma_addr = 0;
+			omap_obj->block = NULL;
 		}
-		omap_obj->dma_addr = 0;
-		omap_obj->block = NULL;
 	}
 }
 
@@ -909,7 +907,7 @@ int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, u32 orient,
 
 	mutex_lock(&omap_obj->lock);
 
-	if ((refcount_read(&omap_obj->dma_addr_cnt) > 0) && omap_obj->block &&
+	if ((refcount_read(&omap_obj->pin_cnt) > 0) && omap_obj->block &&
 			(omap_obj->flags & OMAP_BO_TILED_MASK)) {
 		*dma_addr = tiler_tsptr(omap_obj->block, orient, x, y);
 		ret = 0;
@@ -977,7 +975,8 @@ int omap_gem_put_pages(struct drm_gem_object *obj)
 	return 0;
 }
 
-struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj)
+struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj,
+		enum dma_data_direction dir)
 {
 	struct omap_gem_object *omap_obj = to_omap_bo(obj);
 	dma_addr_t addr;
@@ -1002,28 +1001,44 @@ struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj)
 		goto err_unpin;
 	}
 
-	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
-		enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
+	if (addr) {
+		if (omap_obj->flags & OMAP_BO_TILED_MASK) {
+			enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
 
-		len = omap_obj->width << (int)fmt;
-		count = omap_obj->height;
-		stride = tiler_stride(fmt, 0);
+			len = omap_obj->width << (int)fmt;
+			count = omap_obj->height;
+			stride = tiler_stride(fmt, 0);
+		} else {
+			len = obj->size;
+			count = 1;
+			stride = 0;
+		}
 	} else {
-		len = obj->size;
-		count = 1;
-		stride = 0;
+		count = obj->size >> PAGE_SHIFT;
 	}
 
 	ret = sg_alloc_table(sgt, count, GFP_KERNEL);
 	if (ret)
 		goto err_free;
 
-	for_each_sg(sgt->sgl, sg, count, i) {
-		sg_set_page(sg, phys_to_page(addr), len, offset_in_page(addr));
-		sg_dma_address(sg) = addr;
-		sg_dma_len(sg) = len;
+	/* this must be after omap_gem_pin() to ensure we have pages attached */
+	omap_gem_dma_sync_buffer(obj, dir);
+
+	if (addr) {
+		for_each_sg(sgt->sgl, sg, count, i) {
+			sg_set_page(sg, phys_to_page(addr), len,
+				offset_in_page(addr));
+			sg_dma_address(sg) = addr;
+			sg_dma_len(sg) = len;
 
-		addr += stride;
+			addr += stride;
+		}
+	} else {
+		for_each_sg(sgt->sgl, sg, count, i) {
+			sg_set_page(sg, omap_obj->pages[i], PAGE_SIZE, 0);
+			sg_dma_address(sg) = omap_obj->dma_addrs[i];
+			sg_dma_len(sg) =  PAGE_SIZE;
+		}
 	}
 
 	omap_obj->sgt = sgt;
@@ -1133,7 +1148,7 @@ void omap_gem_describe(struct drm_gem_object *obj, struct seq_file *m)
 	seq_printf(m, "%08x: %2d (%2d) %08llx %pad (%2d) %p %4d",
 			omap_obj->flags, obj->name, kref_read(&obj->refcount),
 			off, &omap_obj->dma_addr,
-			refcount_read(&omap_obj->dma_addr_cnt),
+			refcount_read(&omap_obj->pin_cnt),
 			omap_obj->vaddr, omap_obj->roll);
 
 	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
@@ -1196,7 +1211,7 @@ static void omap_gem_free_object(struct drm_gem_object *obj)
 	mutex_lock(&omap_obj->lock);
 
 	/* The object should not be pinned. */
-	WARN_ON(refcount_read(&omap_obj->dma_addr_cnt) > 0);
+	WARN_ON(refcount_read(&omap_obj->pin_cnt) > 0);
 
 	if (omap_obj->pages) {
 		if (omap_obj->flags & OMAP_BO_MEM_DMABUF)
diff --git a/drivers/gpu/drm/omapdrm/omap_gem.h b/drivers/gpu/drm/omapdrm/omap_gem.h
index 19209e3..4d44889 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.h
+++ b/drivers/gpu/drm/omapdrm/omap_gem.h
@@ -82,7 +82,8 @@ int omap_gem_get_pages(struct drm_gem_object *obj, struct page ***pages,
 int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, u32 orient,
 		int x, int y, dma_addr_t *dma_addr);
 int omap_gem_tiled_stride(struct drm_gem_object *obj, u32 orient);
-struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj);
+struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj,
+		enum dma_data_direction dir);
 void omap_gem_put_sg(struct drm_gem_object *obj, struct sg_table *sgt);
 
 #endif /* __OMAPDRM_GEM_H__ */
diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
index 57af3d9..8d15e07 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
@@ -23,13 +23,10 @@ static struct sg_table *omap_gem_map_dma_buf(
 {
 	struct drm_gem_object *obj = attachment->dmabuf->priv;
 	struct sg_table *sg;
-	sg = omap_gem_get_sg(obj);
+	sg = omap_gem_get_sg(obj, dir);
 	if (IS_ERR(sg))
 		return sg;
 
-	/* this must be after omap_gem_pin() to ensure we have pages attached */
-	omap_gem_dma_sync_buffer(obj, dir);
-
 	return sg;
 }
 
-- 
1.9.1

[PATCH 2/3] drm: omapdrm: Support exporting of non-contiguous GEM BOs

[Ivaylo Dimitrov]

Currently code allocates non-scanout BOs from SHMEM and those objects are
accessible to userspace by mmap(). However, on devices with no DMM (like
OMAP3), the same objects are not accessible by kernel drivers that want to
render to them as code refuses to export them. In turn this means that on
devices with no DMM, all buffers must be allocated as scanout, otherwise
only CPU can access them. On those devices, scanout buffers are allocated
from CMA, making those allocations highly unreliable.

Fix that by implementing functionality to export SHMEM backed buffers on
devices with no DMM. This makes CMA memory only being used when needed,
instead for every buffer that has to be off-CPU rendered.

Tested on Motorola Droid4 and Nokia N900

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c        | 125 +++++++++++++++++-------------
 drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
 drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
 3 files changed, 73 insertions(+), 60 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index bb12cb4..41c1a6d 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -38,7 +38,7 @@ struct omap_gem_object {
 	/** roll applied when mapping to DMM */
 	u32 roll;
 
-	/** protects dma_addr_cnt, block, pages, dma_addrs and vaddr */
+	/** protects pin_cnt, block, pages, dma_addrs and vaddr */
 	struct mutex lock;
 
 	/**
@@ -50,24 +50,24 @@ struct omap_gem_object {
 	 * - buffers imported from dmabuf (with the OMAP_BO_MEM_DMABUF flag set)
 	 *   if they are physically contiguous (when sgt->orig_nents == 1)
 	 *
-	 * - buffers mapped through the TILER when dma_addr_cnt is not zero, in
-	 *   which case the DMA address points to the TILER aperture
+	 * - buffers mapped through the TILER when pin_cnt is not zero, in which
+	 *   case the DMA address points to the TILER aperture
 	 *
 	 * Physically contiguous buffers have their DMA address equal to the
 	 * physical address as we don't remap those buffers through the TILER.
 	 *
 	 * Buffers mapped to the TILER have their DMA address pointing to the
-	 * TILER aperture. As TILER mappings are refcounted (through
-	 * dma_addr_cnt) the DMA address must be accessed through omap_gem_pin()
-	 * to ensure that the mapping won't disappear unexpectedly. References
-	 * must be released with omap_gem_unpin().
+	 * TILER aperture. As TILER mappings are refcounted (through pin_cnt)
+	 * the DMA address must be accessed through omap_gem_pin() to ensure
+	 * that the mapping won't disappear unexpectedly. References must be
+	 * released with omap_gem_unpin().
 	 */
 	dma_addr_t dma_addr;
 
 	/**
-	 * # of users of dma_addr
+	 * # of users
 	 */
-	refcount_t dma_addr_cnt;
+	refcount_t pin_cnt;
 
 	/**
 	 * If the buffer has been imported from a dmabuf the OMAP_DB_DMABUF flag
@@ -812,32 +812,28 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 
 	mutex_lock(&omap_obj->lock);
 
-	if (!omap_gem_is_contiguous(omap_obj) && priv->has_dmm) {
-		if (refcount_read(&omap_obj->dma_addr_cnt) == 0) {
+	if (!omap_gem_is_contiguous(omap_obj)) {
+		if (refcount_read(&omap_obj->pin_cnt) == 0) {
 
-			refcount_set(&omap_obj->dma_addr_cnt, 1);
+			refcount_set(&omap_obj->pin_cnt, 1);
 
 			ret = omap_gem_attach_pages(obj);
 			if (ret)
 				goto fail;
 
-			ret = omap_gem_pin_tiler(obj);
-			if (ret)
-				goto fail;
+			if (priv->has_dmm) {
+				ret = omap_gem_pin_tiler(obj);
+				if (ret)
+					goto fail;
+			}
 		} else {
-			refcount_inc(&omap_obj->dma_addr_cnt);
+			refcount_inc(&omap_obj->pin_cnt);
 		}
-
-		if (dma_addr)
-			*dma_addr = omap_obj->dma_addr;
-	} else if (omap_gem_is_contiguous(omap_obj)) {
-		if (dma_addr)
-			*dma_addr = omap_obj->dma_addr;
-	} else {
-		ret = -EINVAL;
-		goto fail;
 	}
 
+	if (dma_addr)
+		*dma_addr = omap_obj->dma_addr;
+
 fail:
 	mutex_unlock(&omap_obj->lock);
 
@@ -856,27 +852,29 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
 	struct omap_gem_object *omap_obj = to_omap_bo(obj);
 	int ret;
 
-	if (omap_gem_is_contiguous(omap_obj) || !priv->has_dmm)
+	if (omap_gem_is_contiguous(omap_obj))
 		return;
 
-	if (refcount_dec_and_test(&omap_obj->dma_addr_cnt)) {
+	if (refcount_dec_and_test(&omap_obj->pin_cnt)) {
 		if (omap_obj->sgt) {
 			sg_free_table(omap_obj->sgt);
 			kfree(omap_obj->sgt);
 			omap_obj->sgt = NULL;
 		}
-		ret = tiler_unpin(omap_obj->block);
-		if (ret) {
-			dev_err(obj->dev->dev,
-				"could not unpin pages: %d\n", ret);
-		}
-		ret = tiler_release(omap_obj->block);
-		if (ret) {
-			dev_err(obj->dev->dev,
-				"could not release unmap: %d\n", ret);
+		if (priv->has_dmm) {
+			ret = tiler_unpin(omap_obj->block);
+			if (ret) {
+				dev_err(obj->dev->dev,
+					"could not unpin pages: %d\n", ret);
+			}
+			ret = tiler_release(omap_obj->block);
+			if (ret) {
+				dev_err(obj->dev->dev,
+					"could not release unmap: %d\n", ret);
+			}
+			omap_obj->dma_addr = 0;
+			omap_obj->block = NULL;
 		}
-		omap_obj->dma_addr = 0;
-		omap_obj->block = NULL;
 	}
 }
 
@@ -909,7 +907,7 @@ int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, u32 orient,
 
 	mutex_lock(&omap_obj->lock);
 
-	if ((refcount_read(&omap_obj->dma_addr_cnt) > 0) && omap_obj->block &&
+	if ((refcount_read(&omap_obj->pin_cnt) > 0) && omap_obj->block &&
 			(omap_obj->flags & OMAP_BO_TILED_MASK)) {
 		*dma_addr = tiler_tsptr(omap_obj->block, orient, x, y);
 		ret = 0;
@@ -977,7 +975,8 @@ int omap_gem_put_pages(struct drm_gem_object *obj)
 	return 0;
 }
 
-struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj)
+struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj,
+		enum dma_data_direction dir)
 {
 	struct omap_gem_object *omap_obj = to_omap_bo(obj);
 	dma_addr_t addr;
@@ -1002,28 +1001,44 @@ struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj)
 		goto err_unpin;
 	}
 
-	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
-		enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
+	if (addr) {
+		if (omap_obj->flags & OMAP_BO_TILED_MASK) {
+			enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
 
-		len = omap_obj->width << (int)fmt;
-		count = omap_obj->height;
-		stride = tiler_stride(fmt, 0);
+			len = omap_obj->width << (int)fmt;
+			count = omap_obj->height;
+			stride = tiler_stride(fmt, 0);
+		} else {
+			len = obj->size;
+			count = 1;
+			stride = 0;
+		}
 	} else {
-		len = obj->size;
-		count = 1;
-		stride = 0;
+		count = obj->size >> PAGE_SHIFT;
 	}
 
 	ret = sg_alloc_table(sgt, count, GFP_KERNEL);
 	if (ret)
 		goto err_free;
 
-	for_each_sg(sgt->sgl, sg, count, i) {
-		sg_set_page(sg, phys_to_page(addr), len, offset_in_page(addr));
-		sg_dma_address(sg) = addr;
-		sg_dma_len(sg) = len;
+	/* this must be after omap_gem_pin() to ensure we have pages attached */
+	omap_gem_dma_sync_buffer(obj, dir);
+
+	if (addr) {
+		for_each_sg(sgt->sgl, sg, count, i) {
+			sg_set_page(sg, phys_to_page(addr), len,
+				offset_in_page(addr));
+			sg_dma_address(sg) = addr;
+			sg_dma_len(sg) = len;
 
-		addr += stride;
+			addr += stride;
+		}
+	} else {
+		for_each_sg(sgt->sgl, sg, count, i) {
+			sg_set_page(sg, omap_obj->pages[i], PAGE_SIZE, 0);
+			sg_dma_address(sg) = omap_obj->dma_addrs[i];
+			sg_dma_len(sg) =  PAGE_SIZE;
+		}
 	}
 
 	omap_obj->sgt = sgt;
@@ -1133,7 +1148,7 @@ void omap_gem_describe(struct drm_gem_object *obj, struct seq_file *m)
 	seq_printf(m, "%08x: %2d (%2d) %08llx %pad (%2d) %p %4d",
 			omap_obj->flags, obj->name, kref_read(&obj->refcount),
 			off, &omap_obj->dma_addr,
-			refcount_read(&omap_obj->dma_addr_cnt),
+			refcount_read(&omap_obj->pin_cnt),
 			omap_obj->vaddr, omap_obj->roll);
 
 	if (omap_obj->flags & OMAP_BO_TILED_MASK) {
@@ -1196,7 +1211,7 @@ static void omap_gem_free_object(struct drm_gem_object *obj)
 	mutex_lock(&omap_obj->lock);
 
 	/* The object should not be pinned. */
-	WARN_ON(refcount_read(&omap_obj->dma_addr_cnt) > 0);
+	WARN_ON(refcount_read(&omap_obj->pin_cnt) > 0);
 
 	if (omap_obj->pages) {
 		if (omap_obj->flags & OMAP_BO_MEM_DMABUF)
diff --git a/drivers/gpu/drm/omapdrm/omap_gem.h b/drivers/gpu/drm/omapdrm/omap_gem.h
index 19209e3..4d44889 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.h
+++ b/drivers/gpu/drm/omapdrm/omap_gem.h
@@ -82,7 +82,8 @@ int omap_gem_get_pages(struct drm_gem_object *obj, struct page ***pages,
 int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, u32 orient,
 		int x, int y, dma_addr_t *dma_addr);
 int omap_gem_tiled_stride(struct drm_gem_object *obj, u32 orient);
-struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj);
+struct sg_table *omap_gem_get_sg(struct drm_gem_object *obj,
+		enum dma_data_direction dir);
 void omap_gem_put_sg(struct drm_gem_object *obj, struct sg_table *sgt);
 
 #endif /* __OMAPDRM_GEM_H__ */
diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
index 57af3d9..8d15e07 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
@@ -23,13 +23,10 @@ static struct sg_table *omap_gem_map_dma_buf(
 {
 	struct drm_gem_object *obj = attachment->dmabuf->priv;
 	struct sg_table *sg;
-	sg = omap_gem_get_sg(obj);
+	sg = omap_gem_get_sg(obj, dir);
 	if (IS_ERR(sg))
 		return sg;
 
-	/* this must be after omap_gem_pin() to ensure we have pages attached */
-	omap_gem_dma_sync_buffer(obj, dir);
-
 	return sg;
 }
 
-- 
1.9.1

[PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On devices with DMM, all allocations are done through either DMM or TILER.
DMM/TILER being a limited resource means that such allocations will start
to fail before actual free memory is exhausted. What is even worse is that
with time DMM/TILER space gets fragmented to the point that even if we have
enough free DMM/TILER space and free memory, allocation fails because there
is no big enough free block in DMM/TILER space.

Such failures can be easily observed with OMAP xorg DDX, for example -
starting few GUI applications (so buffers for their windows are allocated)
and then rotating landscape<->portrait while closing and opening new
windows soon results in allocation failures.

Fix that by mapping buffers through DMM/TILER only when really needed,
like, for scanout buffers.

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index 41c1a6d..cf57179 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 			if (ret)
 				goto fail;
 
-			if (priv->has_dmm) {
-				ret = omap_gem_pin_tiler(obj);
-				if (ret)
-					goto fail;
+			if (omap_obj->flags & OMAP_BO_SCANOUT) {
+				if (priv->has_dmm) {
+					ret = omap_gem_pin_tiler(obj);
+					if (ret)
+						goto fail;
+				}
 			}
 		} else {
 			refcount_inc(&omap_obj->pin_cnt);
@@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
 			kfree(omap_obj->sgt);
 			omap_obj->sgt = NULL;
 		}
+		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
+			return;
 		if (priv->has_dmm) {
 			ret = tiler_unpin(omap_obj->block);
 			if (ret) {
-- 
1.9.1

[PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On devices with DMM, all allocations are done through either DMM or TILER.
DMM/TILER being a limited resource means that such allocations will start
to fail before actual free memory is exhausted. What is even worse is that
with time DMM/TILER space gets fragmented to the point that even if we have
enough free DMM/TILER space and free memory, allocation fails because there
is no big enough free block in DMM/TILER space.

Such failures can be easily observed with OMAP xorg DDX, for example -
starting few GUI applications (so buffers for their windows are allocated)
and then rotating landscape<->portrait while closing and opening new
windows soon results in allocation failures.

Fix that by mapping buffers through DMM/TILER only when really needed,
like, for scanout buffers.

Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
---
 drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index 41c1a6d..cf57179 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
 			if (ret)
 				goto fail;
 
-			if (priv->has_dmm) {
-				ret = omap_gem_pin_tiler(obj);
-				if (ret)
-					goto fail;
+			if (omap_obj->flags & OMAP_BO_SCANOUT) {
+				if (priv->has_dmm) {
+					ret = omap_gem_pin_tiler(obj);
+					if (ret)
+						goto fail;
+				}
 			}
 		} else {
 			refcount_inc(&omap_obj->pin_cnt);
@@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
 			kfree(omap_obj->sgt);
 			omap_obj->sgt = NULL;
 		}
+		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
+			return;
 		if (priv->has_dmm) {
 			ret = tiler_unpin(omap_obj->block);
 			if (ret) {
-- 
1.9.1

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tomi Valkeinen]

Hi,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> On devices with DMM, all allocations are done through either DMM or TILER.
> DMM/TILER being a limited resource means that such allocations will start
> to fail before actual free memory is exhausted. What is even worse is that
> with time DMM/TILER space gets fragmented to the point that even if we have
> enough free DMM/TILER space and free memory, allocation fails because there
> is no big enough free block in DMM/TILER space.
> 
> Such failures can be easily observed with OMAP xorg DDX, for example -
> starting few GUI applications (so buffers for their windows are allocated)
> and then rotating landscape<->portrait while closing and opening new
> windows soon results in allocation failures.
> 
> Fix that by mapping buffers through DMM/TILER only when really needed,
> like, for scanout buffers.

Doesn't this break users that get a buffer from omapdrm and expect it to 
be contiguous?

  Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tomi Valkeinen]

Hi,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> On devices with DMM, all allocations are done through either DMM or TILER.
> DMM/TILER being a limited resource means that such allocations will start
> to fail before actual free memory is exhausted. What is even worse is that
> with time DMM/TILER space gets fragmented to the point that even if we have
> enough free DMM/TILER space and free memory, allocation fails because there
> is no big enough free block in DMM/TILER space.
> 
> Such failures can be easily observed with OMAP xorg DDX, for example -
> starting few GUI applications (so buffers for their windows are allocated)
> and then rotating landscape<->portrait while closing and opening new
> windows soon results in allocation failures.
> 
> Fix that by mapping buffers through DMM/TILER only when really needed,
> like, for scanout buffers.

Doesn't this break users that get a buffer from omapdrm and expect it to 
be contiguous?

  Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> Hi,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> On devices with DMM, all allocations are done through either DMM or 
>> TILER.
>> DMM/TILER being a limited resource means that such allocations will start
>> to fail before actual free memory is exhausted. What is even worse is 
>> that
>> with time DMM/TILER space gets fragmented to the point that even if we 
>> have
>> enough free DMM/TILER space and free memory, allocation fails because 
>> there
>> is no big enough free block in DMM/TILER space.
>>
>> Such failures can be easily observed with OMAP xorg DDX, for example -
>> starting few GUI applications (so buffers for their windows are 
>> allocated)
>> and then rotating landscape<->portrait while closing and opening new
>> windows soon results in allocation failures.
>>
>> Fix that by mapping buffers through DMM/TILER only when really needed,
>> like, for scanout buffers.
> 
> Doesn't this break users that get a buffer from omapdrm and expect it to 
> be contiguous?
> 

If you mean dumb buffer, then no, this does not break users as dumb 
buffers are allocated as scanout:

https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603

If you mean omap_bo allocated buffers, then if users want 
linear(scanout) buffer, then they request it explicitly by passing 
OMAP_BO_SCANOUT.

Ivo

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> Hi,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> On devices with DMM, all allocations are done through either DMM or 
>> TILER.
>> DMM/TILER being a limited resource means that such allocations will start
>> to fail before actual free memory is exhausted. What is even worse is 
>> that
>> with time DMM/TILER space gets fragmented to the point that even if we 
>> have
>> enough free DMM/TILER space and free memory, allocation fails because 
>> there
>> is no big enough free block in DMM/TILER space.
>>
>> Such failures can be easily observed with OMAP xorg DDX, for example -
>> starting few GUI applications (so buffers for their windows are 
>> allocated)
>> and then rotating landscape<->portrait while closing and opening new
>> windows soon results in allocation failures.
>>
>> Fix that by mapping buffers through DMM/TILER only when really needed,
>> like, for scanout buffers.
> 
> Doesn't this break users that get a buffer from omapdrm and expect it to 
> be contiguous?
> 

If you mean dumb buffer, then no, this does not break users as dumb 
buffers are allocated as scanout:

https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603

If you mean omap_bo allocated buffers, then if users want 
linear(scanout) buffer, then they request it explicitly by passing 
OMAP_BO_SCANOUT.

Ivo

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo, Tomi

We have one X15 Android AOSP master build, it could not have the home
screen displayed
on the hdmi monitor connected with this change, with the following
message printed on the serial console
    [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
    [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
    [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
    [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3

   # for details, please check the link here: http://ix.io/47m1

It will work with home screen displayed on the hdmi monitor if this
change is reverted.

Is this the broken problem you talked about here?

And could you please give some suggestions on how to have the x15
Android build work with this change?

Thanks,
Yongqin Liu
On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
>
>
> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> > Hi,
> >
> > On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >> On devices with DMM, all allocations are done through either DMM or
> >> TILER.
> >> DMM/TILER being a limited resource means that such allocations will start
> >> to fail before actual free memory is exhausted. What is even worse is
> >> that
> >> with time DMM/TILER space gets fragmented to the point that even if we
> >> have
> >> enough free DMM/TILER space and free memory, allocation fails because
> >> there
> >> is no big enough free block in DMM/TILER space.
> >>
> >> Such failures can be easily observed with OMAP xorg DDX, for example -
> >> starting few GUI applications (so buffers for their windows are
> >> allocated)
> >> and then rotating landscape<->portrait while closing and opening new
> >> windows soon results in allocation failures.
> >>
> >> Fix that by mapping buffers through DMM/TILER only when really needed,
> >> like, for scanout buffers.
> >
> > Doesn't this break users that get a buffer from omapdrm and expect it to
> > be contiguous?
> >
>
> If you mean dumb buffer, then no, this does not break users as dumb
> buffers are allocated as scanout:
>
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>
> If you mean omap_bo allocated buffers, then if users want
> linear(scanout) buffer, then they request it explicitly by passing
> OMAP_BO_SCANOUT.
>
> Ivo



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo, Tomi

We have one X15 Android AOSP master build, it could not have the home
screen displayed
on the hdmi monitor connected with this change, with the following
message printed on the serial console
    [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
    [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
    [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
    [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3

   # for details, please check the link here: http://ix.io/47m1

It will work with home screen displayed on the hdmi monitor if this
change is reverted.

Is this the broken problem you talked about here?

And could you please give some suggestions on how to have the x15
Android build work with this change?

Thanks,
Yongqin Liu
On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
>
>
> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> > Hi,
> >
> > On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >> On devices with DMM, all allocations are done through either DMM or
> >> TILER.
> >> DMM/TILER being a limited resource means that such allocations will start
> >> to fail before actual free memory is exhausted. What is even worse is
> >> that
> >> with time DMM/TILER space gets fragmented to the point that even if we
> >> have
> >> enough free DMM/TILER space and free memory, allocation fails because
> >> there
> >> is no big enough free block in DMM/TILER space.
> >>
> >> Such failures can be easily observed with OMAP xorg DDX, for example -
> >> starting few GUI applications (so buffers for their windows are
> >> allocated)
> >> and then rotating landscape<->portrait while closing and opening new
> >> windows soon results in allocation failures.
> >>
> >> Fix that by mapping buffers through DMM/TILER only when really needed,
> >> like, for scanout buffers.
> >
> > Doesn't this break users that get a buffer from omapdrm and expect it to
> > be contiguous?
> >
>
> If you mean dumb buffer, then no, this does not break users as dumb
> buffers are allocated as scanout:
>
> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>
> If you mean omap_bo allocated buffers, then if users want
> linear(scanout) buffer, then they request it explicitly by passing
> OMAP_BO_SCANOUT.
>
> Ivo



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi Liu,

On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> Hi, Ivaylo, Tomi
> 
> We have one X15 Android AOSP master build, it could not have the home
> screen displayed
> on the hdmi monitor connected with this change, with the following
> message printed on the serial console
>      [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>      [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>      [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>      [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> 
>     # for details, please check the link here: http://ix.io/47m1
> 
> It will work with home screen displayed on the hdmi monitor if this
> change is reverted.
> 
> Is this the broken problem you talked about here?
> 
> And could you please give some suggestions on how to have the x15
> Android build work with this change?
> 

Make sure scanout (i.e. those to be displayed) buffers are actually 
allocated as such - OMAP_BO_SCANOUT flag must be set when calling 
omap_bo_new().

Regards,
Ivo.


> Thanks,
> Yongqin Liu
> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>>
>>
>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>> Hi,
>>>
>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>> On devices with DMM, all allocations are done through either DMM or
>>>> TILER.
>>>> DMM/TILER being a limited resource means that such allocations will start
>>>> to fail before actual free memory is exhausted. What is even worse is
>>>> that
>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>> have
>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>> there
>>>> is no big enough free block in DMM/TILER space.
>>>>
>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>> starting few GUI applications (so buffers for their windows are
>>>> allocated)
>>>> and then rotating landscape<->portrait while closing and opening new
>>>> windows soon results in allocation failures.
>>>>
>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>> like, for scanout buffers.
>>>
>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>> be contiguous?
>>>
>>
>> If you mean dumb buffer, then no, this does not break users as dumb
>> buffers are allocated as scanout:
>>
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>
>> If you mean omap_bo allocated buffers, then if users want
>> linear(scanout) buffer, then they request it explicitly by passing
>> OMAP_BO_SCANOUT.
>>
>> Ivo
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi Liu,

On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> Hi, Ivaylo, Tomi
> 
> We have one X15 Android AOSP master build, it could not have the home
> screen displayed
> on the hdmi monitor connected with this change, with the following
> message printed on the serial console
>      [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>      [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>      [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>      [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> 
>     # for details, please check the link here: http://ix.io/47m1
> 
> It will work with home screen displayed on the hdmi monitor if this
> change is reverted.
> 
> Is this the broken problem you talked about here?
> 
> And could you please give some suggestions on how to have the x15
> Android build work with this change?
> 

Make sure scanout (i.e. those to be displayed) buffers are actually 
allocated as such - OMAP_BO_SCANOUT flag must be set when calling 
omap_bo_new().

Regards,
Ivo.


> Thanks,
> Yongqin Liu
> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>>
>>
>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>> Hi,
>>>
>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>> On devices with DMM, all allocations are done through either DMM or
>>>> TILER.
>>>> DMM/TILER being a limited resource means that such allocations will start
>>>> to fail before actual free memory is exhausted. What is even worse is
>>>> that
>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>> have
>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>> there
>>>> is no big enough free block in DMM/TILER space.
>>>>
>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>> starting few GUI applications (so buffers for their windows are
>>>> allocated)
>>>> and then rotating landscape<->portrait while closing and opening new
>>>> windows soon results in allocation failures.
>>>>
>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>> like, for scanout buffers.
>>>
>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>> be contiguous?
>>>
>>
>> If you mean dumb buffer, then no, this does not break users as dumb
>> buffers are allocated as scanout:
>>
>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>
>> If you mean omap_bo allocated buffers, then if users want
>> linear(scanout) buffer, then they request it explicitly by passing
>> OMAP_BO_SCANOUT.
>>
>> Ivo
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, IvayIo

Thanks very much for the reply!

On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi Liu,
>
> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> > Hi, Ivaylo, Tomi
> >
> > We have one X15 Android AOSP master build, it could not have the home
> > screen displayed
> > on the hdmi monitor connected with this change, with the following
> > message printed on the serial console
> >      [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >      [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >      [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >      [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >
> >     # for details, please check the link here: http://ix.io/47m1
> >
> > It will work with home screen displayed on the hdmi monitor if this
> > change is reverted.
> >
> > Is this the broken problem you talked about here?
> >
> > And could you please give some suggestions on how to have the x15
> > Android build work with this change?
> >
>
> Make sure scanout (i.e. those to be displayed) buffers are actually
> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> omap_bo_new().

I am not familiar with this area, I am sorry if I asked quite silly questions:(
I googled omap_bo_new, and found it's a function of libdrm here[1], is
it what you meant here?

If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
before it calls omap_bo_new_impl?

And another question is that, since the userspace(libdrm) will be used
to work with different kernel versions,
like the old 4.14, 4.19, etc, do you think there will be problem to
pass  OMAP_BO_SCANOUT
from the userspace side with the old kernels(which does not have this change)?
does this change need to be backported to the old kernel versions?

And the last question is that, omap_bo_new might be called by some
property binaries what not everyone
could get the source to update, for such case what's your suggestions?

[1]: https://gitlab.freedesktop.org/mesa/drm/-/blob/main/omap/omap_drm.c#L227

Thanks,
Yongqin Liu
> > On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >>
> >>
> >> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>> Hi,
> >>>
> >>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>> On devices with DMM, all allocations are done through either DMM or
> >>>> TILER.
> >>>> DMM/TILER being a limited resource means that such allocations will start
> >>>> to fail before actual free memory is exhausted. What is even worse is
> >>>> that
> >>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>> have
> >>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>> there
> >>>> is no big enough free block in DMM/TILER space.
> >>>>
> >>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>> starting few GUI applications (so buffers for their windows are
> >>>> allocated)
> >>>> and then rotating landscape<->portrait while closing and opening new
> >>>> windows soon results in allocation failures.
> >>>>
> >>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>> like, for scanout buffers.
> >>>
> >>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>> be contiguous?
> >>>
> >>
> >> If you mean dumb buffer, then no, this does not break users as dumb
> >> buffers are allocated as scanout:
> >>
> >> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>
> >> If you mean omap_bo allocated buffers, then if users want
> >> linear(scanout) buffer, then they request it explicitly by passing
> >> OMAP_BO_SCANOUT.
> >>
> >> Ivo
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, IvayIo

Thanks very much for the reply!

On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi Liu,
>
> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> > Hi, Ivaylo, Tomi
> >
> > We have one X15 Android AOSP master build, it could not have the home
> > screen displayed
> > on the hdmi monitor connected with this change, with the following
> > message printed on the serial console
> >      [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >      [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >      [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >      [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >
> >     # for details, please check the link here: http://ix.io/47m1
> >
> > It will work with home screen displayed on the hdmi monitor if this
> > change is reverted.
> >
> > Is this the broken problem you talked about here?
> >
> > And could you please give some suggestions on how to have the x15
> > Android build work with this change?
> >
>
> Make sure scanout (i.e. those to be displayed) buffers are actually
> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> omap_bo_new().

I am not familiar with this area, I am sorry if I asked quite silly questions:(
I googled omap_bo_new, and found it's a function of libdrm here[1], is
it what you meant here?

If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
before it calls omap_bo_new_impl?

And another question is that, since the userspace(libdrm) will be used
to work with different kernel versions,
like the old 4.14, 4.19, etc, do you think there will be problem to
pass  OMAP_BO_SCANOUT
from the userspace side with the old kernels(which does not have this change)?
does this change need to be backported to the old kernel versions?

And the last question is that, omap_bo_new might be called by some
property binaries what not everyone
could get the source to update, for such case what's your suggestions?

[1]: https://gitlab.freedesktop.org/mesa/drm/-/blob/main/omap/omap_drm.c#L227

Thanks,
Yongqin Liu
> > On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >>
> >>
> >> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>> Hi,
> >>>
> >>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>> On devices with DMM, all allocations are done through either DMM or
> >>>> TILER.
> >>>> DMM/TILER being a limited resource means that such allocations will start
> >>>> to fail before actual free memory is exhausted. What is even worse is
> >>>> that
> >>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>> have
> >>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>> there
> >>>> is no big enough free block in DMM/TILER space.
> >>>>
> >>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>> starting few GUI applications (so buffers for their windows are
> >>>> allocated)
> >>>> and then rotating landscape<->portrait while closing and opening new
> >>>> windows soon results in allocation failures.
> >>>>
> >>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>> like, for scanout buffers.
> >>>
> >>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>> be contiguous?
> >>>
> >>
> >> If you mean dumb buffer, then no, this does not break users as dumb
> >> buffers are allocated as scanout:
> >>
> >> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>
> >> If you mean omap_bo allocated buffers, then if users want
> >> linear(scanout) buffer, then they request it explicitly by passing
> >> OMAP_BO_SCANOUT.
> >>
> >> Ivo
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi Liu,

On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> Hi, IvayIo
> 
> Thanks very much for the reply!
> 
> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi Liu,
>>
>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>> Hi, Ivaylo, Tomi
>>>
>>> We have one X15 Android AOSP master build, it could not have the home
>>> screen displayed
>>> on the hdmi monitor connected with this change, with the following
>>> message printed on the serial console
>>>       [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>       [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>       [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>       [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>
>>>      # for details, please check the link here: http://ix.io/47m1
>>>
>>> It will work with home screen displayed on the hdmi monitor if this
>>> change is reverted.
>>>
>>> Is this the broken problem you talked about here?
>>>
>>> And could you please give some suggestions on how to have the x15
>>> Android build work with this change?
>>>
>>
>> Make sure scanout (i.e. those to be displayed) buffers are actually
>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>> omap_bo_new().
> 
> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> it what you meant here?
> 

Yes, calling this function from userspace ends in kernel code the 
$subject patch is part of.

> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> before it calls omap_bo_new_impl?
> 

omap_bo_new() is fine and does not need any updates/fixes, it is the 
code that uses it (whoever it is, I am not familiar with the userspace 
you are using) that shall pass correct flags (third parameter) when 
calling it.

BTW you shall really find who and how uses OMAP BO API, in theory it 
might use ioctls directly and not call omap_bo_xxx functions. strace 
would be your friend there. or gdb, or whatever tools are used on 
android. Or put some printfs() in omap_bo_new() that output the PID of 
the calling process, etc.

> And another question is that, since the userspace(libdrm) will be used
> to work with different kernel versions,
> like the old 4.14, 4.19, etc, do you think there will be problem to
> pass  OMAP_BO_SCANOUT
> from the userspace side with the old kernels(which does not have this change)?
> does this change need to be backported to the old kernel versions?

There should not be any issue. The changes could be backported if one 
hits the issues this $series is fixing, but there is no need.

> 
> And the last question is that, omap_bo_new might be called by some
> property binaries what not everyone
> could get the source to update, for such case what's your suggestions?
> 

Hard to say without knowing what that library would be.

When I hit issues with closed blobs, sometimes I reverse-engineer them 
to fix the issue, example:

https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm

This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible 
for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers 
API, instead of OMAP BO API.

I guess you are using some older version of sgx-ddk-um, so you may fix 
in similar way. Or binary patch.

Regards,
Ivo

> [1]: https://gitlab.freedesktop.org/mesa/drm/-/blob/main/omap/omap_drm.c#L227
> 
> Thanks,
> Yongqin Liu
>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>> Hi,
>>>>>
>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>> TILER.
>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>> that
>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>> have
>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>> there
>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>
>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>> allocated)
>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>> windows soon results in allocation failures.
>>>>>>
>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>> like, for scanout buffers.
>>>>>
>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>> be contiguous?
>>>>>
>>>>
>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>> buffers are allocated as scanout:
>>>>
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>
>>>> If you mean omap_bo allocated buffers, then if users want
>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>> OMAP_BO_SCANOUT.
>>>>
>>>> Ivo
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi Liu,

On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> Hi, IvayIo
> 
> Thanks very much for the reply!
> 
> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi Liu,
>>
>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>> Hi, Ivaylo, Tomi
>>>
>>> We have one X15 Android AOSP master build, it could not have the home
>>> screen displayed
>>> on the hdmi monitor connected with this change, with the following
>>> message printed on the serial console
>>>       [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>       [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>       [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>       [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>
>>>      # for details, please check the link here: http://ix.io/47m1
>>>
>>> It will work with home screen displayed on the hdmi monitor if this
>>> change is reverted.
>>>
>>> Is this the broken problem you talked about here?
>>>
>>> And could you please give some suggestions on how to have the x15
>>> Android build work with this change?
>>>
>>
>> Make sure scanout (i.e. those to be displayed) buffers are actually
>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>> omap_bo_new().
> 
> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> it what you meant here?
> 

Yes, calling this function from userspace ends in kernel code the 
$subject patch is part of.

> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> before it calls omap_bo_new_impl?
> 

omap_bo_new() is fine and does not need any updates/fixes, it is the 
code that uses it (whoever it is, I am not familiar with the userspace 
you are using) that shall pass correct flags (third parameter) when 
calling it.

BTW you shall really find who and how uses OMAP BO API, in theory it 
might use ioctls directly and not call omap_bo_xxx functions. strace 
would be your friend there. or gdb, or whatever tools are used on 
android. Or put some printfs() in omap_bo_new() that output the PID of 
the calling process, etc.

> And another question is that, since the userspace(libdrm) will be used
> to work with different kernel versions,
> like the old 4.14, 4.19, etc, do you think there will be problem to
> pass  OMAP_BO_SCANOUT
> from the userspace side with the old kernels(which does not have this change)?
> does this change need to be backported to the old kernel versions?

There should not be any issue. The changes could be backported if one 
hits the issues this $series is fixing, but there is no need.

> 
> And the last question is that, omap_bo_new might be called by some
> property binaries what not everyone
> could get the source to update, for such case what's your suggestions?
> 

Hard to say without knowing what that library would be.

When I hit issues with closed blobs, sometimes I reverse-engineer them 
to fix the issue, example:

https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm

This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible 
for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers 
API, instead of OMAP BO API.

I guess you are using some older version of sgx-ddk-um, so you may fix 
in similar way. Or binary patch.

Regards,
Ivo

> [1]: https://gitlab.freedesktop.org/mesa/drm/-/blob/main/omap/omap_drm.c#L227
> 
> Thanks,
> Yongqin Liu
>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>> Hi,
>>>>>
>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>> TILER.
>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>> that
>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>> have
>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>> there
>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>
>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>> allocated)
>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>> windows soon results in allocation failures.
>>>>>>
>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>> like, for scanout buffers.
>>>>>
>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>> be contiguous?
>>>>>
>>>>
>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>> buffers are allocated as scanout:
>>>>
>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>
>>>> If you mean omap_bo allocated buffers, then if users want
>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>> OMAP_BO_SCANOUT.
>>>>
>>>> Ivo
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo

On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi Liu,
>
> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> > Hi, IvayIo
> >
> > Thanks very much for the reply!
> >
> > On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >> Hi Liu,
> >>
> >> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>> Hi, Ivaylo, Tomi
> >>>
> >>> We have one X15 Android AOSP master build, it could not have the home
> >>> screen displayed
> >>> on the hdmi monitor connected with this change, with the following
> >>> message printed on the serial console
> >>>       [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>       [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>       [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>       [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>
> >>>      # for details, please check the link here: http://ix.io/47m1
> >>>
> >>> It will work with home screen displayed on the hdmi monitor if this
> >>> change is reverted.
> >>>
> >>> Is this the broken problem you talked about here?
> >>>
> >>> And could you please give some suggestions on how to have the x15
> >>> Android build work with this change?
> >>>
> >>
> >> Make sure scanout (i.e. those to be displayed) buffers are actually
> >> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >> omap_bo_new().
> >
> > I am not familiar with this area, I am sorry if I asked quite silly questions:(
> > I googled omap_bo_new, and found it's a function of libdrm here[1], is
> > it what you meant here?
> >
>
> Yes, calling this function from userspace ends in kernel code the
> $subject patch is part of.
>
> > If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> > then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> > before it calls omap_bo_new_impl?
> >
>
> omap_bo_new() is fine and does not need any updates/fixes, it is the
> code that uses it (whoever it is, I am not familiar with the userspace
> you are using) that shall pass correct flags (third parameter) when
> calling it.

Sorry, I do not get the point here.
Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
(like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
missing the flag by some code,
and also avoids hacks/changes on the possible blob binaries.

Do I misunderstand somewhere?
Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
omap_bo_new is called?

> BTW you shall really find who and how uses OMAP BO API, in theory it
> might use ioctls directly and not call omap_bo_xxx functions.

Do you mean the DRM_OMAP_GEM_NEW ioctl api?
There is no place in the AOSP tree to call that except the
omap_bo_new_impl function,
which is called by the omap_bo_new and omap_bo_new_tiled functions.
The omap_bo_new should not be called with the OMAP_BO_TILED flag,
while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag

Regarding to the omap_bo_new function, there are 2 places call it in
the AOSP tree:
#1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
#2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so

#1 seems not used in AOSP yet, and #2 is one blob binary we do not
have the source for.

> strace
> would be your friend there. or gdb, or whatever tools are used on
> android. Or put some printfs() in omap_bo_new() that output the PID of
> the calling process, etc.

Thanks a lot for these great suggestions! Will use them when possible.

> > And another question is that, since the userspace(libdrm) will be used
> > to work with different kernel versions,
> > like the old 4.14, 4.19, etc, do you think there will be problem to
> > pass  OMAP_BO_SCANOUT
> > from the userspace side with the old kernels(which does not have this change)?
> > does this change need to be backported to the old kernel versions?
>
> There should not be any issue. The changes could be backported if one
> hits the issues this $series is fixing, but there is no need.

Thanks for the confirmation!
I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
and it worked with the current 4.14, 4.19, and the mainline kernels.
# via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.

> >
> > And the last question is that, omap_bo_new might be called by some
> > property binaries what not everyone
> > could get the source to update, for such case what's your suggestions?
> >
>
> Hard to say without knowing what that library would be.
>
> When I hit issues with closed blobs, sometimes I reverse-engineer them
> to fix the issue, example:
>
> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>
> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> API, instead of OMAP BO API.
>
> I guess you are using some older version of sgx-ddk-um, so you may fix
> in similar way. Or binary patch.

The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
any suggestions with it?
# sorry, I am not able to find out how you did the reverse-engineer
work# with the dbm repository shared here,
# not sure if you could give some tutorial steps for the similar
reverse-engineer# work with gralloc.am57x.so

[2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so

Thanks,
Yongqin Liu

> >>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>> TILER.
> >>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>> that
> >>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>> have
> >>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>> there
> >>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>
> >>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>> allocated)
> >>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>> windows soon results in allocation failures.
> >>>>>>
> >>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>> like, for scanout buffers.
> >>>>>
> >>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>> be contiguous?
> >>>>>
> >>>>
> >>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>> buffers are allocated as scanout:
> >>>>
> >>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>
> >>>> If you mean omap_bo allocated buffers, then if users want
> >>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>> OMAP_BO_SCANOUT.
> >>>>
> >>>> Ivo
> >>>
> >>>
> >>>
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo

On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi Liu,
>
> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> > Hi, IvayIo
> >
> > Thanks very much for the reply!
> >
> > On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >> Hi Liu,
> >>
> >> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>> Hi, Ivaylo, Tomi
> >>>
> >>> We have one X15 Android AOSP master build, it could not have the home
> >>> screen displayed
> >>> on the hdmi monitor connected with this change, with the following
> >>> message printed on the serial console
> >>>       [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>       [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>       [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>       [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>
> >>>      # for details, please check the link here: http://ix.io/47m1
> >>>
> >>> It will work with home screen displayed on the hdmi monitor if this
> >>> change is reverted.
> >>>
> >>> Is this the broken problem you talked about here?
> >>>
> >>> And could you please give some suggestions on how to have the x15
> >>> Android build work with this change?
> >>>
> >>
> >> Make sure scanout (i.e. those to be displayed) buffers are actually
> >> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >> omap_bo_new().
> >
> > I am not familiar with this area, I am sorry if I asked quite silly questions:(
> > I googled omap_bo_new, and found it's a function of libdrm here[1], is
> > it what you meant here?
> >
>
> Yes, calling this function from userspace ends in kernel code the
> $subject patch is part of.
>
> > If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> > then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> > before it calls omap_bo_new_impl?
> >
>
> omap_bo_new() is fine and does not need any updates/fixes, it is the
> code that uses it (whoever it is, I am not familiar with the userspace
> you are using) that shall pass correct flags (third parameter) when
> calling it.

Sorry, I do not get the point here.
Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
(like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
missing the flag by some code,
and also avoids hacks/changes on the possible blob binaries.

Do I misunderstand somewhere?
Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
omap_bo_new is called?

> BTW you shall really find who and how uses OMAP BO API, in theory it
> might use ioctls directly and not call omap_bo_xxx functions.

Do you mean the DRM_OMAP_GEM_NEW ioctl api?
There is no place in the AOSP tree to call that except the
omap_bo_new_impl function,
which is called by the omap_bo_new and omap_bo_new_tiled functions.
The omap_bo_new should not be called with the OMAP_BO_TILED flag,
while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag

Regarding to the omap_bo_new function, there are 2 places call it in
the AOSP tree:
#1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
#2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so

#1 seems not used in AOSP yet, and #2 is one blob binary we do not
have the source for.

> strace
> would be your friend there. or gdb, or whatever tools are used on
> android. Or put some printfs() in omap_bo_new() that output the PID of
> the calling process, etc.

Thanks a lot for these great suggestions! Will use them when possible.

> > And another question is that, since the userspace(libdrm) will be used
> > to work with different kernel versions,
> > like the old 4.14, 4.19, etc, do you think there will be problem to
> > pass  OMAP_BO_SCANOUT
> > from the userspace side with the old kernels(which does not have this change)?
> > does this change need to be backported to the old kernel versions?
>
> There should not be any issue. The changes could be backported if one
> hits the issues this $series is fixing, but there is no need.

Thanks for the confirmation!
I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
and it worked with the current 4.14, 4.19, and the mainline kernels.
# via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.

> >
> > And the last question is that, omap_bo_new might be called by some
> > property binaries what not everyone
> > could get the source to update, for such case what's your suggestions?
> >
>
> Hard to say without knowing what that library would be.
>
> When I hit issues with closed blobs, sometimes I reverse-engineer them
> to fix the issue, example:
>
> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>
> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> API, instead of OMAP BO API.
>
> I guess you are using some older version of sgx-ddk-um, so you may fix
> in similar way. Or binary patch.

The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
any suggestions with it?
# sorry, I am not able to find out how you did the reverse-engineer
work# with the dbm repository shared here,
# not sure if you could give some tutorial steps for the similar
reverse-engineer# work with gralloc.am57x.so

[2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so

Thanks,
Yongqin Liu

> >>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>> Hi,
> >>>>>
> >>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>> TILER.
> >>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>> that
> >>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>> have
> >>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>> there
> >>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>
> >>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>> allocated)
> >>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>> windows soon results in allocation failures.
> >>>>>>
> >>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>> like, for scanout buffers.
> >>>>>
> >>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>> be contiguous?
> >>>>>
> >>>>
> >>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>> buffers are allocated as scanout:
> >>>>
> >>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>
> >>>> If you mean omap_bo allocated buffers, then if users want
> >>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>> OMAP_BO_SCANOUT.
> >>>>
> >>>> Ivo
> >>>
> >>>
> >>>
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,

On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> Hi, Ivaylo
> 
> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi Liu,
>>
>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>> Hi, IvayIo
>>>
>>> Thanks very much for the reply!
>>>
>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>> Hi Liu,
>>>>
>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>> Hi, Ivaylo, Tomi
>>>>>
>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>> screen displayed
>>>>> on the hdmi monitor connected with this change, with the following
>>>>> message printed on the serial console
>>>>>        [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>        [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>        [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>        [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>
>>>>>       # for details, please check the link here: http://ix.io/47m1
>>>>>
>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>> change is reverted.
>>>>>
>>>>> Is this the broken problem you talked about here?
>>>>>
>>>>> And could you please give some suggestions on how to have the x15
>>>>> Android build work with this change?
>>>>>
>>>>
>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>> omap_bo_new().
>>>
>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>> it what you meant here?
>>>
>>
>> Yes, calling this function from userspace ends in kernel code the
>> $subject patch is part of.
>>
>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>> before it calls omap_bo_new_impl?
>>>
>>
>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>> code that uses it (whoever it is, I am not familiar with the userspace
>> you are using) that shall pass correct flags (third parameter) when
>> calling it.
> 
> Sorry, I do not get the point here.
> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> missing the flag by some code,
> and also avoids hacks/changes on the possible blob binaries.
> 
> Do I misunderstand somewhere?
> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> omap_bo_new is called?
> 

Exactly. You need to pass OMAP_BO_SCANOUT only when you want your 
buffers to be 'scanout' buffers(i.e. buffers that can be displayed on 
screen), which is not always the case - there is no need offscreen 
buffers or pixmaps to be scanout capable, for example. There are more 
cases like that.

The problem is that scanout buffer on OMAP4 allocate additional 
resources in DMM/TILER (a piece of hardware) and those resources are 
limited. Not only that, but DMM/TILER memory space eventually gets 
fragmented over time (if you have lots of allocataoins/deallocations) 
and you will start getting ENOMEM (or similar) errors.

Ofc, in your particular use case you may never hit such issues.

>> BTW you shall really find who and how uses OMAP BO API, in theory it
>> might use ioctls directly and not call omap_bo_xxx functions.
> 
> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> There is no place in the AOSP tree to call that except the
> omap_bo_new_impl function,
> which is called by the omap_bo_new and omap_bo_new_tiled functions.
> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> 
> Regarding to the omap_bo_new function, there are 2 places call it in
> the AOSP tree:
> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> 
> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> have the source for.
> 

I would bet on gralloc.am57x.so.

>> strace
>> would be your friend there. or gdb, or whatever tools are used on
>> android. Or put some printfs() in omap_bo_new() that output the PID of
>> the calling process, etc.
> 
> Thanks a lot for these great suggestions! Will use them when possible.
> 
>>> And another question is that, since the userspace(libdrm) will be used
>>> to work with different kernel versions,
>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>> pass  OMAP_BO_SCANOUT
>>> from the userspace side with the old kernels(which does not have this change)?
>>> does this change need to be backported to the old kernel versions?
>>
>> There should not be any issue. The changes could be backported if one
>> hits the issues this $series is fixing, but there is no need.
> 
> Thanks for the confirmation!
> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> and it worked with the current 4.14, 4.19, and the mainline kernels.
> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> 

sure, the point is that with this change *every* BO will be allocated as 
scanout BO, potentially leading to the above explained issues.

>>>
>>> And the last question is that, omap_bo_new might be called by some
>>> property binaries what not everyone
>>> could get the source to update, for such case what's your suggestions?
>>>
>>
>> Hard to say without knowing what that library would be.
>>
>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>> to fix the issue, example:
>>
>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>
>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>> API, instead of OMAP BO API.
>>
>> I guess you are using some older version of sgx-ddk-um, so you may fix
>> in similar way. Or binary patch.
> 
> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> any suggestions with it?
> # sorry, I am not able to find out how you did the reverse-engineer
> work# with the dbm repository shared here,
> # not sure if you could give some tutorial steps for the similar
> reverse-engineer# work with gralloc.am57x.so
> 

Sorry, but it is like if you ask me to provide you with a tutorial on 
how to do brain surgery :)

> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> 

I investigated this a bit and it seems it calls omap_bo_new() in a 
wrapper function like:

bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 & 
0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);

Didn't investigate further what param5 is, but it controls if 
OMAP_BO_SCANOUT is passed to omap_bo_new or not.

However, this library was not made with upstream kernel in mind, as 
AFAIK OMAP_BO_MEM_CONTIG never made it upstream:

https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/

@Tomi - any comment?

So, you have couple of options:

1. Ask TI for upstream-compatible library.
2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
3. Modify omap_bo_new() to something like:

.
.
.
#define OMAP_BO_MEM_CONTIG	0x00000008	/* only use contiguous dma mem */
.
.
.
if (flags & OMAP_BO_MEM_CONTIG)
   flags |= OMAP_BO_SCANOUT;
.
.
.

This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do, 
but should make it work, at least.

Regards,
Ivo


> Thanks,
> Yongqin Liu
> 
>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>> TILER.
>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>> that
>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>> have
>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>> there
>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>
>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>> allocated)
>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>> windows soon results in allocation failures.
>>>>>>>>
>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>> like, for scanout buffers.
>>>>>>>
>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>> be contiguous?
>>>>>>>
>>>>>>
>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>> buffers are allocated as scanout:
>>>>>>
>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>
>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>> OMAP_BO_SCANOUT.
>>>>>>
>>>>>> Ivo
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,

On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> Hi, Ivaylo
> 
> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi Liu,
>>
>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>> Hi, IvayIo
>>>
>>> Thanks very much for the reply!
>>>
>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>> Hi Liu,
>>>>
>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>> Hi, Ivaylo, Tomi
>>>>>
>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>> screen displayed
>>>>> on the hdmi monitor connected with this change, with the following
>>>>> message printed on the serial console
>>>>>        [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>        [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>        [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>        [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>
>>>>>       # for details, please check the link here: http://ix.io/47m1
>>>>>
>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>> change is reverted.
>>>>>
>>>>> Is this the broken problem you talked about here?
>>>>>
>>>>> And could you please give some suggestions on how to have the x15
>>>>> Android build work with this change?
>>>>>
>>>>
>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>> omap_bo_new().
>>>
>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>> it what you meant here?
>>>
>>
>> Yes, calling this function from userspace ends in kernel code the
>> $subject patch is part of.
>>
>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>> before it calls omap_bo_new_impl?
>>>
>>
>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>> code that uses it (whoever it is, I am not familiar with the userspace
>> you are using) that shall pass correct flags (third parameter) when
>> calling it.
> 
> Sorry, I do not get the point here.
> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> missing the flag by some code,
> and also avoids hacks/changes on the possible blob binaries.
> 
> Do I misunderstand somewhere?
> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> omap_bo_new is called?
> 

Exactly. You need to pass OMAP_BO_SCANOUT only when you want your 
buffers to be 'scanout' buffers(i.e. buffers that can be displayed on 
screen), which is not always the case - there is no need offscreen 
buffers or pixmaps to be scanout capable, for example. There are more 
cases like that.

The problem is that scanout buffer on OMAP4 allocate additional 
resources in DMM/TILER (a piece of hardware) and those resources are 
limited. Not only that, but DMM/TILER memory space eventually gets 
fragmented over time (if you have lots of allocataoins/deallocations) 
and you will start getting ENOMEM (or similar) errors.

Ofc, in your particular use case you may never hit such issues.

>> BTW you shall really find who and how uses OMAP BO API, in theory it
>> might use ioctls directly and not call omap_bo_xxx functions.
> 
> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> There is no place in the AOSP tree to call that except the
> omap_bo_new_impl function,
> which is called by the omap_bo_new and omap_bo_new_tiled functions.
> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> 
> Regarding to the omap_bo_new function, there are 2 places call it in
> the AOSP tree:
> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> 
> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> have the source for.
> 

I would bet on gralloc.am57x.so.

>> strace
>> would be your friend there. or gdb, or whatever tools are used on
>> android. Or put some printfs() in omap_bo_new() that output the PID of
>> the calling process, etc.
> 
> Thanks a lot for these great suggestions! Will use them when possible.
> 
>>> And another question is that, since the userspace(libdrm) will be used
>>> to work with different kernel versions,
>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>> pass  OMAP_BO_SCANOUT
>>> from the userspace side with the old kernels(which does not have this change)?
>>> does this change need to be backported to the old kernel versions?
>>
>> There should not be any issue. The changes could be backported if one
>> hits the issues this $series is fixing, but there is no need.
> 
> Thanks for the confirmation!
> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> and it worked with the current 4.14, 4.19, and the mainline kernels.
> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> 

sure, the point is that with this change *every* BO will be allocated as 
scanout BO, potentially leading to the above explained issues.

>>>
>>> And the last question is that, omap_bo_new might be called by some
>>> property binaries what not everyone
>>> could get the source to update, for such case what's your suggestions?
>>>
>>
>> Hard to say without knowing what that library would be.
>>
>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>> to fix the issue, example:
>>
>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>
>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>> API, instead of OMAP BO API.
>>
>> I guess you are using some older version of sgx-ddk-um, so you may fix
>> in similar way. Or binary patch.
> 
> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> any suggestions with it?
> # sorry, I am not able to find out how you did the reverse-engineer
> work# with the dbm repository shared here,
> # not sure if you could give some tutorial steps for the similar
> reverse-engineer# work with gralloc.am57x.so
> 

Sorry, but it is like if you ask me to provide you with a tutorial on 
how to do brain surgery :)

> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> 

I investigated this a bit and it seems it calls omap_bo_new() in a 
wrapper function like:

bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 & 
0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);

Didn't investigate further what param5 is, but it controls if 
OMAP_BO_SCANOUT is passed to omap_bo_new or not.

However, this library was not made with upstream kernel in mind, as 
AFAIK OMAP_BO_MEM_CONTIG never made it upstream:

https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/

@Tomi - any comment?

So, you have couple of options:

1. Ask TI for upstream-compatible library.
2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
3. Modify omap_bo_new() to something like:

.
.
.
#define OMAP_BO_MEM_CONTIG	0x00000008	/* only use contiguous dma mem */
.
.
.
if (flags & OMAP_BO_MEM_CONTIG)
   flags |= OMAP_BO_SCANOUT;
.
.
.

This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do, 
but should make it work, at least.

Regards,
Ivo


> Thanks,
> Yongqin Liu
> 
>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>> TILER.
>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>> that
>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>> have
>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>> there
>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>
>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>> allocated)
>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>> windows soon results in allocation failures.
>>>>>>>>
>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>> like, for scanout buffers.
>>>>>>>
>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>> be contiguous?
>>>>>>>
>>>>>>
>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>> buffers are allocated as scanout:
>>>>>>
>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>
>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>> OMAP_BO_SCANOUT.
>>>>>>
>>>>>> Ivo
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo

Sorry for the late response, and Thanks very much for the detailed explanations!

On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi,
>
> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> > Hi, Ivaylo
> >
> > On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >> Hi Liu,
> >>
> >> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> >>> Hi, IvayIo
> >>>
> >>> Thanks very much for the reply!
> >>>
> >>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> >>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>
> >>>> Hi Liu,
> >>>>
> >>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>>>> Hi, Ivaylo, Tomi
> >>>>>
> >>>>> We have one X15 Android AOSP master build, it could not have the home
> >>>>> screen displayed
> >>>>> on the hdmi monitor connected with this change, with the following
> >>>>> message printed on the serial console
> >>>>>        [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>>>        [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>>>        [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>>>        [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>>>
> >>>>>       # for details, please check the link here: http://ix.io/47m1
> >>>>>
> >>>>> It will work with home screen displayed on the hdmi monitor if this
> >>>>> change is reverted.
> >>>>>
> >>>>> Is this the broken problem you talked about here?
> >>>>>
> >>>>> And could you please give some suggestions on how to have the x15
> >>>>> Android build work with this change?
> >>>>>
> >>>>
> >>>> Make sure scanout (i.e. those to be displayed) buffers are actually
> >>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >>>> omap_bo_new().
> >>>
> >>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> >>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> >>> it what you meant here?
> >>>
> >>
> >> Yes, calling this function from userspace ends in kernel code the
> >> $subject patch is part of.
> >>
> >>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> >>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> >>> before it calls omap_bo_new_impl?
> >>>
> >>
> >> omap_bo_new() is fine and does not need any updates/fixes, it is the
> >> code that uses it (whoever it is, I am not familiar with the userspace
> >> you are using) that shall pass correct flags (third parameter) when
> >> calling it.
> >
> > Sorry, I do not get the point here.
> > Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> > then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> > (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> > missing the flag by some code,
> > and also avoids hacks/changes on the possible blob binaries.
> >
> > Do I misunderstand somewhere?
> > Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> > omap_bo_new is called?
> >
>
> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
> screen), which is not always the case - there is no need offscreen
> buffers or pixmaps to be scanout capable, for example. There are more
> cases like that.
>
> The problem is that scanout buffer on OMAP4 allocate additional
> resources in DMM/TILER (a piece of hardware) and those resources are
> limited. Not only that, but DMM/TILER memory space eventually gets
> fragmented over time (if you have lots of allocataoins/deallocations)
> and you will start getting ENOMEM (or similar) errors.
>
> Ofc, in your particular use case you may never hit such issues.

Thanks, I understand the cases now.


> >> BTW you shall really find who and how uses OMAP BO API, in theory it
> >> might use ioctls directly and not call omap_bo_xxx functions.
> >
> > Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> > There is no place in the AOSP tree to call that except the
> > omap_bo_new_impl function,
> > which is called by the omap_bo_new and omap_bo_new_tiled functions.
> > The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> > while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> >
> > Regarding to the omap_bo_new function, there are 2 places call it in
> > the AOSP tree:
> > #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> > #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> >
> > #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> > have the source for.
> >
>
> I would bet on gralloc.am57x.so.
yeah, that's my guess as well.

> >> strace
> >> would be your friend there. or gdb, or whatever tools are used on
> >> android. Or put some printfs() in omap_bo_new() that output the PID of
> >> the calling process, etc.
> >
> > Thanks a lot for these great suggestions! Will use them when possible.
> >
> >>> And another question is that, since the userspace(libdrm) will be used
> >>> to work with different kernel versions,
> >>> like the old 4.14, 4.19, etc, do you think there will be problem to
> >>> pass  OMAP_BO_SCANOUT
> >>> from the userspace side with the old kernels(which does not have this change)?
> >>> does this change need to be backported to the old kernel versions?
> >>
> >> There should not be any issue. The changes could be backported if one
> >> hits the issues this $series is fixing, but there is no need.
> >
> > Thanks for the confirmation!
> > I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> > and it worked with the current 4.14, 4.19, and the mainline kernels.
> > # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> >
>
> sure, the point is that with this change *every* BO will be allocated as
> scanout BO, potentially leading to the above explained issues.

get it.

> >>>
> >>> And the last question is that, omap_bo_new might be called by some
> >>> property binaries what not everyone
> >>> could get the source to update, for such case what's your suggestions?
> >>>
> >>
> >> Hard to say without knowing what that library would be.
> >>
> >> When I hit issues with closed blobs, sometimes I reverse-engineer them
> >> to fix the issue, example:
> >>
> >> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
> >>
> >> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> >> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> >> API, instead of OMAP BO API.
> >>
> >> I guess you are using some older version of sgx-ddk-um, so you may fix
> >> in similar way. Or binary patch.
> >
> > The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> > any suggestions with it?
> > # sorry, I am not able to find out how you did the reverse-engineer
> > work# with the dbm repository shared here,
> > # not sure if you could give some tutorial steps for the similar
> > reverse-engineer# work with gralloc.am57x.so
> >
>
> Sorry, but it is like if you ask me to provide you with a tutorial on
> how to do brain surgery :)
>
> > [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> >
>
> I investigated this a bit and it seems it calls omap_bo_new() in a
> wrapper function like:
>
> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>
> Didn't investigate further what param5 is, but it controls if
> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>
> However, this library was not made with upstream kernel in mind, as
> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>
> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>
> @Tomi - any comment?
>
> So, you have couple of options:
>
> 1. Ask TI for upstream-compatible library.
check is in progress, but it would take quite a long time I guess
> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
hmm, sounds like one impossible thing...
> 3. Modify omap_bo_new() to something like:
> .
> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
> .
> if (flags & OMAP_BO_MEM_CONTIG)
>    flags |= OMAP_BO_SCANOUT;
> .
> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
> but should make it work, at least.

This looks like the only doable thing at the moment, maybe one change
needs to be submitted to the mesa/drm repository.
I can submit a request on your #3 change to the mesa/drm repository
for discussion after some check if you do not mind.

Thanks,
Yongqin Liu

> >>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>>>> TILER.
> >>>>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>>>> that
> >>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>>>> have
> >>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>>>> there
> >>>>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>>>
> >>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>>>> allocated)
> >>>>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>>>> windows soon results in allocation failures.
> >>>>>>>>
> >>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>>>> like, for scanout buffers.
> >>>>>>>
> >>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>>>> be contiguous?
> >>>>>>>
> >>>>>>
> >>>>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>>>> buffers are allocated as scanout:
> >>>>>>
> >>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>>>
> >>>>>> If you mean omap_bo allocated buffers, then if users want
> >>>>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>>>> OMAP_BO_SCANOUT.
> >>>>>>
> >>>>>> Ivo
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

Hi, Ivaylo

Sorry for the late response, and Thanks very much for the detailed explanations!

On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
<ivo.g.dimitrov.75@gmail.com> wrote:
>
> Hi,
>
> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> > Hi, Ivaylo
> >
> > On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> > <ivo.g.dimitrov.75@gmail.com> wrote:
> >>
> >> Hi Liu,
> >>
> >> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> >>> Hi, IvayIo
> >>>
> >>> Thanks very much for the reply!
> >>>
> >>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> >>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>
> >>>> Hi Liu,
> >>>>
> >>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>>>> Hi, Ivaylo, Tomi
> >>>>>
> >>>>> We have one X15 Android AOSP master build, it could not have the home
> >>>>> screen displayed
> >>>>> on the hdmi monitor connected with this change, with the following
> >>>>> message printed on the serial console
> >>>>>        [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>>>        [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>>>        [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>>>        [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>>>
> >>>>>       # for details, please check the link here: http://ix.io/47m1
> >>>>>
> >>>>> It will work with home screen displayed on the hdmi monitor if this
> >>>>> change is reverted.
> >>>>>
> >>>>> Is this the broken problem you talked about here?
> >>>>>
> >>>>> And could you please give some suggestions on how to have the x15
> >>>>> Android build work with this change?
> >>>>>
> >>>>
> >>>> Make sure scanout (i.e. those to be displayed) buffers are actually
> >>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >>>> omap_bo_new().
> >>>
> >>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> >>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> >>> it what you meant here?
> >>>
> >>
> >> Yes, calling this function from userspace ends in kernel code the
> >> $subject patch is part of.
> >>
> >>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> >>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> >>> before it calls omap_bo_new_impl?
> >>>
> >>
> >> omap_bo_new() is fine and does not need any updates/fixes, it is the
> >> code that uses it (whoever it is, I am not familiar with the userspace
> >> you are using) that shall pass correct flags (third parameter) when
> >> calling it.
> >
> > Sorry, I do not get the point here.
> > Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> > then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> > (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> > missing the flag by some code,
> > and also avoids hacks/changes on the possible blob binaries.
> >
> > Do I misunderstand somewhere?
> > Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> > omap_bo_new is called?
> >
>
> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
> screen), which is not always the case - there is no need offscreen
> buffers or pixmaps to be scanout capable, for example. There are more
> cases like that.
>
> The problem is that scanout buffer on OMAP4 allocate additional
> resources in DMM/TILER (a piece of hardware) and those resources are
> limited. Not only that, but DMM/TILER memory space eventually gets
> fragmented over time (if you have lots of allocataoins/deallocations)
> and you will start getting ENOMEM (or similar) errors.
>
> Ofc, in your particular use case you may never hit such issues.

Thanks, I understand the cases now.


> >> BTW you shall really find who and how uses OMAP BO API, in theory it
> >> might use ioctls directly and not call omap_bo_xxx functions.
> >
> > Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> > There is no place in the AOSP tree to call that except the
> > omap_bo_new_impl function,
> > which is called by the omap_bo_new and omap_bo_new_tiled functions.
> > The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> > while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> >
> > Regarding to the omap_bo_new function, there are 2 places call it in
> > the AOSP tree:
> > #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> > #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> >
> > #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> > have the source for.
> >
>
> I would bet on gralloc.am57x.so.
yeah, that's my guess as well.

> >> strace
> >> would be your friend there. or gdb, or whatever tools are used on
> >> android. Or put some printfs() in omap_bo_new() that output the PID of
> >> the calling process, etc.
> >
> > Thanks a lot for these great suggestions! Will use them when possible.
> >
> >>> And another question is that, since the userspace(libdrm) will be used
> >>> to work with different kernel versions,
> >>> like the old 4.14, 4.19, etc, do you think there will be problem to
> >>> pass  OMAP_BO_SCANOUT
> >>> from the userspace side with the old kernels(which does not have this change)?
> >>> does this change need to be backported to the old kernel versions?
> >>
> >> There should not be any issue. The changes could be backported if one
> >> hits the issues this $series is fixing, but there is no need.
> >
> > Thanks for the confirmation!
> > I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> > and it worked with the current 4.14, 4.19, and the mainline kernels.
> > # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> >
>
> sure, the point is that with this change *every* BO will be allocated as
> scanout BO, potentially leading to the above explained issues.

get it.

> >>>
> >>> And the last question is that, omap_bo_new might be called by some
> >>> property binaries what not everyone
> >>> could get the source to update, for such case what's your suggestions?
> >>>
> >>
> >> Hard to say without knowing what that library would be.
> >>
> >> When I hit issues with closed blobs, sometimes I reverse-engineer them
> >> to fix the issue, example:
> >>
> >> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
> >>
> >> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> >> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> >> API, instead of OMAP BO API.
> >>
> >> I guess you are using some older version of sgx-ddk-um, so you may fix
> >> in similar way. Or binary patch.
> >
> > The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> > any suggestions with it?
> > # sorry, I am not able to find out how you did the reverse-engineer
> > work# with the dbm repository shared here,
> > # not sure if you could give some tutorial steps for the similar
> > reverse-engineer# work with gralloc.am57x.so
> >
>
> Sorry, but it is like if you ask me to provide you with a tutorial on
> how to do brain surgery :)
>
> > [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> >
>
> I investigated this a bit and it seems it calls omap_bo_new() in a
> wrapper function like:
>
> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>
> Didn't investigate further what param5 is, but it controls if
> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>
> However, this library was not made with upstream kernel in mind, as
> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>
> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>
> @Tomi - any comment?
>
> So, you have couple of options:
>
> 1. Ask TI for upstream-compatible library.
check is in progress, but it would take quite a long time I guess
> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
hmm, sounds like one impossible thing...
> 3. Modify omap_bo_new() to something like:
> .
> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
> .
> if (flags & OMAP_BO_MEM_CONTIG)
>    flags |= OMAP_BO_SCANOUT;
> .
> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
> but should make it work, at least.

This looks like the only doable thing at the moment, maybe one change
needs to be submitted to the mesa/drm repository.
I can submit a request on your #3 change to the mesa/drm repository
for discussion after some check if you do not mind.

Thanks,
Yongqin Liu

> >>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>>>> TILER.
> >>>>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>>>> that
> >>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>>>> have
> >>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>>>> there
> >>>>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>>>
> >>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>>>> allocated)
> >>>>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>>>> windows soon results in allocation failures.
> >>>>>>>>
> >>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>>>> like, for scanout buffers.
> >>>>>>>
> >>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>>>> be contiguous?
> >>>>>>>
> >>>>>>
> >>>>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>>>> buffers are allocated as scanout:
> >>>>>>
> >>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>>>
> >>>>>> If you mean omap_bo allocated buffers, then if users want
> >>>>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>>>> OMAP_BO_SCANOUT.
> >>>>>>
> >>>>>> Ivo
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
> >
> >
> >



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,


On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
> Hi, Ivaylo
> 
> Sorry for the late response, and Thanks very much for the detailed explanations!
> 
> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi,
>>
>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>> Hi, Ivaylo
>>>
>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>> Hi Liu,
>>>>
>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>> Hi, IvayIo
>>>>>
>>>>> Thanks very much for the reply!
>>>>>
>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>
>>>>>> Hi Liu,
>>>>>>
>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>
>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>> screen displayed
>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>> message printed on the serial console
>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>
>>>>>>>        # for details, please check the link here: http://ix.io/47m1
>>>>>>>
>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>> change is reverted.
>>>>>>>
>>>>>>> Is this the broken problem you talked about here?
>>>>>>>
>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>> Android build work with this change?
>>>>>>>
>>>>>>
>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>> omap_bo_new().
>>>>>
>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>> it what you meant here?
>>>>>
>>>>
>>>> Yes, calling this function from userspace ends in kernel code the
>>>> $subject patch is part of.
>>>>
>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>> before it calls omap_bo_new_impl?
>>>>>
>>>>
>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>> you are using) that shall pass correct flags (third parameter) when
>>>> calling it.
>>>
>>> Sorry, I do not get the point here.
>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>> missing the flag by some code,
>>> and also avoids hacks/changes on the possible blob binaries.
>>>
>>> Do I misunderstand somewhere?
>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>> omap_bo_new is called?
>>>
>>
>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>> screen), which is not always the case - there is no need offscreen
>> buffers or pixmaps to be scanout capable, for example. There are more
>> cases like that.
>>
>> The problem is that scanout buffer on OMAP4 allocate additional
>> resources in DMM/TILER (a piece of hardware) and those resources are
>> limited. Not only that, but DMM/TILER memory space eventually gets
>> fragmented over time (if you have lots of allocataoins/deallocations)
>> and you will start getting ENOMEM (or similar) errors.
>>
>> Ofc, in your particular use case you may never hit such issues.
> 
> Thanks, I understand the cases now.
> 
> 
>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>
>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>> There is no place in the AOSP tree to call that except the
>>> omap_bo_new_impl function,
>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>
>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>> the AOSP tree:
>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>
>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>> have the source for.
>>>
>>
>> I would bet on gralloc.am57x.so.
> yeah, that's my guess as well.
> 
>>>> strace
>>>> would be your friend there. or gdb, or whatever tools are used on
>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>> the calling process, etc.
>>>
>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>
>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>> to work with different kernel versions,
>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>> pass  OMAP_BO_SCANOUT
>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>> does this change need to be backported to the old kernel versions?
>>>>
>>>> There should not be any issue. The changes could be backported if one
>>>> hits the issues this $series is fixing, but there is no need.
>>>
>>> Thanks for the confirmation!
>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>
>>
>> sure, the point is that with this change *every* BO will be allocated as
>> scanout BO, potentially leading to the above explained issues.
> 
> get it.
> 
>>>>>
>>>>> And the last question is that, omap_bo_new might be called by some
>>>>> property binaries what not everyone
>>>>> could get the source to update, for such case what's your suggestions?
>>>>>
>>>>
>>>> Hard to say without knowing what that library would be.
>>>>
>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>> to fix the issue, example:
>>>>
>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>
>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>> API, instead of OMAP BO API.
>>>>
>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>> in similar way. Or binary patch.
>>>
>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>> any suggestions with it?
>>> # sorry, I am not able to find out how you did the reverse-engineer
>>> work# with the dbm repository shared here,
>>> # not sure if you could give some tutorial steps for the similar
>>> reverse-engineer# work with gralloc.am57x.so
>>>
>>
>> Sorry, but it is like if you ask me to provide you with a tutorial on
>> how to do brain surgery :)
>>
>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>
>>
>> I investigated this a bit and it seems it calls omap_bo_new() in a
>> wrapper function like:
>>
>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>
>> Didn't investigate further what param5 is, but it controls if
>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>
>> However, this library was not made with upstream kernel in mind, as
>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>
>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>
>> @Tomi - any comment?
>>
>> So, you have couple of options:
>>
>> 1. Ask TI for upstream-compatible library.
> check is in progress, but it would take quite a long time I guess
>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
> hmm, sounds like one impossible thing...
>> 3. Modify omap_bo_new() to something like:
>> .
>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>> .
>> if (flags & OMAP_BO_MEM_CONTIG)
>>     flags |= OMAP_BO_SCANOUT;
>> .
>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>> but should make it work, at least.
> 
> This looks like the only doable thing at the moment, maybe one change
> needs to be submitted to the mesa/drm repository.
> I can submit a request on your #3 change to the mesa/drm repository
> for discussion after some check if you do not mind.
> 

I doubt mesa/drm will accept such hack, I think you will need to support 
your drm clone (with the above fix) until TI fixes the closed library.

Regards,
Ivo

> Thanks,
> Yongqin Liu
> 
>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>> TILER.
>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>> that
>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>> have
>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>> there
>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>
>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>> allocated)
>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>
>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>
>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>> be contiguous?
>>>>>>>>>
>>>>>>>>
>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>> buffers are allocated as scanout:
>>>>>>>>
>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>
>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>
>>>>>>>> Ivo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,


On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
> Hi, Ivaylo
> 
> Sorry for the late response, and Thanks very much for the detailed explanations!
> 
> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
> <ivo.g.dimitrov.75@gmail.com> wrote:
>>
>> Hi,
>>
>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>> Hi, Ivaylo
>>>
>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>
>>>> Hi Liu,
>>>>
>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>> Hi, IvayIo
>>>>>
>>>>> Thanks very much for the reply!
>>>>>
>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>
>>>>>> Hi Liu,
>>>>>>
>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>
>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>> screen displayed
>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>> message printed on the serial console
>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>
>>>>>>>        # for details, please check the link here: http://ix.io/47m1
>>>>>>>
>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>> change is reverted.
>>>>>>>
>>>>>>> Is this the broken problem you talked about here?
>>>>>>>
>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>> Android build work with this change?
>>>>>>>
>>>>>>
>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>> omap_bo_new().
>>>>>
>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>> it what you meant here?
>>>>>
>>>>
>>>> Yes, calling this function from userspace ends in kernel code the
>>>> $subject patch is part of.
>>>>
>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>> before it calls omap_bo_new_impl?
>>>>>
>>>>
>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>> you are using) that shall pass correct flags (third parameter) when
>>>> calling it.
>>>
>>> Sorry, I do not get the point here.
>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>> missing the flag by some code,
>>> and also avoids hacks/changes on the possible blob binaries.
>>>
>>> Do I misunderstand somewhere?
>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>> omap_bo_new is called?
>>>
>>
>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>> screen), which is not always the case - there is no need offscreen
>> buffers or pixmaps to be scanout capable, for example. There are more
>> cases like that.
>>
>> The problem is that scanout buffer on OMAP4 allocate additional
>> resources in DMM/TILER (a piece of hardware) and those resources are
>> limited. Not only that, but DMM/TILER memory space eventually gets
>> fragmented over time (if you have lots of allocataoins/deallocations)
>> and you will start getting ENOMEM (or similar) errors.
>>
>> Ofc, in your particular use case you may never hit such issues.
> 
> Thanks, I understand the cases now.
> 
> 
>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>
>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>> There is no place in the AOSP tree to call that except the
>>> omap_bo_new_impl function,
>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>
>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>> the AOSP tree:
>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>
>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>> have the source for.
>>>
>>
>> I would bet on gralloc.am57x.so.
> yeah, that's my guess as well.
> 
>>>> strace
>>>> would be your friend there. or gdb, or whatever tools are used on
>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>> the calling process, etc.
>>>
>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>
>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>> to work with different kernel versions,
>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>> pass  OMAP_BO_SCANOUT
>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>> does this change need to be backported to the old kernel versions?
>>>>
>>>> There should not be any issue. The changes could be backported if one
>>>> hits the issues this $series is fixing, but there is no need.
>>>
>>> Thanks for the confirmation!
>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>
>>
>> sure, the point is that with this change *every* BO will be allocated as
>> scanout BO, potentially leading to the above explained issues.
> 
> get it.
> 
>>>>>
>>>>> And the last question is that, omap_bo_new might be called by some
>>>>> property binaries what not everyone
>>>>> could get the source to update, for such case what's your suggestions?
>>>>>
>>>>
>>>> Hard to say without knowing what that library would be.
>>>>
>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>> to fix the issue, example:
>>>>
>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>
>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>> API, instead of OMAP BO API.
>>>>
>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>> in similar way. Or binary patch.
>>>
>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>> any suggestions with it?
>>> # sorry, I am not able to find out how you did the reverse-engineer
>>> work# with the dbm repository shared here,
>>> # not sure if you could give some tutorial steps for the similar
>>> reverse-engineer# work with gralloc.am57x.so
>>>
>>
>> Sorry, but it is like if you ask me to provide you with a tutorial on
>> how to do brain surgery :)
>>
>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>
>>
>> I investigated this a bit and it seems it calls omap_bo_new() in a
>> wrapper function like:
>>
>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>
>> Didn't investigate further what param5 is, but it controls if
>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>
>> However, this library was not made with upstream kernel in mind, as
>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>
>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>
>> @Tomi - any comment?
>>
>> So, you have couple of options:
>>
>> 1. Ask TI for upstream-compatible library.
> check is in progress, but it would take quite a long time I guess
>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
> hmm, sounds like one impossible thing...
>> 3. Modify omap_bo_new() to something like:
>> .
>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>> .
>> if (flags & OMAP_BO_MEM_CONTIG)
>>     flags |= OMAP_BO_SCANOUT;
>> .
>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>> but should make it work, at least.
> 
> This looks like the only doable thing at the moment, maybe one change
> needs to be submitted to the mesa/drm repository.
> I can submit a request on your #3 change to the mesa/drm repository
> for discussion after some check if you do not mind.
> 

I doubt mesa/drm will accept such hack, I think you will need to support 
your drm clone (with the above fix) until TI fixes the closed library.

Regards,
Ivo

> Thanks,
> Yongqin Liu
> 
>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>> TILER.
>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>> that
>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>> have
>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>> there
>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>
>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>> allocated)
>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>
>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>
>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>> be contiguous?
>>>>>>>>>
>>>>>>>>
>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>> buffers are allocated as scanout:
>>>>>>>>
>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>
>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>
>>>>>>>> Ivo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Andrew Davis]

On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
> Hi,
> 
> 
> On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
>> Hi, Ivaylo
>>
>> Sorry for the late response, and Thanks very much for the detailed explanations!
>>
>> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>>> Hi, Ivaylo
>>>>
>>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>
>>>>> Hi Liu,
>>>>>
>>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>>> Hi, IvayIo
>>>>>>
>>>>>> Thanks very much for the reply!
>>>>>>
>>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Liu,
>>>>>>>
>>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>>
>>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>>> screen displayed
>>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>>> message printed on the serial console
>>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>>
>>>>>>>>        # for details, please check the link here: http://ix.io/47m1
>>>>>>>>
>>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>>> change is reverted.
>>>>>>>>
>>>>>>>> Is this the broken problem you talked about here?
>>>>>>>>
>>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>>> Android build work with this change?
>>>>>>>>
>>>>>>>
>>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>>> omap_bo_new().
>>>>>>
>>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>>> it what you meant here?
>>>>>>
>>>>>
>>>>> Yes, calling this function from userspace ends in kernel code the
>>>>> $subject patch is part of.
>>>>>
>>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>>> before it calls omap_bo_new_impl?
>>>>>>
>>>>>
>>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>>> you are using) that shall pass correct flags (third parameter) when
>>>>> calling it.
>>>>
>>>> Sorry, I do not get the point here.
>>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>>> missing the flag by some code,
>>>> and also avoids hacks/changes on the possible blob binaries.
>>>>
>>>> Do I misunderstand somewhere?
>>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>>> omap_bo_new is called?
>>>>
>>>
>>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>>> screen), which is not always the case - there is no need offscreen
>>> buffers or pixmaps to be scanout capable, for example. There are more
>>> cases like that.
>>>
>>> The problem is that scanout buffer on OMAP4 allocate additional
>>> resources in DMM/TILER (a piece of hardware) and those resources are
>>> limited. Not only that, but DMM/TILER memory space eventually gets
>>> fragmented over time (if you have lots of allocataoins/deallocations)
>>> and you will start getting ENOMEM (or similar) errors.
>>>
>>> Ofc, in your particular use case you may never hit such issues.
>>
>> Thanks, I understand the cases now.
>>
>>
>>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>>
>>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>>> There is no place in the AOSP tree to call that except the
>>>> omap_bo_new_impl function,
>>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>>
>>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>>> the AOSP tree:
>>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>>
>>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>>> have the source for.
>>>>
>>>
>>> I would bet on gralloc.am57x.so.
>> yeah, that's my guess as well.
>>
>>>>> strace
>>>>> would be your friend there. or gdb, or whatever tools are used on
>>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>>> the calling process, etc.
>>>>
>>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>>
>>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>>> to work with different kernel versions,
>>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>>> pass  OMAP_BO_SCANOUT
>>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>>> does this change need to be backported to the old kernel versions?
>>>>>
>>>>> There should not be any issue. The changes could be backported if one
>>>>> hits the issues this $series is fixing, but there is no need.
>>>>
>>>> Thanks for the confirmation!
>>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>>
>>>
>>> sure, the point is that with this change *every* BO will be allocated as
>>> scanout BO, potentially leading to the above explained issues.
>>
>> get it.
>>
>>>>>>
>>>>>> And the last question is that, omap_bo_new might be called by some
>>>>>> property binaries what not everyone
>>>>>> could get the source to update, for such case what's your suggestions?
>>>>>>
>>>>>
>>>>> Hard to say without knowing what that library would be.
>>>>>
>>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>>> to fix the issue, example:
>>>>>
>>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>>
>>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>>> API, instead of OMAP BO API.
>>>>>
>>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>>> in similar way. Or binary patch.
>>>>
>>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>>> any suggestions with it?
>>>> # sorry, I am not able to find out how you did the reverse-engineer
>>>> work# with the dbm repository shared here,
>>>> # not sure if you could give some tutorial steps for the similar
>>>> reverse-engineer# work with gralloc.am57x.so
>>>>
>>>
>>> Sorry, but it is like if you ask me to provide you with a tutorial on
>>> how to do brain surgery :)
>>>
>>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>>
>>>
>>> I investigated this a bit and it seems it calls omap_bo_new() in a
>>> wrapper function like:
>>>
>>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>>
>>> Didn't investigate further what param5 is, but it controls if
>>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>>
>>> However, this library was not made with upstream kernel in mind, as
>>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>>
>>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>>
>>> @Tomi - any comment?
>>>
>>> So, you have couple of options:
>>>
>>> 1. Ask TI for upstream-compatible library.
>> check is in progress, but it would take quite a long time I guess
>>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
>> hmm, sounds like one impossible thing...
>>> 3. Modify omap_bo_new() to something like:
>>> .
>>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>>> .
>>> if (flags & OMAP_BO_MEM_CONTIG)
>>>     flags |= OMAP_BO_SCANOUT;
>>> .
>>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>>> but should make it work, at least.
>>
>> This looks like the only doable thing at the moment, maybe one change
>> needs to be submitted to the mesa/drm repository.
>> I can submit a request on your #3 change to the mesa/drm repository
>> for discussion after some check if you do not mind.
>>
> 
> I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
> 


Hi all,

Just got around to reading this thread. I work with the TI gralloc lib
and can generate new versions as needed (I was probably the one who compiled
the version you have now). I've wanted to have our gralloc layer open source'd
as there is nothing really propriety in it (and I re-wrote a lot of it already)
and to avoid issues like this. But it interacts with the GPU code in some places,
so it's up to Imagination :(. The actual code in question if it helps is:

	if(ui32Flags & PVRSRV_MEM_CACHED)
		flags &= ~OMAP_BO_CACHE_MASK;
	else
		flags |= OMAP_BO_WC;

	if (ui32Flags & PVRSRV_HAP_CONTIG)
		flags |= OMAP_BO_SCANOUT;

	flags &= ~OMAP_BO_TILED_MASK;
	flags |= 0x00000008;
	flags |= OMAP_BO_WC;

	bo = omap_bo_new(dev, size, flags);

As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
This was a hack added since even non-scanout buffers sometimes need
to be contiguous (video decoder surfaces), but we had no way back
then to communicate this to the gralloc layer. I think your best
bet would be to modify the gralloc lib to not do that, or put it
under the CONTIG check.

If you tell me what the code should look like, I can rebuild the
lib and post a copy.

Long term, I'd like to start using DMA-BUF Heaps for CMA memory
allocations in gralloc and elsewhere, then drop out the DMM/TILER
support from OMAPDRM, since it never really belonged there in
the first place (being a IOMMU unrelated to the display/GPU).

Thanks,
Andrew


> Regards,
> Ivo
> 
>> Thanks,
>> Yongqin Liu
>>
>>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>>> TILER.
>>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>>> that
>>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>>> have
>>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>>> there
>>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>>
>>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>>> allocated)
>>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>>
>>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>>
>>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>>> be contiguous?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>>> buffers are allocated as scanout:
>>>>>>>>>
>>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>>
>>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>>
>>>>>>>>> Ivo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Andrew Davis]

On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
> Hi,
> 
> 
> On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
>> Hi, Ivaylo
>>
>> Sorry for the late response, and Thanks very much for the detailed explanations!
>>
>> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>>> Hi, Ivaylo
>>>>
>>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>
>>>>> Hi Liu,
>>>>>
>>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>>> Hi, IvayIo
>>>>>>
>>>>>> Thanks very much for the reply!
>>>>>>
>>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Liu,
>>>>>>>
>>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>>
>>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>>> screen displayed
>>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>>> message printed on the serial console
>>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>>
>>>>>>>>        # for details, please check the link here: http://ix.io/47m1
>>>>>>>>
>>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>>> change is reverted.
>>>>>>>>
>>>>>>>> Is this the broken problem you talked about here?
>>>>>>>>
>>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>>> Android build work with this change?
>>>>>>>>
>>>>>>>
>>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>>> omap_bo_new().
>>>>>>
>>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>>> it what you meant here?
>>>>>>
>>>>>
>>>>> Yes, calling this function from userspace ends in kernel code the
>>>>> $subject patch is part of.
>>>>>
>>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>>> before it calls omap_bo_new_impl?
>>>>>>
>>>>>
>>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>>> you are using) that shall pass correct flags (third parameter) when
>>>>> calling it.
>>>>
>>>> Sorry, I do not get the point here.
>>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>>> missing the flag by some code,
>>>> and also avoids hacks/changes on the possible blob binaries.
>>>>
>>>> Do I misunderstand somewhere?
>>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>>> omap_bo_new is called?
>>>>
>>>
>>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>>> screen), which is not always the case - there is no need offscreen
>>> buffers or pixmaps to be scanout capable, for example. There are more
>>> cases like that.
>>>
>>> The problem is that scanout buffer on OMAP4 allocate additional
>>> resources in DMM/TILER (a piece of hardware) and those resources are
>>> limited. Not only that, but DMM/TILER memory space eventually gets
>>> fragmented over time (if you have lots of allocataoins/deallocations)
>>> and you will start getting ENOMEM (or similar) errors.
>>>
>>> Ofc, in your particular use case you may never hit such issues.
>>
>> Thanks, I understand the cases now.
>>
>>
>>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>>
>>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>>> There is no place in the AOSP tree to call that except the
>>>> omap_bo_new_impl function,
>>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>>
>>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>>> the AOSP tree:
>>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>>
>>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>>> have the source for.
>>>>
>>>
>>> I would bet on gralloc.am57x.so.
>> yeah, that's my guess as well.
>>
>>>>> strace
>>>>> would be your friend there. or gdb, or whatever tools are used on
>>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>>> the calling process, etc.
>>>>
>>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>>
>>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>>> to work with different kernel versions,
>>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>>> pass  OMAP_BO_SCANOUT
>>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>>> does this change need to be backported to the old kernel versions?
>>>>>
>>>>> There should not be any issue. The changes could be backported if one
>>>>> hits the issues this $series is fixing, but there is no need.
>>>>
>>>> Thanks for the confirmation!
>>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>>
>>>
>>> sure, the point is that with this change *every* BO will be allocated as
>>> scanout BO, potentially leading to the above explained issues.
>>
>> get it.
>>
>>>>>>
>>>>>> And the last question is that, omap_bo_new might be called by some
>>>>>> property binaries what not everyone
>>>>>> could get the source to update, for such case what's your suggestions?
>>>>>>
>>>>>
>>>>> Hard to say without knowing what that library would be.
>>>>>
>>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>>> to fix the issue, example:
>>>>>
>>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>>
>>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>>> API, instead of OMAP BO API.
>>>>>
>>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>>> in similar way. Or binary patch.
>>>>
>>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>>> any suggestions with it?
>>>> # sorry, I am not able to find out how you did the reverse-engineer
>>>> work# with the dbm repository shared here,
>>>> # not sure if you could give some tutorial steps for the similar
>>>> reverse-engineer# work with gralloc.am57x.so
>>>>
>>>
>>> Sorry, but it is like if you ask me to provide you with a tutorial on
>>> how to do brain surgery :)
>>>
>>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>>
>>>
>>> I investigated this a bit and it seems it calls omap_bo_new() in a
>>> wrapper function like:
>>>
>>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>>
>>> Didn't investigate further what param5 is, but it controls if
>>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>>
>>> However, this library was not made with upstream kernel in mind, as
>>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>>
>>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>>
>>> @Tomi - any comment?
>>>
>>> So, you have couple of options:
>>>
>>> 1. Ask TI for upstream-compatible library.
>> check is in progress, but it would take quite a long time I guess
>>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
>> hmm, sounds like one impossible thing...
>>> 3. Modify omap_bo_new() to something like:
>>> .
>>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>>> .
>>> if (flags & OMAP_BO_MEM_CONTIG)
>>>     flags |= OMAP_BO_SCANOUT;
>>> .
>>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>>> but should make it work, at least.
>>
>> This looks like the only doable thing at the moment, maybe one change
>> needs to be submitted to the mesa/drm repository.
>> I can submit a request on your #3 change to the mesa/drm repository
>> for discussion after some check if you do not mind.
>>
> 
> I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
> 


Hi all,

Just got around to reading this thread. I work with the TI gralloc lib
and can generate new versions as needed (I was probably the one who compiled
the version you have now). I've wanted to have our gralloc layer open source'd
as there is nothing really propriety in it (and I re-wrote a lot of it already)
and to avoid issues like this. But it interacts with the GPU code in some places,
so it's up to Imagination :(. The actual code in question if it helps is:

	if(ui32Flags & PVRSRV_MEM_CACHED)
		flags &= ~OMAP_BO_CACHE_MASK;
	else
		flags |= OMAP_BO_WC;

	if (ui32Flags & PVRSRV_HAP_CONTIG)
		flags |= OMAP_BO_SCANOUT;

	flags &= ~OMAP_BO_TILED_MASK;
	flags |= 0x00000008;
	flags |= OMAP_BO_WC;

	bo = omap_bo_new(dev, size, flags);

As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
This was a hack added since even non-scanout buffers sometimes need
to be contiguous (video decoder surfaces), but we had no way back
then to communicate this to the gralloc layer. I think your best
bet would be to modify the gralloc lib to not do that, or put it
under the CONTIG check.

If you tell me what the code should look like, I can rebuild the
lib and post a copy.

Long term, I'd like to start using DMA-BUF Heaps for CMA memory
allocations in gralloc and elsewhere, then drop out the DMM/TILER
support from OMAPDRM, since it never really belonged there in
the first place (being a IOMMU unrelated to the display/GPU).

Thanks,
Andrew


> Regards,
> Ivo
> 
>> Thanks,
>> Yongqin Liu
>>
>>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>>> TILER.
>>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>>> that
>>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>>> have
>>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>>> there
>>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>>
>>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>>> allocated)
>>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>>
>>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>>
>>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>>> be contiguous?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>>> buffers are allocated as scanout:
>>>>>>>>>
>>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>>
>>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>>
>>>>>>>>> Ivo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

HI, Andrew

Thanks a lot for the information! And great to have you here!

Hi, Ivaylo

With the code provided by Andrew, could you please help give suggestions
on how to modify it in the gralloc lib side?

to add the OMAP_BO_SCANOUT flag unconditionally as OMAP_BO_MEM_CONTIG?

Thanks,
Yongqin Liu

On Mon, 29 Aug 2022 at 22:36, Andrew Davis <afd@ti.com> wrote:
>
> On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
> > Hi,
> >
> >
> > On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
> >> Hi, Ivaylo
> >>
> >> Sorry for the late response, and Thanks very much for the detailed explanations!
> >>
> >> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
> >> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> >>>> Hi, Ivaylo
> >>>>
> >>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> >>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>
> >>>>> Hi Liu,
> >>>>>
> >>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> >>>>>> Hi, IvayIo
> >>>>>>
> >>>>>> Thanks very much for the reply!
> >>>>>>
> >>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> >>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi Liu,
> >>>>>>>
> >>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>>>>>>> Hi, Ivaylo, Tomi
> >>>>>>>>
> >>>>>>>> We have one X15 Android AOSP master build, it could not have the home
> >>>>>>>> screen displayed
> >>>>>>>> on the hdmi monitor connected with this change, with the following
> >>>>>>>> message printed on the serial console
> >>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>>>>>>
> >>>>>>>>        # for details, please check the link here: http://ix.io/47m1
> >>>>>>>>
> >>>>>>>> It will work with home screen displayed on the hdmi monitor if this
> >>>>>>>> change is reverted.
> >>>>>>>>
> >>>>>>>> Is this the broken problem you talked about here?
> >>>>>>>>
> >>>>>>>> And could you please give some suggestions on how to have the x15
> >>>>>>>> Android build work with this change?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
> >>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >>>>>>> omap_bo_new().
> >>>>>>
> >>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> >>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> >>>>>> it what you meant here?
> >>>>>>
> >>>>>
> >>>>> Yes, calling this function from userspace ends in kernel code the
> >>>>> $subject patch is part of.
> >>>>>
> >>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> >>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> >>>>>> before it calls omap_bo_new_impl?
> >>>>>>
> >>>>>
> >>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
> >>>>> code that uses it (whoever it is, I am not familiar with the userspace
> >>>>> you are using) that shall pass correct flags (third parameter) when
> >>>>> calling it.
> >>>>
> >>>> Sorry, I do not get the point here.
> >>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> >>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> >>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> >>>> missing the flag by some code,
> >>>> and also avoids hacks/changes on the possible blob binaries.
> >>>>
> >>>> Do I misunderstand somewhere?
> >>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> >>>> omap_bo_new is called?
> >>>>
> >>>
> >>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
> >>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
> >>> screen), which is not always the case - there is no need offscreen
> >>> buffers or pixmaps to be scanout capable, for example. There are more
> >>> cases like that.
> >>>
> >>> The problem is that scanout buffer on OMAP4 allocate additional
> >>> resources in DMM/TILER (a piece of hardware) and those resources are
> >>> limited. Not only that, but DMM/TILER memory space eventually gets
> >>> fragmented over time (if you have lots of allocataoins/deallocations)
> >>> and you will start getting ENOMEM (or similar) errors.
> >>>
> >>> Ofc, in your particular use case you may never hit such issues.
> >>
> >> Thanks, I understand the cases now.
> >>
> >>
> >>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
> >>>>> might use ioctls directly and not call omap_bo_xxx functions.
> >>>>
> >>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> >>>> There is no place in the AOSP tree to call that except the
> >>>> omap_bo_new_impl function,
> >>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
> >>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> >>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> >>>>
> >>>> Regarding to the omap_bo_new function, there are 2 places call it in
> >>>> the AOSP tree:
> >>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> >>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> >>>>
> >>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> >>>> have the source for.
> >>>>
> >>>
> >>> I would bet on gralloc.am57x.so.
> >> yeah, that's my guess as well.
> >>
> >>>>> strace
> >>>>> would be your friend there. or gdb, or whatever tools are used on
> >>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
> >>>>> the calling process, etc.
> >>>>
> >>>> Thanks a lot for these great suggestions! Will use them when possible.
> >>>>
> >>>>>> And another question is that, since the userspace(libdrm) will be used
> >>>>>> to work with different kernel versions,
> >>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
> >>>>>> pass  OMAP_BO_SCANOUT
> >>>>>> from the userspace side with the old kernels(which does not have this change)?
> >>>>>> does this change need to be backported to the old kernel versions?
> >>>>>
> >>>>> There should not be any issue. The changes could be backported if one
> >>>>> hits the issues this $series is fixing, but there is no need.
> >>>>
> >>>> Thanks for the confirmation!
> >>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> >>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
> >>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> >>>>
> >>>
> >>> sure, the point is that with this change *every* BO will be allocated as
> >>> scanout BO, potentially leading to the above explained issues.
> >>
> >> get it.
> >>
> >>>>>>
> >>>>>> And the last question is that, omap_bo_new might be called by some
> >>>>>> property binaries what not everyone
> >>>>>> could get the source to update, for such case what's your suggestions?
> >>>>>>
> >>>>>
> >>>>> Hard to say without knowing what that library would be.
> >>>>>
> >>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
> >>>>> to fix the issue, example:
> >>>>>
> >>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
> >>>>>
> >>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> >>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> >>>>> API, instead of OMAP BO API.
> >>>>>
> >>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
> >>>>> in similar way. Or binary patch.
> >>>>
> >>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> >>>> any suggestions with it?
> >>>> # sorry, I am not able to find out how you did the reverse-engineer
> >>>> work# with the dbm repository shared here,
> >>>> # not sure if you could give some tutorial steps for the similar
> >>>> reverse-engineer# work with gralloc.am57x.so
> >>>>
> >>>
> >>> Sorry, but it is like if you ask me to provide you with a tutorial on
> >>> how to do brain surgery :)
> >>>
> >>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> >>>>
> >>>
> >>> I investigated this a bit and it seems it calls omap_bo_new() in a
> >>> wrapper function like:
> >>>
> >>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
> >>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
> >>>
> >>> Didn't investigate further what param5 is, but it controls if
> >>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
> >>>
> >>> However, this library was not made with upstream kernel in mind, as
> >>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
> >>>
> >>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
> >>>
> >>> @Tomi - any comment?
> >>>
> >>> So, you have couple of options:
> >>>
> >>> 1. Ask TI for upstream-compatible library.
> >> check is in progress, but it would take quite a long time I guess
> >>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
> >> hmm, sounds like one impossible thing...
> >>> 3. Modify omap_bo_new() to something like:
> >>> .
> >>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
> >>> .
> >>> if (flags & OMAP_BO_MEM_CONTIG)
> >>>     flags |= OMAP_BO_SCANOUT;
> >>> .
> >>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
> >>> but should make it work, at least.
> >>
> >> This looks like the only doable thing at the moment, maybe one change
> >> needs to be submitted to the mesa/drm repository.
> >> I can submit a request on your #3 change to the mesa/drm repository
> >> for discussion after some check if you do not mind.
> >>
> >
> > I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
> >
>
>
> Hi all,
>
> Just got around to reading this thread. I work with the TI gralloc lib
> and can generate new versions as needed (I was probably the one who compiled
> the version you have now). I've wanted to have our gralloc layer open source'd
> as there is nothing really propriety in it (and I re-wrote a lot of it already)
> and to avoid issues like this. But it interacts with the GPU code in some places,
> so it's up to Imagination :(. The actual code in question if it helps is:
>
>         if(ui32Flags & PVRSRV_MEM_CACHED)
>                 flags &= ~OMAP_BO_CACHE_MASK;
>         else
>                 flags |= OMAP_BO_WC;
>
>         if (ui32Flags & PVRSRV_HAP_CONTIG)
>                 flags |= OMAP_BO_SCANOUT;
>
>         flags &= ~OMAP_BO_TILED_MASK;
>         flags |= 0x00000008;
>         flags |= OMAP_BO_WC;
>
>         bo = omap_bo_new(dev, size, flags);
>
> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
> This was a hack added since even non-scanout buffers sometimes need
> to be contiguous (video decoder surfaces), but we had no way back
> then to communicate this to the gralloc layer. I think your best
> bet would be to modify the gralloc lib to not do that, or put it
> under the CONTIG check.
>
> If you tell me what the code should look like, I can rebuild the
> lib and post a copy.
>
> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
> allocations in gralloc and elsewhere, then drop out the DMM/TILER
> support from OMAPDRM, since it never really belonged there in
> the first place (being a IOMMU unrelated to the display/GPU).
>
> Thanks,
> Andrew
>
>
> > Regards,
> > Ivo
> >
> >> Thanks,
> >> Yongqin Liu
> >>
> >>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>>>>>>> TILER.
> >>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>>>>>>> that
> >>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>>>>>>> have
> >>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>>>>>>> there
> >>>>>>>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>>>>>>
> >>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>>>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>>>>>>> allocated)
> >>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>>>>>>> windows soon results in allocation failures.
> >>>>>>>>>>>
> >>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>>>>>>> like, for scanout buffers.
> >>>>>>>>>>
> >>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>>>>>>> be contiguous?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>>>>>>> buffers are allocated as scanout:
> >>>>>>>>>
> >>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>>>>>>
> >>>>>>>>> If you mean omap_bo allocated buffers, then if users want
> >>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>>>>>>> OMAP_BO_SCANOUT.
> >>>>>>>>>
> >>>>>>>>> Ivo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Yongqin Liu]

HI, Andrew

Thanks a lot for the information! And great to have you here!

Hi, Ivaylo

With the code provided by Andrew, could you please help give suggestions
on how to modify it in the gralloc lib side?

to add the OMAP_BO_SCANOUT flag unconditionally as OMAP_BO_MEM_CONTIG?

Thanks,
Yongqin Liu

On Mon, 29 Aug 2022 at 22:36, Andrew Davis <afd@ti.com> wrote:
>
> On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
> > Hi,
> >
> >
> > On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
> >> Hi, Ivaylo
> >>
> >> Sorry for the late response, and Thanks very much for the detailed explanations!
> >>
> >> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
> >> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
> >>>> Hi, Ivaylo
> >>>>
> >>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
> >>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>
> >>>>> Hi Liu,
> >>>>>
> >>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
> >>>>>> Hi, IvayIo
> >>>>>>
> >>>>>> Thanks very much for the reply!
> >>>>>>
> >>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
> >>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi Liu,
> >>>>>>>
> >>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
> >>>>>>>> Hi, Ivaylo, Tomi
> >>>>>>>>
> >>>>>>>> We have one X15 Android AOSP master build, it could not have the home
> >>>>>>>> screen displayed
> >>>>>>>> on the hdmi monitor connected with this change, with the following
> >>>>>>>> message printed on the serial console
> >>>>>>>>         [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
> >>>>>>>>         [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
> >>>>>>>>         [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
> >>>>>>>>         [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
> >>>>>>>>
> >>>>>>>>        # for details, please check the link here: http://ix.io/47m1
> >>>>>>>>
> >>>>>>>> It will work with home screen displayed on the hdmi monitor if this
> >>>>>>>> change is reverted.
> >>>>>>>>
> >>>>>>>> Is this the broken problem you talked about here?
> >>>>>>>>
> >>>>>>>> And could you please give some suggestions on how to have the x15
> >>>>>>>> Android build work with this change?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
> >>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
> >>>>>>> omap_bo_new().
> >>>>>>
> >>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
> >>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
> >>>>>> it what you meant here?
> >>>>>>
> >>>>>
> >>>>> Yes, calling this function from userspace ends in kernel code the
> >>>>> $subject patch is part of.
> >>>>>
> >>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
> >>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
> >>>>>> before it calls omap_bo_new_impl?
> >>>>>>
> >>>>>
> >>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
> >>>>> code that uses it (whoever it is, I am not familiar with the userspace
> >>>>> you are using) that shall pass correct flags (third parameter) when
> >>>>> calling it.
> >>>>
> >>>> Sorry, I do not get the point here.
> >>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
> >>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
> >>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
> >>>> missing the flag by some code,
> >>>> and also avoids hacks/changes on the possible blob binaries.
> >>>>
> >>>> Do I misunderstand somewhere?
> >>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
> >>>> omap_bo_new is called?
> >>>>
> >>>
> >>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
> >>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
> >>> screen), which is not always the case - there is no need offscreen
> >>> buffers or pixmaps to be scanout capable, for example. There are more
> >>> cases like that.
> >>>
> >>> The problem is that scanout buffer on OMAP4 allocate additional
> >>> resources in DMM/TILER (a piece of hardware) and those resources are
> >>> limited. Not only that, but DMM/TILER memory space eventually gets
> >>> fragmented over time (if you have lots of allocataoins/deallocations)
> >>> and you will start getting ENOMEM (or similar) errors.
> >>>
> >>> Ofc, in your particular use case you may never hit such issues.
> >>
> >> Thanks, I understand the cases now.
> >>
> >>
> >>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
> >>>>> might use ioctls directly and not call omap_bo_xxx functions.
> >>>>
> >>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
> >>>> There is no place in the AOSP tree to call that except the
> >>>> omap_bo_new_impl function,
> >>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
> >>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
> >>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
> >>>>
> >>>> Regarding to the omap_bo_new function, there are 2 places call it in
> >>>> the AOSP tree:
> >>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
> >>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
> >>>>
> >>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
> >>>> have the source for.
> >>>>
> >>>
> >>> I would bet on gralloc.am57x.so.
> >> yeah, that's my guess as well.
> >>
> >>>>> strace
> >>>>> would be your friend there. or gdb, or whatever tools are used on
> >>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
> >>>>> the calling process, etc.
> >>>>
> >>>> Thanks a lot for these great suggestions! Will use them when possible.
> >>>>
> >>>>>> And another question is that, since the userspace(libdrm) will be used
> >>>>>> to work with different kernel versions,
> >>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
> >>>>>> pass  OMAP_BO_SCANOUT
> >>>>>> from the userspace side with the old kernels(which does not have this change)?
> >>>>>> does this change need to be backported to the old kernel versions?
> >>>>>
> >>>>> There should not be any issue. The changes could be backported if one
> >>>>> hits the issues this $series is fixing, but there is no need.
> >>>>
> >>>> Thanks for the confirmation!
> >>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
> >>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
> >>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
> >>>>
> >>>
> >>> sure, the point is that with this change *every* BO will be allocated as
> >>> scanout BO, potentially leading to the above explained issues.
> >>
> >> get it.
> >>
> >>>>>>
> >>>>>> And the last question is that, omap_bo_new might be called by some
> >>>>>> property binaries what not everyone
> >>>>>> could get the source to update, for such case what's your suggestions?
> >>>>>>
> >>>>>
> >>>>> Hard to say without knowing what that library would be.
> >>>>>
> >>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
> >>>>> to fix the issue, example:
> >>>>>
> >>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
> >>>>>
> >>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
> >>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
> >>>>> API, instead of OMAP BO API.
> >>>>>
> >>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
> >>>>> in similar way. Or binary patch.
> >>>>
> >>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
> >>>> any suggestions with it?
> >>>> # sorry, I am not able to find out how you did the reverse-engineer
> >>>> work# with the dbm repository shared here,
> >>>> # not sure if you could give some tutorial steps for the similar
> >>>> reverse-engineer# work with gralloc.am57x.so
> >>>>
> >>>
> >>> Sorry, but it is like if you ask me to provide you with a tutorial on
> >>> how to do brain surgery :)
> >>>
> >>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
> >>>>
> >>>
> >>> I investigated this a bit and it seems it calls omap_bo_new() in a
> >>> wrapper function like:
> >>>
> >>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
> >>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
> >>>
> >>> Didn't investigate further what param5 is, but it controls if
> >>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
> >>>
> >>> However, this library was not made with upstream kernel in mind, as
> >>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
> >>>
> >>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
> >>>
> >>> @Tomi - any comment?
> >>>
> >>> So, you have couple of options:
> >>>
> >>> 1. Ask TI for upstream-compatible library.
> >> check is in progress, but it would take quite a long time I guess
> >>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
> >> hmm, sounds like one impossible thing...
> >>> 3. Modify omap_bo_new() to something like:
> >>> .
> >>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
> >>> .
> >>> if (flags & OMAP_BO_MEM_CONTIG)
> >>>     flags |= OMAP_BO_SCANOUT;
> >>> .
> >>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
> >>> but should make it work, at least.
> >>
> >> This looks like the only doable thing at the moment, maybe one change
> >> needs to be submitted to the mesa/drm repository.
> >> I can submit a request on your #3 change to the mesa/drm repository
> >> for discussion after some check if you do not mind.
> >>
> >
> > I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
> >
>
>
> Hi all,
>
> Just got around to reading this thread. I work with the TI gralloc lib
> and can generate new versions as needed (I was probably the one who compiled
> the version you have now). I've wanted to have our gralloc layer open source'd
> as there is nothing really propriety in it (and I re-wrote a lot of it already)
> and to avoid issues like this. But it interacts with the GPU code in some places,
> so it's up to Imagination :(. The actual code in question if it helps is:
>
>         if(ui32Flags & PVRSRV_MEM_CACHED)
>                 flags &= ~OMAP_BO_CACHE_MASK;
>         else
>                 flags |= OMAP_BO_WC;
>
>         if (ui32Flags & PVRSRV_HAP_CONTIG)
>                 flags |= OMAP_BO_SCANOUT;
>
>         flags &= ~OMAP_BO_TILED_MASK;
>         flags |= 0x00000008;
>         flags |= OMAP_BO_WC;
>
>         bo = omap_bo_new(dev, size, flags);
>
> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
> This was a hack added since even non-scanout buffers sometimes need
> to be contiguous (video decoder surfaces), but we had no way back
> then to communicate this to the gralloc layer. I think your best
> bet would be to modify the gralloc lib to not do that, or put it
> under the CONTIG check.
>
> If you tell me what the code should look like, I can rebuild the
> lib and post a copy.
>
> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
> allocations in gralloc and elsewhere, then drop out the DMM/TILER
> support from OMAPDRM, since it never really belonged there in
> the first place (being a IOMMU unrelated to the display/GPU).
>
> Thanks,
> Andrew
>
>
> > Regards,
> > Ivo
> >
> >> Thanks,
> >> Yongqin Liu
> >>
> >>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
> >>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> >>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
> >>>>>>>>>>> TILER.
> >>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
> >>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
> >>>>>>>>>>> that
> >>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
> >>>>>>>>>>> have
> >>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
> >>>>>>>>>>> there
> >>>>>>>>>>> is no big enough free block in DMM/TILER space.
> >>>>>>>>>>>
> >>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
> >>>>>>>>>>> starting few GUI applications (so buffers for their windows are
> >>>>>>>>>>> allocated)
> >>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
> >>>>>>>>>>> windows soon results in allocation failures.
> >>>>>>>>>>>
> >>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
> >>>>>>>>>>> like, for scanout buffers.
> >>>>>>>>>>
> >>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
> >>>>>>>>>> be contiguous?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
> >>>>>>>>> buffers are allocated as scanout:
> >>>>>>>>>
> >>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
> >>>>>>>>>
> >>>>>>>>> If you mean omap_bo allocated buffers, then if users want
> >>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
> >>>>>>>>> OMAP_BO_SCANOUT.
> >>>>>>>>>
> >>>>>>>>> Ivo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>



-- 
Best Regards,
Yongqin Liu
---------------------------------------------------------------
#mailing list
linaro-android@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-android

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,

On 30.08.22 г. 18:08 ч., Yongqin Liu wrote:
> HI, Andrew
> 
> Thanks a lot for the information! And great to have you here!
> 
> Hi, Ivaylo
> 
> With the code provided by Andrew, could you please help give suggestions
> on how to modify it in the gralloc lib side?
> 
> to add the OMAP_BO_SCANOUT flag unconditionally as OMAP_BO_MEM_CONTIG?
> 

I don't think adding OMAP_BO_SCANOUT unconditionally is a good idea - we 
already agreed on why. Without having access to the whole source code, I 
would not make blind suggestions and would leave between you (as user) 
and Andrew (as a provider) to agree on what is the best way to fix the 
issue. Still, see the comments bellow.

> Thanks,
> Yongqin Liu
> 
> On Mon, 29 Aug 2022 at 22:36, Andrew Davis <afd@ti.com> wrote:
>>
>> On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
>>> Hi,
>>>
>>>
>>> On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
>>>> Hi, Ivaylo
>>>>
>>>> Sorry for the late response, and Thanks very much for the detailed explanations!
>>>>
>>>> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>>>>> Hi, Ivaylo
>>>>>>
>>>>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Liu,
>>>>>>>
>>>>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>>>>> Hi, IvayIo
>>>>>>>>
>>>>>>>> Thanks very much for the reply!
>>>>>>>>
>>>>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi Liu,
>>>>>>>>>
>>>>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>>>>
>>>>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>>>>> screen displayed
>>>>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>>>>> message printed on the serial console
>>>>>>>>>>          [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>>>>          [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>>>>          [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>>>>          [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>>>>
>>>>>>>>>>         # for details, please check the link here: http://ix.io/47m1
>>>>>>>>>>
>>>>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>>>>> change is reverted.
>>>>>>>>>>
>>>>>>>>>> Is this the broken problem you talked about here?
>>>>>>>>>>
>>>>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>>>>> Android build work with this change?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>>>>> omap_bo_new().
>>>>>>>>
>>>>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>>>>> it what you meant here?
>>>>>>>>
>>>>>>>
>>>>>>> Yes, calling this function from userspace ends in kernel code the
>>>>>>> $subject patch is part of.
>>>>>>>
>>>>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>>>>> before it calls omap_bo_new_impl?
>>>>>>>>
>>>>>>>
>>>>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>>>>> you are using) that shall pass correct flags (third parameter) when
>>>>>>> calling it.
>>>>>>
>>>>>> Sorry, I do not get the point here.
>>>>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>>>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>>>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>>>>> missing the flag by some code,
>>>>>> and also avoids hacks/changes on the possible blob binaries.
>>>>>>
>>>>>> Do I misunderstand somewhere?
>>>>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>>>>> omap_bo_new is called?
>>>>>>
>>>>>
>>>>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>>>>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>>>>> screen), which is not always the case - there is no need offscreen
>>>>> buffers or pixmaps to be scanout capable, for example. There are more
>>>>> cases like that.
>>>>>
>>>>> The problem is that scanout buffer on OMAP4 allocate additional
>>>>> resources in DMM/TILER (a piece of hardware) and those resources are
>>>>> limited. Not only that, but DMM/TILER memory space eventually gets
>>>>> fragmented over time (if you have lots of allocataoins/deallocations)
>>>>> and you will start getting ENOMEM (or similar) errors.
>>>>>
>>>>> Ofc, in your particular use case you may never hit such issues.
>>>>
>>>> Thanks, I understand the cases now.
>>>>
>>>>
>>>>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>>>>
>>>>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>>>>> There is no place in the AOSP tree to call that except the
>>>>>> omap_bo_new_impl function,
>>>>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>>>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>>>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>>>>
>>>>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>>>>> the AOSP tree:
>>>>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>>>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>>>>
>>>>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>>>>> have the source for.
>>>>>>
>>>>>
>>>>> I would bet on gralloc.am57x.so.
>>>> yeah, that's my guess as well.
>>>>
>>>>>>> strace
>>>>>>> would be your friend there. or gdb, or whatever tools are used on
>>>>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>>>>> the calling process, etc.
>>>>>>
>>>>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>>>>
>>>>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>>>>> to work with different kernel versions,
>>>>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>>>>> pass  OMAP_BO_SCANOUT
>>>>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>>>>> does this change need to be backported to the old kernel versions?
>>>>>>>
>>>>>>> There should not be any issue. The changes could be backported if one
>>>>>>> hits the issues this $series is fixing, but there is no need.
>>>>>>
>>>>>> Thanks for the confirmation!
>>>>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>>>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>>>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>>>>
>>>>>
>>>>> sure, the point is that with this change *every* BO will be allocated as
>>>>> scanout BO, potentially leading to the above explained issues.
>>>>
>>>> get it.
>>>>
>>>>>>>>
>>>>>>>> And the last question is that, omap_bo_new might be called by some
>>>>>>>> property binaries what not everyone
>>>>>>>> could get the source to update, for such case what's your suggestions?
>>>>>>>>
>>>>>>>
>>>>>>> Hard to say without knowing what that library would be.
>>>>>>>
>>>>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>>>>> to fix the issue, example:
>>>>>>>
>>>>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>>>>
>>>>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>>>>> API, instead of OMAP BO API.
>>>>>>>
>>>>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>>>>> in similar way. Or binary patch.
>>>>>>
>>>>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>>>>> any suggestions with it?
>>>>>> # sorry, I am not able to find out how you did the reverse-engineer
>>>>>> work# with the dbm repository shared here,
>>>>>> # not sure if you could give some tutorial steps for the similar
>>>>>> reverse-engineer# work with gralloc.am57x.so
>>>>>>
>>>>>
>>>>> Sorry, but it is like if you ask me to provide you with a tutorial on
>>>>> how to do brain surgery :)
>>>>>
>>>>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>>>>
>>>>>
>>>>> I investigated this a bit and it seems it calls omap_bo_new() in a
>>>>> wrapper function like:
>>>>>
>>>>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>>>>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>>>>
>>>>> Didn't investigate further what param5 is, but it controls if
>>>>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>>>>
>>>>> However, this library was not made with upstream kernel in mind, as
>>>>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>>>>
>>>>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>>>>
>>>>> @Tomi - any comment?
>>>>>
>>>>> So, you have couple of options:
>>>>>
>>>>> 1. Ask TI for upstream-compatible library.
>>>> check is in progress, but it would take quite a long time I guess
>>>>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
>>>> hmm, sounds like one impossible thing...
>>>>> 3. Modify omap_bo_new() to something like:
>>>>> .
>>>>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>>>>> .
>>>>> if (flags & OMAP_BO_MEM_CONTIG)
>>>>>      flags |= OMAP_BO_SCANOUT;
>>>>> .
>>>>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>>>>> but should make it work, at least.
>>>>
>>>> This looks like the only doable thing at the moment, maybe one change
>>>> needs to be submitted to the mesa/drm repository.
>>>> I can submit a request on your #3 change to the mesa/drm repository
>>>> for discussion after some check if you do not mind.
>>>>
>>>
>>> I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
>>>
>>
>>
>> Hi all,
>>

Hi, glad to see you are doing fine :)

>> Just got around to reading this thread. I work with the TI gralloc lib
>> and can generate new versions as needed (I was probably the one who compiled
>> the version you have now). I've wanted to have our gralloc layer open source'd
>> as there is nothing really propriety in it (and I re-wrote a lot of it already)
>> and to avoid issues like this. But it interacts with the GPU code in some places,
>> so it's up to Imagination :(. The actual code in question if it helps is:
>>
>>          if(ui32Flags & PVRSRV_MEM_CACHED)
>>                  flags &= ~OMAP_BO_CACHE_MASK;
>>          else
>>                  flags |= OMAP_BO_WC;
>>
>>          if (ui32Flags & PVRSRV_HAP_CONTIG)
>>                  flags |= OMAP_BO_SCANOUT;
>>

Why is PVRSRV_HAP_CONTIG not set if the code requesting the buffer needs 
contiguous memory? Who is responsible for setting it?

>>          flags &= ~OMAP_BO_TILED_MASK;
>>          flags |= 0x00000008;
>>          flags |= OMAP_BO_WC;
>>
>>          bo = omap_bo_new(dev, size, flags);
>>
>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>> This was a hack added since even non-scanout buffers sometimes need
>> to be contiguous (video decoder surfaces), but we had no way back

Hmm, why would video decoder need linear memory? No MMU?

>> then to communicate this to the gralloc layer. I think your best
>> bet would be to modify the gralloc lib to not do that, or put it
>> under the CONTIG check.
>>

Does that mean that now there is a way? If that's the case, then what 
needs to be fixed is the application requesting the buffer to send the 
truth about what it needs to gralloc.

>> If you tell me what the code should look like, I can rebuild the
>> lib and post a copy.
>>
>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>> support from OMAPDRM, since it never really belonged there in
>> the first place (being a IOMMU unrelated to the display/GPU).
>>

Umm, how will we rotate scanout buffers then?

>> Thanks,
>> Andrew
>>


>>
>>> Regards,
>>> Ivo
>>>
>>>> Thanks,
>>>> Yongqin Liu
>>>>
>>>>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>>>>> TILER.
>>>>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>>>>> that
>>>>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>>>>> have
>>>>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>>>>> there
>>>>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>>>>> allocated)
>>>>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>>>>
>>>>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>>>>> be contiguous?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>>>>> buffers are allocated as scanout:
>>>>>>>>>>>
>>>>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>>>>
>>>>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>>>>
>>>>>>>>>>> Ivo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

Hi,

On 30.08.22 г. 18:08 ч., Yongqin Liu wrote:
> HI, Andrew
> 
> Thanks a lot for the information! And great to have you here!
> 
> Hi, Ivaylo
> 
> With the code provided by Andrew, could you please help give suggestions
> on how to modify it in the gralloc lib side?
> 
> to add the OMAP_BO_SCANOUT flag unconditionally as OMAP_BO_MEM_CONTIG?
> 

I don't think adding OMAP_BO_SCANOUT unconditionally is a good idea - we 
already agreed on why. Without having access to the whole source code, I 
would not make blind suggestions and would leave between you (as user) 
and Andrew (as a provider) to agree on what is the best way to fix the 
issue. Still, see the comments bellow.

> Thanks,
> Yongqin Liu
> 
> On Mon, 29 Aug 2022 at 22:36, Andrew Davis <afd@ti.com> wrote:
>>
>> On 8/29/22 8:24 AM, Ivaylo Dimitrov wrote:
>>> Hi,
>>>
>>>
>>> On 29.08.22 г. 5:51 ч., Yongqin Liu wrote:
>>>> Hi, Ivaylo
>>>>
>>>> Sorry for the late response, and Thanks very much for the detailed explanations!
>>>>
>>>> On Thu, 18 Aug 2022 at 18:23, Ivaylo Dimitrov
>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 17.08.22 г. 7:52 ч., Yongqin Liu wrote:
>>>>>> Hi, Ivaylo
>>>>>>
>>>>>> On Mon, 15 Aug 2022 at 14:23, Ivaylo Dimitrov
>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Liu,
>>>>>>>
>>>>>>> On 14.08.22 г. 17:27 ч., Yongqin Liu wrote:
>>>>>>>> Hi, IvayIo
>>>>>>>>
>>>>>>>> Thanks very much for the reply!
>>>>>>>>
>>>>>>>> On Sat, 13 Aug 2022 at 14:58, Ivaylo Dimitrov
>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi Liu,
>>>>>>>>>
>>>>>>>>> On 12.08.22 г. 7:35 ч., Yongqin Liu wrote:
>>>>>>>>>> Hi, Ivaylo, Tomi
>>>>>>>>>>
>>>>>>>>>> We have one X15 Android AOSP master build, it could not have the home
>>>>>>>>>> screen displayed
>>>>>>>>>> on the hdmi monitor connected with this change, with the following
>>>>>>>>>> message printed on the serial console
>>>>>>>>>>          [  607.404205] omapdrm omapdrm.0: Failed to setup plane plane-0
>>>>>>>>>>          [  607.410522] omapdrm omapdrm.0: Failed to setup plane plane-1
>>>>>>>>>>          [  607.416381] omapdrm omapdrm.0: Failed to setup plane plane-2
>>>>>>>>>>          [  607.422088] omapdrm omapdrm.0: Failed to setup plane plane-3
>>>>>>>>>>
>>>>>>>>>>         # for details, please check the link here: http://ix.io/47m1
>>>>>>>>>>
>>>>>>>>>> It will work with home screen displayed on the hdmi monitor if this
>>>>>>>>>> change is reverted.
>>>>>>>>>>
>>>>>>>>>> Is this the broken problem you talked about here?
>>>>>>>>>>
>>>>>>>>>> And could you please give some suggestions on how to have the x15
>>>>>>>>>> Android build work with this change?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Make sure scanout (i.e. those to be displayed) buffers are actually
>>>>>>>>> allocated as such - OMAP_BO_SCANOUT flag must be set when calling
>>>>>>>>> omap_bo_new().
>>>>>>>>
>>>>>>>> I am not familiar with this area, I am sorry if I asked quite silly questions:(
>>>>>>>> I googled omap_bo_new, and found it's a function of libdrm here[1], is
>>>>>>>> it what you meant here?
>>>>>>>>
>>>>>>>
>>>>>>> Yes, calling this function from userspace ends in kernel code the
>>>>>>> $subject patch is part of.
>>>>>>>
>>>>>>>> If it's the omap_bo_new that we should pass OMAP_BO_SCANOUT when it is called,
>>>>>>>> then is it the correct way to update omap_bo_new to add the OMAP_BO_SCANOUT flag
>>>>>>>> before it calls omap_bo_new_impl?
>>>>>>>>
>>>>>>>
>>>>>>> omap_bo_new() is fine and does not need any updates/fixes, it is the
>>>>>>> code that uses it (whoever it is, I am not familiar with the userspace
>>>>>>> you are using) that shall pass correct flags (third parameter) when
>>>>>>> calling it.
>>>>>>
>>>>>> Sorry, I do not get the point here.
>>>>>> Like you said, the code that calls omap_bo_new needs to pass OMAP_BO_SCANOUT,
>>>>>> then IMO omap_bo_new should be the best place to add the OMAP_BO_SCANOUT flag,
>>>>>> (like via flags = flags | OMAP_BO_SCANOUT), that could help avoid
>>>>>> missing the flag by some code,
>>>>>> and also avoids hacks/changes on the possible blob binaries.
>>>>>>
>>>>>> Do I misunderstand somewhere?
>>>>>> Or is there some case that OMAP_BO_SCANOUT shouldn't be passed when
>>>>>> omap_bo_new is called?
>>>>>>
>>>>>
>>>>> Exactly. You need to pass OMAP_BO_SCANOUT only when you want your
>>>>> buffers to be 'scanout' buffers(i.e. buffers that can be displayed on
>>>>> screen), which is not always the case - there is no need offscreen
>>>>> buffers or pixmaps to be scanout capable, for example. There are more
>>>>> cases like that.
>>>>>
>>>>> The problem is that scanout buffer on OMAP4 allocate additional
>>>>> resources in DMM/TILER (a piece of hardware) and those resources are
>>>>> limited. Not only that, but DMM/TILER memory space eventually gets
>>>>> fragmented over time (if you have lots of allocataoins/deallocations)
>>>>> and you will start getting ENOMEM (or similar) errors.
>>>>>
>>>>> Ofc, in your particular use case you may never hit such issues.
>>>>
>>>> Thanks, I understand the cases now.
>>>>
>>>>
>>>>>>> BTW you shall really find who and how uses OMAP BO API, in theory it
>>>>>>> might use ioctls directly and not call omap_bo_xxx functions.
>>>>>>
>>>>>> Do you mean the DRM_OMAP_GEM_NEW ioctl api?
>>>>>> There is no place in the AOSP tree to call that except the
>>>>>> omap_bo_new_impl function,
>>>>>> which is called by the omap_bo_new and omap_bo_new_tiled functions.
>>>>>> The omap_bo_new should not be called with the OMAP_BO_TILED flag,
>>>>>> while the omap_bo_new_tiled should be called with the OMAP_BO_TILED flag
>>>>>>
>>>>>> Regarding to the omap_bo_new function, there are 2 places call it in
>>>>>> the AOSP tree:
>>>>>> #1 ./external/libkmsxx/kms++/src/omap/omapframebuffer.cpp
>>>>>> #2 ./device/ti/beagle_x15/gpu/gralloc.am57x.so
>>>>>>
>>>>>> #1 seems not used in AOSP yet, and #2 is one blob binary we do not
>>>>>> have the source for.
>>>>>>
>>>>>
>>>>> I would bet on gralloc.am57x.so.
>>>> yeah, that's my guess as well.
>>>>
>>>>>>> strace
>>>>>>> would be your friend there. or gdb, or whatever tools are used on
>>>>>>> android. Or put some printfs() in omap_bo_new() that output the PID of
>>>>>>> the calling process, etc.
>>>>>>
>>>>>> Thanks a lot for these great suggestions! Will use them when possible.
>>>>>>
>>>>>>>> And another question is that, since the userspace(libdrm) will be used
>>>>>>>> to work with different kernel versions,
>>>>>>>> like the old 4.14, 4.19, etc, do you think there will be problem to
>>>>>>>> pass  OMAP_BO_SCANOUT
>>>>>>>> from the userspace side with the old kernels(which does not have this change)?
>>>>>>>> does this change need to be backported to the old kernel versions?
>>>>>>>
>>>>>>> There should not be any issue. The changes could be backported if one
>>>>>>> hits the issues this $series is fixing, but there is no need.
>>>>>>
>>>>>> Thanks for the confirmation!
>>>>>> I just boot-tested with adding OMAP_BO_SCANOUT in the omap_bo_new function,
>>>>>> and it worked with the current 4.14, 4.19, and the mainline kernels.
>>>>>> # via adding line "flags = flags | OMAP_BO_SCANOUT" in the omap_bo_new function.
>>>>>>
>>>>>
>>>>> sure, the point is that with this change *every* BO will be allocated as
>>>>> scanout BO, potentially leading to the above explained issues.
>>>>
>>>> get it.
>>>>
>>>>>>>>
>>>>>>>> And the last question is that, omap_bo_new might be called by some
>>>>>>>> property binaries what not everyone
>>>>>>>> could get the source to update, for such case what's your suggestions?
>>>>>>>>
>>>>>>>
>>>>>>> Hard to say without knowing what that library would be.
>>>>>>>
>>>>>>> When I hit issues with closed blobs, sometimes I reverse-engineer them
>>>>>>> to fix the issue, example:
>>>>>>>
>>>>>>> https://github.com/maemo-leste/sgx-ddk-um/tree/master/dbm
>>>>>>>
>>>>>>> This is REed libdbm from sgx-ddk-um 1.17.4948957, that is responsible
>>>>>>> for allocating BOs (what omap_bo_new() does) but it uses DUMB buffers
>>>>>>> API, instead of OMAP BO API.
>>>>>>>
>>>>>>> I guess you are using some older version of sgx-ddk-um, so you may fix
>>>>>>> in similar way. Or binary patch.
>>>>>>
>>>>>> The blob binary that calls omap_bo_new is the gralloc.am57x.so here[2]:
>>>>>> any suggestions with it?
>>>>>> # sorry, I am not able to find out how you did the reverse-engineer
>>>>>> work# with the dbm repository shared here,
>>>>>> # not sure if you could give some tutorial steps for the similar
>>>>>> reverse-engineer# work with gralloc.am57x.so
>>>>>>
>>>>>
>>>>> Sorry, but it is like if you ask me to provide you with a tutorial on
>>>>> how to do brain surgery :)
>>>>>
>>>>>> [2]: https://android.googlesource.com/device/ti/beagle-x15/+/refs/heads/master/gpu/gralloc.am57x.so
>>>>>>
>>>>>
>>>>> I investigated this a bit and it seems it calls omap_bo_new() in a
>>>>> wrapper function like:
>>>>>
>>>>> bo = omap_bo_new(dev, -page_size & (size + page_size - 1), ((param5 &
>>>>> 0x800000) != 0) | OMAP_BO_WC | OMAP_BO_MEM_CONTIG);
>>>>>
>>>>> Didn't investigate further what param5 is, but it controls if
>>>>> OMAP_BO_SCANOUT is passed to omap_bo_new or not.
>>>>>
>>>>> However, this library was not made with upstream kernel in mind, as
>>>>> AFAIK OMAP_BO_MEM_CONTIG never made it upstream:
>>>>>
>>>>> https://yhbt.net/lore/all/2580272.MiZDHyRxZo@avalon/T/
>>>>>
>>>>> @Tomi - any comment?
>>>>>
>>>>> So, you have couple of options:
>>>>>
>>>>> 1. Ask TI for upstream-compatible library.
>>>> check is in progress, but it would take quite a long time I guess
>>>>> 2. Try to push OMAP_BO_MEM_CONTIG patch upstream.
>>>> hmm, sounds like one impossible thing...
>>>>> 3. Modify omap_bo_new() to something like:
>>>>> .
>>>>> #define OMAP_BO_MEM_CONTIG      0x00000008      /* only use contiguous dma mem */
>>>>> .
>>>>> if (flags & OMAP_BO_MEM_CONTIG)
>>>>>      flags |= OMAP_BO_SCANOUT;
>>>>> .
>>>>> This will not achieve exactly what OMAP_BO_MEM_CONTIG is supposed to do,
>>>>> but should make it work, at least.
>>>>
>>>> This looks like the only doable thing at the moment, maybe one change
>>>> needs to be submitted to the mesa/drm repository.
>>>> I can submit a request on your #3 change to the mesa/drm repository
>>>> for discussion after some check if you do not mind.
>>>>
>>>
>>> I doubt mesa/drm will accept such hack, I think you will need to support your drm clone (with the above fix) until TI fixes the closed library.
>>>
>>
>>
>> Hi all,
>>

Hi, glad to see you are doing fine :)

>> Just got around to reading this thread. I work with the TI gralloc lib
>> and can generate new versions as needed (I was probably the one who compiled
>> the version you have now). I've wanted to have our gralloc layer open source'd
>> as there is nothing really propriety in it (and I re-wrote a lot of it already)
>> and to avoid issues like this. But it interacts with the GPU code in some places,
>> so it's up to Imagination :(. The actual code in question if it helps is:
>>
>>          if(ui32Flags & PVRSRV_MEM_CACHED)
>>                  flags &= ~OMAP_BO_CACHE_MASK;
>>          else
>>                  flags |= OMAP_BO_WC;
>>
>>          if (ui32Flags & PVRSRV_HAP_CONTIG)
>>                  flags |= OMAP_BO_SCANOUT;
>>

Why is PVRSRV_HAP_CONTIG not set if the code requesting the buffer needs 
contiguous memory? Who is responsible for setting it?

>>          flags &= ~OMAP_BO_TILED_MASK;
>>          flags |= 0x00000008;
>>          flags |= OMAP_BO_WC;
>>
>>          bo = omap_bo_new(dev, size, flags);
>>
>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>> This was a hack added since even non-scanout buffers sometimes need
>> to be contiguous (video decoder surfaces), but we had no way back

Hmm, why would video decoder need linear memory? No MMU?

>> then to communicate this to the gralloc layer. I think your best
>> bet would be to modify the gralloc lib to not do that, or put it
>> under the CONTIG check.
>>

Does that mean that now there is a way? If that's the case, then what 
needs to be fixed is the application requesting the buffer to send the 
truth about what it needs to gralloc.

>> If you tell me what the code should look like, I can rebuild the
>> lib and post a copy.
>>
>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>> support from OMAPDRM, since it never really belonged there in
>> the first place (being a IOMMU unrelated to the display/GPU).
>>

Umm, how will we rotate scanout buffers then?

>> Thanks,
>> Andrew
>>


>>
>>> Regards,
>>> Ivo
>>>
>>>> Thanks,
>>>> Yongqin Liu
>>>>
>>>>>>>>>> On Thu, 17 Feb 2022 at 23:29, Ivaylo Dimitrov
>>>>>>>>>> <ivo.g.dimitrov.75@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 17.02.22 г. 14:46 ч., Tomi Valkeinen wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>>>>>>>>>>>> On devices with DMM, all allocations are done through either DMM or
>>>>>>>>>>>>> TILER.
>>>>>>>>>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>>>>>>>>>> to fail before actual free memory is exhausted. What is even worse is
>>>>>>>>>>>>> that
>>>>>>>>>>>>> with time DMM/TILER space gets fragmented to the point that even if we
>>>>>>>>>>>>> have
>>>>>>>>>>>>> enough free DMM/TILER space and free memory, allocation fails because
>>>>>>>>>>>>> there
>>>>>>>>>>>>> is no big enough free block in DMM/TILER space.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>>>>>>>>>> starting few GUI applications (so buffers for their windows are
>>>>>>>>>>>>> allocated)
>>>>>>>>>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>>>>>>>>>> windows soon results in allocation failures.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>>>>>>>>>> like, for scanout buffers.
>>>>>>>>>>>>
>>>>>>>>>>>> Doesn't this break users that get a buffer from omapdrm and expect it to
>>>>>>>>>>>> be contiguous?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If you mean dumb buffer, then no, this does not break users as dumb
>>>>>>>>>>> buffers are allocated as scanout:
>>>>>>>>>>>
>>>>>>>>>>> https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/omapdrm/omap_gem.c#L603
>>>>>>>>>>>
>>>>>>>>>>> If you mean omap_bo allocated buffers, then if users want
>>>>>>>>>>> linear(scanout) buffer, then they request it explicitly by passing
>>>>>>>>>>> OMAP_BO_SCANOUT.
>>>>>>>>>>>
>>>>>>>>>>> Ivo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
> 
> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tomi Valkeinen]

On 30/08/2022 21:08, Ivaylo Dimitrov wrote:

>>>          flags &= ~OMAP_BO_TILED_MASK;
>>>          flags |= 0x00000008;
>>>          flags |= OMAP_BO_WC;
>>>
>>>          bo = omap_bo_new(dev, size, flags);
>>>
>>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>>> This was a hack added since even non-scanout buffers sometimes need
>>> to be contiguous (video decoder surfaces), but we had no way back
> 
> Hmm, why would video decoder need linear memory? No MMU?

Not sure about this case, but many/most IPs don't have MMU. E.g. CSI-2 
or parallel capture.

>>> If you tell me what the code should look like, I can rebuild the
>>> lib and post a copy.
>>>
>>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>>> support from OMAPDRM, since it never really belonged there in
>>> the first place (being a IOMMU unrelated to the display/GPU).
>>>
> 
> Umm, how will we rotate scanout buffers then?

Didn't we discuss this earlier in this thread. Or some other thread. 
Related to VRFB... I'm not sure =).

Anyway, neither VRFB nor DMM/TILER are part of the DSS. They're part of 
the memory subsystem. They can be used without DSS being in the setup. 
Thus the code for VRFB and DMM/TILER should not be in the DSS driver.

The DSS driver should still, of course, support DMM/TILER (and maybe 
VRFB some day) in the "use" sense, i.e. so that DSS can use the 
DMM/TILER provided from another driver.

But how exactly that's to be implemented, I don't know.

  Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tomi Valkeinen]

On 30/08/2022 21:08, Ivaylo Dimitrov wrote:

>>>          flags &= ~OMAP_BO_TILED_MASK;
>>>          flags |= 0x00000008;
>>>          flags |= OMAP_BO_WC;
>>>
>>>          bo = omap_bo_new(dev, size, flags);
>>>
>>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>>> This was a hack added since even non-scanout buffers sometimes need
>>> to be contiguous (video decoder surfaces), but we had no way back
> 
> Hmm, why would video decoder need linear memory? No MMU?

Not sure about this case, but many/most IPs don't have MMU. E.g. CSI-2 
or parallel capture.

>>> If you tell me what the code should look like, I can rebuild the
>>> lib and post a copy.
>>>
>>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>>> support from OMAPDRM, since it never really belonged there in
>>> the first place (being a IOMMU unrelated to the display/GPU).
>>>
> 
> Umm, how will we rotate scanout buffers then?

Didn't we discuss this earlier in this thread. Or some other thread. 
Related to VRFB... I'm not sure =).

Anyway, neither VRFB nor DMM/TILER are part of the DSS. They're part of 
the memory subsystem. They can be used without DSS being in the setup. 
Thus the code for VRFB and DMM/TILER should not be in the DSS driver.

The DSS driver should still, of course, support DMM/TILER (and maybe 
VRFB some day) in the "use" sense, i.e. so that DSS can use the 
DMM/TILER provided from another driver.

But how exactly that's to be implemented, I don't know.

  Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 30.08.22 г. 21:23 ч., Tomi Valkeinen wrote:
> On 30/08/2022 21:08, Ivaylo Dimitrov wrote:
> 
>>>>          flags &= ~OMAP_BO_TILED_MASK;
>>>>          flags |= 0x00000008;
>>>>          flags |= OMAP_BO_WC;
>>>>
>>>>          bo = omap_bo_new(dev, size, flags);
>>>>
>>>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>>>> This was a hack added since even non-scanout buffers sometimes need
>>>> to be contiguous (video decoder surfaces), but we had no way back
>>
>> Hmm, why would video decoder need linear memory? No MMU?
> 
> Not sure about this case, but many/most IPs don't have MMU. E.g. CSI-2 
> or parallel capture.
> 
>>>> If you tell me what the code should look like, I can rebuild the
>>>> lib and post a copy.
>>>>
>>>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>>>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>>>> support from OMAPDRM, since it never really belonged there in
>>>> the first place (being a IOMMU unrelated to the display/GPU).
>>>>
>>
>> Umm, how will we rotate scanout buffers then?
> 
> Didn't we discuss this earlier in this thread. Or some other thread. 
> Related to VRFB... I'm not sure =).
> 

Yeah, I think so. VRFB is still on my list though, along with TE support 
for droid4 :).

> Anyway, neither VRFB nor DMM/TILER are part of the DSS. They're part of 
> the memory subsystem. They can be used without DSS being in the setup. 
> Thus the code for VRFB and DMM/TILER should not be in the DSS driver.
>

Makes sense.

> The DSS driver should still, of course, support DMM/TILER (and maybe 
> VRFB some day) in the "use" sense, i.e. so that DSS can use the 
> DMM/TILER provided from another driver.
> 

Ah, this is what I was missing all the time, for some reason I was left 
with the impression that userland will have to know about those details. 
Now it is clear.

> But how exactly that's to be implemented, I don't know.
> 

Seems Andrew has an idea.

Ivo

>   Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 30.08.22 г. 21:23 ч., Tomi Valkeinen wrote:
> On 30/08/2022 21:08, Ivaylo Dimitrov wrote:
> 
>>>>          flags &= ~OMAP_BO_TILED_MASK;
>>>>          flags |= 0x00000008;
>>>>          flags |= OMAP_BO_WC;
>>>>
>>>>          bo = omap_bo_new(dev, size, flags);
>>>>
>>>> As you can see we use 0x00000008 (OMAP_BO_MEM_CONTIG) unconditionally.
>>>> This was a hack added since even non-scanout buffers sometimes need
>>>> to be contiguous (video decoder surfaces), but we had no way back
>>
>> Hmm, why would video decoder need linear memory? No MMU?
> 
> Not sure about this case, but many/most IPs don't have MMU. E.g. CSI-2 
> or parallel capture.
> 
>>>> If you tell me what the code should look like, I can rebuild the
>>>> lib and post a copy.
>>>>
>>>> Long term, I'd like to start using DMA-BUF Heaps for CMA memory
>>>> allocations in gralloc and elsewhere, then drop out the DMM/TILER
>>>> support from OMAPDRM, since it never really belonged there in
>>>> the first place (being a IOMMU unrelated to the display/GPU).
>>>>
>>
>> Umm, how will we rotate scanout buffers then?
> 
> Didn't we discuss this earlier in this thread. Or some other thread. 
> Related to VRFB... I'm not sure =).
> 

Yeah, I think so. VRFB is still on my list though, along with TE support 
for droid4 :).

> Anyway, neither VRFB nor DMM/TILER are part of the DSS. They're part of 
> the memory subsystem. They can be used without DSS being in the setup. 
> Thus the code for VRFB and DMM/TILER should not be in the DSS driver.
>

Makes sense.

> The DSS driver should still, of course, support DMM/TILER (and maybe 
> VRFB some day) in the "use" sense, i.e. so that DSS can use the 
> DMM/TILER provided from another driver.
> 

Ah, this is what I was missing all the time, for some reason I was left 
with the impression that userland will have to know about those details. 
Now it is clear.

> But how exactly that's to be implemented, I don't know.
> 

Seems Andrew has an idea.

Ivo

>   Tomi

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

it took a while until I found time to test newer kernels (mainline + Letux additions)
on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
the console was flooded by

[   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
[   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
...

making the system unuseable.

After doing some manual bisect by installing different kernel versions on the boot SD card,
I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.

After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.

Now I wonder why this patch breaks my system?

BR and thanks,
Nikolaus


> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> On devices with DMM, all allocations are done through either DMM or TILER.
> DMM/TILER being a limited resource means that such allocations will start
> to fail before actual free memory is exhausted. What is even worse is that
> with time DMM/TILER space gets fragmented to the point that even if we have
> enough free DMM/TILER space and free memory, allocation fails because there
> is no big enough free block in DMM/TILER space.
> 
> Such failures can be easily observed with OMAP xorg DDX, for example -
> starting few GUI applications (so buffers for their windows are allocated)
> and then rotating landscape<->portrait while closing and opening new
> windows soon results in allocation failures.
> 
> Fix that by mapping buffers through DMM/TILER only when really needed,
> like, for scanout buffers.
> 
> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
> ---
> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
> 1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
> index 41c1a6d..cf57179 100644
> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
> 			if (ret)
> 				goto fail;
> 
> -			if (priv->has_dmm) {
> -				ret = omap_gem_pin_tiler(obj);
> -				if (ret)
> -					goto fail;
> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
> +				if (priv->has_dmm) {
> +					ret = omap_gem_pin_tiler(obj);
> +					if (ret)
> +						goto fail;
> +				}
> 			}
> 		} else {
> 			refcount_inc(&omap_obj->pin_cnt);
> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
> 			kfree(omap_obj->sgt);
> 			omap_obj->sgt = NULL;
> 		}
> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
> +			return;
> 		if (priv->has_dmm) {
> 			ret = tiler_unpin(omap_obj->block);
> 			if (ret) {
> -- 
> 1.9.1
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

it took a while until I found time to test newer kernels (mainline + Letux additions)
on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
the console was flooded by

[   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
[   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
...

making the system unuseable.

After doing some manual bisect by installing different kernel versions on the boot SD card,
I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.

After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.

Now I wonder why this patch breaks my system?

BR and thanks,
Nikolaus


> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> On devices with DMM, all allocations are done through either DMM or TILER.
> DMM/TILER being a limited resource means that such allocations will start
> to fail before actual free memory is exhausted. What is even worse is that
> with time DMM/TILER space gets fragmented to the point that even if we have
> enough free DMM/TILER space and free memory, allocation fails because there
> is no big enough free block in DMM/TILER space.
> 
> Such failures can be easily observed with OMAP xorg DDX, for example -
> starting few GUI applications (so buffers for their windows are allocated)
> and then rotating landscape<->portrait while closing and opening new
> windows soon results in allocation failures.
> 
> Fix that by mapping buffers through DMM/TILER only when really needed,
> like, for scanout buffers.
> 
> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
> ---
> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
> 1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
> index 41c1a6d..cf57179 100644
> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
> 			if (ret)
> 				goto fail;
> 
> -			if (priv->has_dmm) {
> -				ret = omap_gem_pin_tiler(obj);
> -				if (ret)
> -					goto fail;
> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
> +				if (priv->has_dmm) {
> +					ret = omap_gem_pin_tiler(obj);
> +					if (ret)
> +						goto fail;
> +				}
> 			}
> 		} else {
> 			refcount_inc(&omap_obj->pin_cnt);
> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
> 			kfree(omap_obj->sgt);
> 			omap_obj->sgt = NULL;
> 		}
> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
> +			return;
> 		if (priv->has_dmm) {
> 			ret = tiler_unpin(omap_obj->block);
> 			if (ret) {
> -- 
> 1.9.1
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

HI Nikolaus,

On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
> Hi Ivaylo,
> 
> it took a while until I found time to test newer kernels (mainline + Letux additions)
> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
> the console was flooded by

Could you elaborate on that - do you have anything on the display 
(during boot or dunno). Do you have simplefb enabled, so boot log to be 
visible on the display? Is that wayland you are trying to run? Do you 
have PVR driver enabled? Did you try to boot vanilla kernel?

> 
> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
> ...
> 

I have no idea what that error is supposed to mean. @Tony?

> making the system unuseable.
> 
> After doing some manual bisect by installing different kernel versions on the boot SD card,
> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
> 
> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
> 
> Now I wonder why this patch breaks my system?
> 

A wild guess - omap5 has some cache issues (as is visible from 
7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. 
Before the patch *all* access to the BO backing memory was done through 
TILER/DMM, mitigating the issue. After the patch, whoever tries to 
render to non-scanout buffer is doing it directly to the memory, causing 
the issue.

Another possibility - someone assumes that memory is always linear, 
which is true when it is accessed through DMM, but it is not after the 
patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume 
scatterlist memory is contiguous" patch in your PVR driver? Maybe there 
is another driver that lacks similar patch.

Regards,
Ivo

> BR and thanks,
> Nikolaus
> 
> 
>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>
>> On devices with DMM, all allocations are done through either DMM or TILER.
>> DMM/TILER being a limited resource means that such allocations will start
>> to fail before actual free memory is exhausted. What is even worse is that
>> with time DMM/TILER space gets fragmented to the point that even if we have
>> enough free DMM/TILER space and free memory, allocation fails because there
>> is no big enough free block in DMM/TILER space.
>>
>> Such failures can be easily observed with OMAP xorg DDX, for example -
>> starting few GUI applications (so buffers for their windows are allocated)
>> and then rotating landscape<->portrait while closing and opening new
>> windows soon results in allocation failures.
>>
>> Fix that by mapping buffers through DMM/TILER only when really needed,
>> like, for scanout buffers.
>>
>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>> ---
>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>> index 41c1a6d..cf57179 100644
>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>> 			if (ret)
>> 				goto fail;
>>
>> -			if (priv->has_dmm) {
>> -				ret = omap_gem_pin_tiler(obj);
>> -				if (ret)
>> -					goto fail;
>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>> +				if (priv->has_dmm) {
>> +					ret = omap_gem_pin_tiler(obj);
>> +					if (ret)
>> +						goto fail;
>> +				}
>> 			}
>> 		} else {
>> 			refcount_inc(&omap_obj->pin_cnt);
>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>> 			kfree(omap_obj->sgt);
>> 			omap_obj->sgt = NULL;
>> 		}
>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>> +			return;
>> 		if (priv->has_dmm) {
>> 			ret = tiler_unpin(omap_obj->block);
>> 			if (ret) {
>> -- 
>> 1.9.1
>>
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

HI Nikolaus,

On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
> Hi Ivaylo,
> 
> it took a while until I found time to test newer kernels (mainline + Letux additions)
> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
> the console was flooded by

Could you elaborate on that - do you have anything on the display 
(during boot or dunno). Do you have simplefb enabled, so boot log to be 
visible on the display? Is that wayland you are trying to run? Do you 
have PVR driver enabled? Did you try to boot vanilla kernel?

> 
> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
> ...
> 

I have no idea what that error is supposed to mean. @Tony?

> making the system unuseable.
> 
> After doing some manual bisect by installing different kernel versions on the boot SD card,
> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
> 
> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
> 
> Now I wonder why this patch breaks my system?
> 

A wild guess - omap5 has some cache issues (as is visible from 
7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. 
Before the patch *all* access to the BO backing memory was done through 
TILER/DMM, mitigating the issue. After the patch, whoever tries to 
render to non-scanout buffer is doing it directly to the memory, causing 
the issue.

Another possibility - someone assumes that memory is always linear, 
which is true when it is accessed through DMM, but it is not after the 
patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume 
scatterlist memory is contiguous" patch in your PVR driver? Maybe there 
is another driver that lacks similar patch.

Regards,
Ivo

> BR and thanks,
> Nikolaus
> 
> 
>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>
>> On devices with DMM, all allocations are done through either DMM or TILER.
>> DMM/TILER being a limited resource means that such allocations will start
>> to fail before actual free memory is exhausted. What is even worse is that
>> with time DMM/TILER space gets fragmented to the point that even if we have
>> enough free DMM/TILER space and free memory, allocation fails because there
>> is no big enough free block in DMM/TILER space.
>>
>> Such failures can be easily observed with OMAP xorg DDX, for example -
>> starting few GUI applications (so buffers for their windows are allocated)
>> and then rotating landscape<->portrait while closing and opening new
>> windows soon results in allocation failures.
>>
>> Fix that by mapping buffers through DMM/TILER only when really needed,
>> like, for scanout buffers.
>>
>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>> ---
>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>> index 41c1a6d..cf57179 100644
>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>> 			if (ret)
>> 				goto fail;
>>
>> -			if (priv->has_dmm) {
>> -				ret = omap_gem_pin_tiler(obj);
>> -				if (ret)
>> -					goto fail;
>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>> +				if (priv->has_dmm) {
>> +					ret = omap_gem_pin_tiler(obj);
>> +					if (ret)
>> +						goto fail;
>> +				}
>> 			}
>> 		} else {
>> 			refcount_inc(&omap_obj->pin_cnt);
>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>> 			kfree(omap_obj->sgt);
>> 			omap_obj->sgt = NULL;
>> 		}
>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>> +			return;
>> 		if (priv->has_dmm) {
>> 			ret = tiler_unpin(omap_obj->block);
>> 			if (ret) {
>> -- 
>> 1.9.1
>>
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> HI Nikolaus,
> 
> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>> Hi Ivaylo,
>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>> the console was flooded by
> 
> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?

No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...

> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?

I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled. And without your patch everything is fine on all kernels since 4.something.

Vanilla kernel can not be booted on that machine - there is not even a device tree...

> 
>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>> ...
> 
> I have no idea what that error is supposed to mean. @Tony?

A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?

> 
>> making the system unuseable.
>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>> Now I wonder why this patch breaks my system?
> 
> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
> 
> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.

Yes, it is included.

Best regards,
Nikolaus

> 
> Regards,
> Ivo
> 
>> BR and thanks,
>> Nikolaus
>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>> 
>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>> DMM/TILER being a limited resource means that such allocations will start
>>> to fail before actual free memory is exhausted. What is even worse is that
>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>> enough free DMM/TILER space and free memory, allocation fails because there
>>> is no big enough free block in DMM/TILER space.
>>> 
>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>> starting few GUI applications (so buffers for their windows are allocated)
>>> and then rotating landscape<->portrait while closing and opening new
>>> windows soon results in allocation failures.
>>> 
>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>> like, for scanout buffers.
>>> 
>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>> ---
>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>> 
>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>> index 41c1a6d..cf57179 100644
>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>> 			if (ret)
>>> 				goto fail;
>>> 
>>> -			if (priv->has_dmm) {
>>> -				ret = omap_gem_pin_tiler(obj);
>>> -				if (ret)
>>> -					goto fail;
>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>> +				if (priv->has_dmm) {
>>> +					ret = omap_gem_pin_tiler(obj);
>>> +					if (ret)
>>> +						goto fail;
>>> +				}
>>> 			}
>>> 		} else {
>>> 			refcount_inc(&omap_obj->pin_cnt);
>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>> 			kfree(omap_obj->sgt);
>>> 			omap_obj->sgt = NULL;
>>> 		}
>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>> +			return;
>>> 		if (priv->has_dmm) {
>>> 			ret = tiler_unpin(omap_obj->block);
>>> 			if (ret) {
>>> -- 
>>> 1.9.1
>>> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> HI Nikolaus,
> 
> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>> Hi Ivaylo,
>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>> the console was flooded by
> 
> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?

No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...

> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?

I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled. And without your patch everything is fine on all kernels since 4.something.

Vanilla kernel can not be booted on that machine - there is not even a device tree...

> 
>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>> ...
> 
> I have no idea what that error is supposed to mean. @Tony?

A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?

> 
>> making the system unuseable.
>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>> Now I wonder why this patch breaks my system?
> 
> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
> 
> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.

Yes, it is included.

Best regards,
Nikolaus

> 
> Regards,
> Ivo
> 
>> BR and thanks,
>> Nikolaus
>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>> 
>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>> DMM/TILER being a limited resource means that such allocations will start
>>> to fail before actual free memory is exhausted. What is even worse is that
>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>> enough free DMM/TILER space and free memory, allocation fails because there
>>> is no big enough free block in DMM/TILER space.
>>> 
>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>> starting few GUI applications (so buffers for their windows are allocated)
>>> and then rotating landscape<->portrait while closing and opening new
>>> windows soon results in allocation failures.
>>> 
>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>> like, for scanout buffers.
>>> 
>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>> ---
>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>> 
>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>> index 41c1a6d..cf57179 100644
>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>> 			if (ret)
>>> 				goto fail;
>>> 
>>> -			if (priv->has_dmm) {
>>> -				ret = omap_gem_pin_tiler(obj);
>>> -				if (ret)
>>> -					goto fail;
>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>> +				if (priv->has_dmm) {
>>> +					ret = omap_gem_pin_tiler(obj);
>>> +					if (ret)
>>> +						goto fail;
>>> +				}
>>> 			}
>>> 		} else {
>>> 			refcount_inc(&omap_obj->pin_cnt);
>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>> 			kfree(omap_obj->sgt);
>>> 			omap_obj->sgt = NULL;
>>> 		}
>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>> +			return;
>>> 		if (priv->has_dmm) {
>>> 			ret = tiler_unpin(omap_obj->block);
>>> 			if (ret) {
>>> -- 
>>> 1.9.1
>>> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
> 
> Hi Ivaylo,
> 
>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>> 
>> HI Nikolaus,
>> 
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> Hi Ivaylo,
>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>> the console was flooded by
>> 
>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
> 
> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
> 
>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
> 
> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.

Have cross-checked: my setup uses the fbdev driver.

> And without your patch everything is fine on all kernels since 4.something.
> 
> Vanilla kernel can not be booted on that machine - there is not even a device tree...
> 
>> 
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>> 
>> I have no idea what that error is supposed to mean. @Tony?
> 
> A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?
> 
>> 
>>> making the system unuseable.
>>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>>> Now I wonder why this patch breaks my system?
>> 
>> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
>> 
>> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.
> 
> Yes, it is included.
> 
> Best regards,
> Nikolaus
> 
>> 
>> Regards,
>> Ivo
>> 
>>> BR and thanks,
>>> Nikolaus
>>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>> 
>>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>>> DMM/TILER being a limited resource means that such allocations will start
>>>> to fail before actual free memory is exhausted. What is even worse is that
>>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>>> enough free DMM/TILER space and free memory, allocation fails because there
>>>> is no big enough free block in DMM/TILER space.
>>>> 
>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>> starting few GUI applications (so buffers for their windows are allocated)
>>>> and then rotating landscape<->portrait while closing and opening new
>>>> windows soon results in allocation failures.
>>>> 
>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>> like, for scanout buffers.
>>>> 
>>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>>> ---
>>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>>> 
>>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> index 41c1a6d..cf57179 100644
>>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>>> 			if (ret)
>>>> 				goto fail;
>>>> 
>>>> -			if (priv->has_dmm) {
>>>> -				ret = omap_gem_pin_tiler(obj);
>>>> -				if (ret)
>>>> -					goto fail;
>>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>>> +				if (priv->has_dmm) {
>>>> +					ret = omap_gem_pin_tiler(obj);
>>>> +					if (ret)
>>>> +						goto fail;
>>>> +				}
>>>> 			}
>>>> 		} else {
>>>> 			refcount_inc(&omap_obj->pin_cnt);
>>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>>> 			kfree(omap_obj->sgt);
>>>> 			omap_obj->sgt = NULL;
>>>> 		}
>>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>>> +			return;
>>>> 		if (priv->has_dmm) {
>>>> 			ret = tiler_unpin(omap_obj->block);
>>>> 			if (ret) {
>>>> -- 
>>>> 1.9.1
>>>> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
> 
> Hi Ivaylo,
> 
>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>> 
>> HI Nikolaus,
>> 
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> Hi Ivaylo,
>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>> the console was flooded by
>> 
>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
> 
> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
> 
>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
> 
> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.

Have cross-checked: my setup uses the fbdev driver.

> And without your patch everything is fine on all kernels since 4.something.
> 
> Vanilla kernel can not be booted on that machine - there is not even a device tree...
> 
>> 
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>> 
>> I have no idea what that error is supposed to mean. @Tony?
> 
> A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?
> 
>> 
>>> making the system unuseable.
>>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>>> Now I wonder why this patch breaks my system?
>> 
>> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
>> 
>> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.
> 
> Yes, it is included.
> 
> Best regards,
> Nikolaus
> 
>> 
>> Regards,
>> Ivo
>> 
>>> BR and thanks,
>>> Nikolaus
>>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>> 
>>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>>> DMM/TILER being a limited resource means that such allocations will start
>>>> to fail before actual free memory is exhausted. What is even worse is that
>>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>>> enough free DMM/TILER space and free memory, allocation fails because there
>>>> is no big enough free block in DMM/TILER space.
>>>> 
>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>> starting few GUI applications (so buffers for their windows are allocated)
>>>> and then rotating landscape<->portrait while closing and opening new
>>>> windows soon results in allocation failures.
>>>> 
>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>> like, for scanout buffers.
>>>> 
>>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>>> ---
>>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>>> 
>>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> index 41c1a6d..cf57179 100644
>>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>>> 			if (ret)
>>>> 				goto fail;
>>>> 
>>>> -			if (priv->has_dmm) {
>>>> -				ret = omap_gem_pin_tiler(obj);
>>>> -				if (ret)
>>>> -					goto fail;
>>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>>> +				if (priv->has_dmm) {
>>>> +					ret = omap_gem_pin_tiler(obj);
>>>> +					if (ret)
>>>> +						goto fail;
>>>> +				}
>>>> 			}
>>>> 		} else {
>>>> 			refcount_inc(&omap_obj->pin_cnt);
>>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>>> 			kfree(omap_obj->sgt);
>>>> 			omap_obj->sgt = NULL;
>>>> 		}
>>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>>> +			return;
>>>> 		if (priv->has_dmm) {
>>>> 			ret = tiler_unpin(omap_obj->block);
>>>> 			if (ret) {
>>>> -- 
>>>> 1.9.1
>>>> 
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 9:57 ч., H. Nikolaus Schaller wrote:
> 
> 
>> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
>>
>> Hi Ivaylo,
>>
>>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>
>>> HI Nikolaus,
>>>
>>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>>> Hi Ivaylo,
>>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>>> the console was flooded by
>>>
>>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
>>
>> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
>>
>>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
>>
>> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.
> 
> Have cross-checked: my setup uses the fbdev driver.
> 

omapfb and not omapdrm? or I am missing something.

>> And without your patch everything is fine on all kernels since 4.something.
>>
>> Vanilla kernel can not be booted on that machine - there is not even a device tree...
>>
>>>
>>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>>> ...
>>>
>>> I have no idea what that error is supposed to mean. @Tony?
>>
>> A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?
>>
>>>
>>>> making the system unuseable.
>>>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>>>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>>>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>>>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>>>> Now I wonder why this patch breaks my system?
>>>
>>> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
>>>
>>> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.
>>
>> Yes, it is included.
>>
>> Best regards,
>> Nikolaus
>>
>>>
>>> Regards,
>>> Ivo
>>>
>>>> BR and thanks,
>>>> Nikolaus
>>>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>>>
>>>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>> to fail before actual free memory is exhausted. What is even worse is that
>>>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>>>> enough free DMM/TILER space and free memory, allocation fails because there
>>>>> is no big enough free block in DMM/TILER space.
>>>>>
>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>> starting few GUI applications (so buffers for their windows are allocated)
>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>> windows soon results in allocation failures.
>>>>>
>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>> like, for scanout buffers.
>>>>>
>>>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>>>> ---
>>>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> index 41c1a6d..cf57179 100644
>>>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>>>> 			if (ret)
>>>>> 				goto fail;
>>>>>
>>>>> -			if (priv->has_dmm) {
>>>>> -				ret = omap_gem_pin_tiler(obj);
>>>>> -				if (ret)
>>>>> -					goto fail;
>>>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>>>> +				if (priv->has_dmm) {
>>>>> +					ret = omap_gem_pin_tiler(obj);
>>>>> +					if (ret)
>>>>> +						goto fail;
>>>>> +				}
>>>>> 			}
>>>>> 		} else {
>>>>> 			refcount_inc(&omap_obj->pin_cnt);
>>>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>>>> 			kfree(omap_obj->sgt);
>>>>> 			omap_obj->sgt = NULL;
>>>>> 		}
>>>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>>>> +			return;
>>>>> 		if (priv->has_dmm) {
>>>>> 			ret = tiler_unpin(omap_obj->block);
>>>>> 			if (ret) {
>>>>> -- 
>>>>> 1.9.1
>>>>>
>>
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 9:57 ч., H. Nikolaus Schaller wrote:
> 
> 
>> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
>>
>> Hi Ivaylo,
>>
>>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>
>>> HI Nikolaus,
>>>
>>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>>> Hi Ivaylo,
>>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>>> the console was flooded by
>>>
>>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
>>
>> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
>>
>>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
>>
>> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.
> 
> Have cross-checked: my setup uses the fbdev driver.
> 

omapfb and not omapdrm? or I am missing something.

>> And without your patch everything is fine on all kernels since 4.something.
>>
>> Vanilla kernel can not be booted on that machine - there is not even a device tree...
>>
>>>
>>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>>> ...
>>>
>>> I have no idea what that error is supposed to mean. @Tony?
>>
>> A coincidence is that the display is sometimes showing some artistic patterns. So maybe DMA accesses non-existing memory?
>>
>>>
>>>> making the system unuseable.
>>>> After doing some manual bisect by installing different kernel versions on the boot SD card,
>>>> I was able to identify that it crept in between v5.18 and v5.19-rc1. A git bisect on this
>>>> range (adding Letux patches on top of each bisect base) did reveal this patch as the first bad one.
>>>> After reverting it seems as if I can use any v5.19 .. v6.1-rc2 kernel without issues.
>>>> Now I wonder why this patch breaks my system?
>>>
>>> A wild guess - omap5 has some cache issues (as is visible from 7cb0d6c17b96b8bf3c25de2dfde4fdeb9191f4c3), which lead to the above. Before the patch *all* access to the BO backing memory was done through TILER/DMM, mitigating the issue. After the patch, whoever tries to render to non-scanout buffer is doing it directly to the memory, causing the issue.
>>>
>>> Another possibility - someone assumes that memory is always linear, which is true when it is accessed through DMM, but it is not after the patch. Do you have my "drm: pvrsgx: dmabuf import - Do not assume scatterlist memory is contiguous" patch in your PVR driver? Maybe there is another driver that lacks similar patch.
>>
>> Yes, it is included.
>>
>> Best regards,
>> Nikolaus
>>
>>>
>>> Regards,
>>> Ivo
>>>
>>>> BR and thanks,
>>>> Nikolaus
>>>>> Am 19.01.2022 um 11:23 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>>>
>>>>> On devices with DMM, all allocations are done through either DMM or TILER.
>>>>> DMM/TILER being a limited resource means that such allocations will start
>>>>> to fail before actual free memory is exhausted. What is even worse is that
>>>>> with time DMM/TILER space gets fragmented to the point that even if we have
>>>>> enough free DMM/TILER space and free memory, allocation fails because there
>>>>> is no big enough free block in DMM/TILER space.
>>>>>
>>>>> Such failures can be easily observed with OMAP xorg DDX, for example -
>>>>> starting few GUI applications (so buffers for their windows are allocated)
>>>>> and then rotating landscape<->portrait while closing and opening new
>>>>> windows soon results in allocation failures.
>>>>>
>>>>> Fix that by mapping buffers through DMM/TILER only when really needed,
>>>>> like, for scanout buffers.
>>>>>
>>>>> Signed-off-by: Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>
>>>>> ---
>>>>> drivers/gpu/drm/omapdrm/omap_gem.c | 12 ++++++++----
>>>>> 1 file changed, 8 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> index 41c1a6d..cf57179 100644
>>>>> --- a/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> +++ b/drivers/gpu/drm/omapdrm/omap_gem.c
>>>>> @@ -821,10 +821,12 @@ int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
>>>>> 			if (ret)
>>>>> 				goto fail;
>>>>>
>>>>> -			if (priv->has_dmm) {
>>>>> -				ret = omap_gem_pin_tiler(obj);
>>>>> -				if (ret)
>>>>> -					goto fail;
>>>>> +			if (omap_obj->flags & OMAP_BO_SCANOUT) {
>>>>> +				if (priv->has_dmm) {
>>>>> +					ret = omap_gem_pin_tiler(obj);
>>>>> +					if (ret)
>>>>> +						goto fail;
>>>>> +				}
>>>>> 			}
>>>>> 		} else {
>>>>> 			refcount_inc(&omap_obj->pin_cnt);
>>>>> @@ -861,6 +863,8 @@ static void omap_gem_unpin_locked(struct drm_gem_object *obj)
>>>>> 			kfree(omap_obj->sgt);
>>>>> 			omap_obj->sgt = NULL;
>>>>> 		}
>>>>> +		if (!(omap_obj->flags & OMAP_BO_SCANOUT))
>>>>> +			return;
>>>>> 		if (priv->has_dmm) {
>>>>> 			ret = tiler_unpin(omap_obj->block);
>>>>> 			if (ret) {
>>>>> -- 
>>>>> 1.9.1
>>>>>
>>
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

> Am 31.10.2022 um 10:58 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> 
> 
> On 31.10.22 г. 9:57 ч., H. Nikolaus Schaller wrote:
>>> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
>>> 
>>> Hi Ivaylo,
>>> 
>>>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>> 
>>>> HI Nikolaus,
>>>> 
>>>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>>>> Hi Ivaylo,
>>>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>>>> the console was flooded by
>>>> 
>>>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
>>> 
>>> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
>>> 
>>>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
>>> 
>>> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.
>> Have cross-checked: my setup uses the fbdev driver.
> 
> omapfb and not omapdrm? or I am missing something.

I just have "fbdev" in the xorg.conf. Certainly not the best choice but it seems to be the most generic setup without having to tweak every combination of user-space release and kernel version and hardware variant.

Section "Monitor"
                Identifier              "lcd"
                VendorName              "BOE"
                ModelName               "W677L"
                Option                  "Rotate" "right"        # use TILER rotation
                DisplaySize             111 62          # LCD size in millimeters
EndSection

Section "Device"
                Identifier              "display0"
                Driver                  "fbdev"
                Option                  "Monitor-DSI-1" "lcd"
                Option                  "HWcursor" "False"
EndSection

Of course the kernel config uses DRM_OMAP:

root@letux:~# uname -a
Linux letux 6.1.0-rc2-letux-lpae+ #11029 SMP PREEMPT Sun Oct 30 22:41:25 CET 2022 armv7l GNU/Linux
root@letux:~# 
root@letux:~# ls -l /sys/devices/platform/omapdrm.0/drm/card1/card1-DSI-1
total 0
lrwxrwxrwx 1 root root    0 Oct 31 10:03 device -> ../../card1
-r--r--r-- 1 root root 4096 Oct 31 10:03 dpms
-r--r--r-- 1 root root    0 Oct 31 10:03 edid
-r--r--r-- 1 root root 4096 Oct 31 10:03 enabled
-r--r--r-- 1 root root 4096 Oct 31 10:03 modes
drwxr-xr-x 2 root root    0 Oct 31 10:03 power
-rw-r--r-- 1 root root 4096 Oct 31 10:03 status
lrwxrwxrwx 1 root root    0 Oct 31 10:00 subsystem -> ../../../../../../class/drm
-rw-r--r-- 1 root root 4096 Oct 31 10:00 uevent
root@letux:~# 
root@letux:~# gunzip </proc/config.gz | fgrep DRM | fgrep =
CONFIG_DRM=m
CONFIG_DRM_MIPI_DSI=y
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_FBDEV_OVERALLOC=100
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_DISPLAY_HELPER=m
CONFIG_DRM_DISPLAY_HDMI_HELPER=y
CONFIG_DRM_GEM_DMA_HELPER=m
CONFIG_DRM_SCHED=m
CONFIG_DRM_I2C_NXP_TDA998X=m
CONFIG_DRM_ATMEL_HLCDC=m
CONFIG_DRM_OMAP=m
CONFIG_DRM_TILCDC=m
CONFIG_DRM_STM=m
CONFIG_DRM_PANEL=y
CONFIG_DRM_PANEL_DSI_CM=m
CONFIG_DRM_PANEL_SIMPLE=m
CONFIG_DRM_PANEL_BOE_BTL507212_W677L=m
CONFIG_DRM_PANEL_SONY_ACX565AKM=m
CONFIG_DRM_PANEL_TPO_TD028TTEC1=m
CONFIG_DRM_PANEL_TPO_TD043MTEA1=m
CONFIG_DRM_BRIDGE=y
CONFIG_DRM_PANEL_BRIDGE=y
CONFIG_DRM_DISPLAY_CONNECTOR=m
CONFIG_DRM_SIMPLE_BRIDGE=m
CONFIG_DRM_TI_TFP410=m
CONFIG_DRM_TI_TPD12S015=m
CONFIG_DRM_DW_HDMI=m
CONFIG_DRM_DW_HDMI_CEC=m
CONFIG_DRM_IMX=m
CONFIG_DRM_IMX_PARALLEL_DISPLAY=m
CONFIG_DRM_IMX_TVE=m
CONFIG_DRM_IMX_LDB=m
CONFIG_DRM_IMX_HDMI=m
CONFIG_DRM_VC4=m
CONFIG_DRM_VC4_HDMI_CEC=y
CONFIG_DRM_ETNAVIV=m
CONFIG_DRM_ETNAVIV_THERMAL=y
CONFIG_DRM_MXS=y
CONFIG_DRM_MXSFB=m
CONFIG_SGX_DRM=y
CONFIG_DRM_MXC_EPDC=m
CONFIG_DRM_LEGACY=y
CONFIG_DRM_PANEL_ORIENTATION_QUIRKS=m
CONFIG_DRM_NOMODESET=y
root@letux:~# 
root@letux:~# lsmod | fgrep drm
omapdrm               225280  4
cec                    45056  1 omapdrm
drm_kms_helper        110592  2 display_connector,omapdrm
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   380928  7 panel_boe_btl507212_w677l,ti_tpd12s015,pvrsrvkm_omap5_sgx544_116,display_connector,omapdrm,drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
root@letux:~# 

Hope this is useful information,
Nikolaus

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[H. Nikolaus Schaller]

Hi Ivaylo,

> Am 31.10.2022 um 10:58 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
> 
> 
> 
> On 31.10.22 г. 9:57 ч., H. Nikolaus Schaller wrote:
>>> Am 31.10.2022 um 08:44 schrieb H. Nikolaus Schaller <hns@goldelico.com>:
>>> 
>>> Hi Ivaylo,
>>> 
>>>> Am 31.10.2022 um 08:05 schrieb Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com>:
>>>> 
>>>> HI Nikolaus,
>>>> 
>>>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>>>> Hi Ivaylo,
>>>>> it took a while until I found time to test newer kernels (mainline + Letux additions)
>>>>> on the OMAP5 Pyra but unfortunately I did not get screen display for v6.1. Even worse,
>>>>> the console was flooded by
>>>> 
>>>> Could you elaborate on that - do you have anything on the display (during boot or dunno). Do you have simplefb enabled, so boot log to be visible on the display?
>>> 
>>> No bootlog enabled but I have some printk in the panel driver. It is initially enabled, then disabled and enabled again. Then the issues start...
>>> 
>>>> Is that wayland you are trying to run? Do you have PVR driver enabled? Did you try to boot vanilla kernel?
>>> 
>>> I have tested with Debian Stretch with standard Xorg with "omap" driver. PVR is not enabled.
>> Have cross-checked: my setup uses the fbdev driver.
> 
> omapfb and not omapdrm? or I am missing something.

I just have "fbdev" in the xorg.conf. Certainly not the best choice but it seems to be the most generic setup without having to tweak every combination of user-space release and kernel version and hardware variant.

Section "Monitor"
                Identifier              "lcd"
                VendorName              "BOE"
                ModelName               "W677L"
                Option                  "Rotate" "right"        # use TILER rotation
                DisplaySize             111 62          # LCD size in millimeters
EndSection

Section "Device"
                Identifier              "display0"
                Driver                  "fbdev"
                Option                  "Monitor-DSI-1" "lcd"
                Option                  "HWcursor" "False"
EndSection

Of course the kernel config uses DRM_OMAP:

root@letux:~# uname -a
Linux letux 6.1.0-rc2-letux-lpae+ #11029 SMP PREEMPT Sun Oct 30 22:41:25 CET 2022 armv7l GNU/Linux
root@letux:~# 
root@letux:~# ls -l /sys/devices/platform/omapdrm.0/drm/card1/card1-DSI-1
total 0
lrwxrwxrwx 1 root root    0 Oct 31 10:03 device -> ../../card1
-r--r--r-- 1 root root 4096 Oct 31 10:03 dpms
-r--r--r-- 1 root root    0 Oct 31 10:03 edid
-r--r--r-- 1 root root 4096 Oct 31 10:03 enabled
-r--r--r-- 1 root root 4096 Oct 31 10:03 modes
drwxr-xr-x 2 root root    0 Oct 31 10:03 power
-rw-r--r-- 1 root root 4096 Oct 31 10:03 status
lrwxrwxrwx 1 root root    0 Oct 31 10:00 subsystem -> ../../../../../../class/drm
-rw-r--r-- 1 root root 4096 Oct 31 10:00 uevent
root@letux:~# 
root@letux:~# gunzip </proc/config.gz | fgrep DRM | fgrep =
CONFIG_DRM=m
CONFIG_DRM_MIPI_DSI=y
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_FBDEV_OVERALLOC=100
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_DISPLAY_HELPER=m
CONFIG_DRM_DISPLAY_HDMI_HELPER=y
CONFIG_DRM_GEM_DMA_HELPER=m
CONFIG_DRM_SCHED=m
CONFIG_DRM_I2C_NXP_TDA998X=m
CONFIG_DRM_ATMEL_HLCDC=m
CONFIG_DRM_OMAP=m
CONFIG_DRM_TILCDC=m
CONFIG_DRM_STM=m
CONFIG_DRM_PANEL=y
CONFIG_DRM_PANEL_DSI_CM=m
CONFIG_DRM_PANEL_SIMPLE=m
CONFIG_DRM_PANEL_BOE_BTL507212_W677L=m
CONFIG_DRM_PANEL_SONY_ACX565AKM=m
CONFIG_DRM_PANEL_TPO_TD028TTEC1=m
CONFIG_DRM_PANEL_TPO_TD043MTEA1=m
CONFIG_DRM_BRIDGE=y
CONFIG_DRM_PANEL_BRIDGE=y
CONFIG_DRM_DISPLAY_CONNECTOR=m
CONFIG_DRM_SIMPLE_BRIDGE=m
CONFIG_DRM_TI_TFP410=m
CONFIG_DRM_TI_TPD12S015=m
CONFIG_DRM_DW_HDMI=m
CONFIG_DRM_DW_HDMI_CEC=m
CONFIG_DRM_IMX=m
CONFIG_DRM_IMX_PARALLEL_DISPLAY=m
CONFIG_DRM_IMX_TVE=m
CONFIG_DRM_IMX_LDB=m
CONFIG_DRM_IMX_HDMI=m
CONFIG_DRM_VC4=m
CONFIG_DRM_VC4_HDMI_CEC=y
CONFIG_DRM_ETNAVIV=m
CONFIG_DRM_ETNAVIV_THERMAL=y
CONFIG_DRM_MXS=y
CONFIG_DRM_MXSFB=m
CONFIG_SGX_DRM=y
CONFIG_DRM_MXC_EPDC=m
CONFIG_DRM_LEGACY=y
CONFIG_DRM_PANEL_ORIENTATION_QUIRKS=m
CONFIG_DRM_NOMODESET=y
root@letux:~# 
root@letux:~# lsmod | fgrep drm
omapdrm               225280  4
cec                    45056  1 omapdrm
drm_kms_helper        110592  2 display_connector,omapdrm
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm                   380928  7 panel_boe_btl507212_w677l,ti_tpd12s015,pvrsrvkm_omap5_sgx544_116,display_connector,omapdrm,drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
root@letux:~# 

Hope this is useful information,
Nikolaus

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tony Lindgren]

* Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
> > [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
> > [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
> > ...
> > 
> 
> I have no idea what that error is supposed to mean. @Tony?

The error above is GPMC related. It means GPMC is not properly clocked or powered
while trying to access it's IO range.

Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?

Regards,

Tony

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Tony Lindgren]

* Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
> > [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
> > [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
> > ...
> > 
> 
> I have no idea what that error is supposed to mean. @Tony?

The error above is GPMC related. It means GPMC is not properly clocked or powered
while trying to access it's IO range.

Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?

Regards,

Tony

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 12:13 ч., Tony Lindgren wrote:
> * Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>>>
>>
>> I have no idea what that error is supposed to mean. @Tony?
> 
> The error above is GPMC related. It means GPMC is not properly clocked or powered
> while trying to access it's IO range.
> 
> Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
> GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?
> 

Seems like, but I can't see how this can happen given that 
omap_fbdev_create() calls omap_gem_new() with OMAP_BO_SCANOUT. Unless 
something bad happens in omap_framebuffer_init(), which is called before 
omap_gem_pin() is called.

Regards,
Ivo

> Regards,
> 
> Tony
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 12:13 ч., Tony Lindgren wrote:
> * Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>>>
>>
>> I have no idea what that error is supposed to mean. @Tony?
> 
> The error above is GPMC related. It means GPMC is not properly clocked or powered
> while trying to access it's IO range.
> 
> Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
> GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?
> 

Seems like, but I can't see how this can happen given that 
omap_fbdev_create() calls omap_gem_new() with OMAP_BO_SCANOUT. Unless 
something bad happens in omap_framebuffer_init(), which is called before 
omap_gem_pin() is called.

Regards,
Ivo

> Regards,
> 
> Tony
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 12:13 ч., Tony Lindgren wrote:
> * Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>>>
>>
>> I have no idea what that error is supposed to mean. @Tony?
> 
> The error above is GPMC related. It means GPMC is not properly clocked or powered
> while trying to access it's IO range.
> 
> Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
> GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?

I think I have an idea:

omap_framebuffer_pin() calls omap_gem_pin() without verifying the 
returned plane->dma_addr. To me it seems the assumption is that plane 
BOs are scanout/linear, which most-probably isn't the case. I was unable 
to find who provides those BOs though (they are passed as handles to 
omap_framebuffer_create(), most-probably set by the userspace)


> 
> Regards,
> 
> Tony
> 

Re: [PATCH 3/3] drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

[Ivaylo Dimitrov]

On 31.10.22 г. 12:13 ч., Tony Lindgren wrote:
> * Ivaylo Dimitrov <ivo.g.dimitrov.75@gmail.com> [221031 06:55]:
>> On 31.10.22 г. 0:08 ч., H. Nikolaus Schaller wrote:
>>> [   39.419846] WARNING: CPU: 0 PID: 3673 at drivers/bus/omap_l3_noc.c:139 l3_interrupt_handler+0x23c/0x330
>>> [   39.429914] 44000000.l3-noc:L3 Custom Error: MASTER MPU TARGET GPMC (Idle): Data Access in Supervisor mode during Functional access
>>> ...
>>>
>>
>> I have no idea what that error is supposed to mean. @Tony?
> 
> The error above is GPMC related. It means GPMC is not properly clocked or powered
> while trying to access it's IO range.
> 
> Maybe DSS is somehow trying to access GPMC address space registers with DMA? The
> GPMC address space starts at 0. Maybe the DSS DMA address is 0 somwhere?

I think I have an idea:

omap_framebuffer_pin() calls omap_gem_pin() without verifying the 
returned plane->dma_addr. To me it seems the assumption is that plane 
BOs are scanout/linear, which most-probably isn't the case. I was unable 
to find who provides those BOs though (they are passed as handles to 
omap_framebuffer_create(), most-probably set by the userspace)


> 
> Regards,
> 
> Tony
> 

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

gentle ping

On 19.01.22 г. 12:23 ч., Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

gentle ping

On 19.01.22 г. 12:23 ч., Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 2014 bytes --]

Hi

Am 19.01.22 um 11:23 schrieb Ivaylo Dimitrov:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

With the little expertise I have with omapdrm:

Acked-by: Thomas Zimmermann <tzimmermann@suse.de>

> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Thomas Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 2014 bytes --]

Hi

Am 19.01.22 um 11:23 schrieb Ivaylo Dimitrov:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER

With the little expertise I have with omapdrm:

Acked-by: Thomas Zimmermann <tzimmermann@suse.de>

> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

Hi Ivaylo,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.

I think this is just hacking around the problem. The problem is that 
omapdrm is being used by some as a generic buffer allocator. Those users 
should be changed to use a their own allocator or a generic allocator. 
And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would be 
scanout buffers.

Or do we have a regression in the driver? My understanding is that this 
has never really worked.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

Hi Ivaylo,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.

I think this is just hacking around the problem. The problem is that 
omapdrm is being used by some as a generic buffer allocator. Those users 
should be changed to use a their own allocator or a generic allocator. 
And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would be 
scanout buffers.

Or do we have a regression in the driver? My understanding is that this 
has never really worked.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

Hi Tomi,

On 17.02.22 г. 15:03 ч., Tomi Valkeinen wrote:
> Hi Ivaylo,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>> leading to
>> various runtime allocation failures. The series enables daily usage of 
>> devices
>> without exausting limited resources like CMA or DMM space if GPU 
>> rendering is
>> needed.
>>
>> The first patch doesn't bring any functional changes, it just moves some
>> TILER/DMM related code to a separate function, to simplify the review 
>> of the
>> next two patches.
>>
>> The second patch allows off-CPU rendering to non-scanout buffers. 
>> Without that
>> patch, it is basically impossible to use the driver allocated GEM 
>> buffers on
>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>> want GPU
>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>> allocated.
>> CMA soon gets fragmented and we start seeing allocation failures. Such 
>> failres
>> in Xorg cannot be handeled gracefully, so the system is basically 
>> unusable.
>>
>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>> fragmented with time, leading to allocation failures.
> 
> I think this is just hacking around the problem. The problem is that 
> omapdrm is being used by some as a generic buffer allocator. Those users 

Well, the user of omap_bo interface I know is xf86-video-omap. Unless if 
by users you mean 'kernel users' which I know none.

I think that if 'we' are to teach xorg omap DDX (or any other user in 
that regard) to use GPU driver allocator for non-scanout buffers and 
omapdrm for scanout, it will become a mess. Not impossible though, just 
way more complicated than the $series. Also, why do omapdrm allow 
allocation of non-linear buffers and CPU (userspace) access to them, but 
refuses to export them to kernel drivers? Isn't that the whole point of 
DMABUF stuff? This is not consistent to me. The series fixes that 
inconsistency, nothing more.

> should be changed to use a their own allocator or a generic allocator. 

SGX driver/userspace has and uses its own allocator, however, I think 
there is more than that - what about TILER/VRFB? Do you say that SGX 
userspace shall be smart enough to requests TILER buffers from omapdrm 
when scanout buffer is requested and use its own allocator when not?

Actually I was thinking about something like that, and it is achievable 
now we have:

https://github.com/maemo-leste/sgx-ddk-um/blob/master/dbm/dbm.c (REed 
SGX 1.17 ddk gbm backend)

> And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would be 
> scanout buffers.
> 

And what about OMAP_BO_TILED_XX stuff? To me this is even more of a 
hack, but it is what it is.

Do I get it correctly that you want to get rid of omap_bo_new/_tiled and 
have only dumb buffers available in omapdrm? TBH this would be great, 
however I still don't see how a TILER/VRFB buffer would be allocated, 
given that flags in drm_mode_create_dumb is not used anywhere in the 
kernel(AFAIK). Unless all scanout buffers are allocated through 
TILER/VRFB (which is a good idea IMO).

> Or do we have a regression in the driver? My understanding is that this 
> has never really worked.
> 

There are couple of patches in omapdrm that change around BO flags and 
their meaning so I think there is a regression, as the same 
userspace/DDX on linux 5.9 results in only 2 linear buffers being 
allocated, but as SGX driver has different version as well, I can't be 
100% sure without going through a lengthy assessment of SGX 
driver/omapdrm code and patches since 5.9. Which I am not going to do as 
I don't see what the benefit will be.

Please consider this patch series as a fix to an inconsistency, as it is 
merely that, it does not really bring any new functionality in terms of 
what is allocated.

Thanks and regards,
Ivo

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

Hi Tomi,

On 17.02.22 г. 15:03 ч., Tomi Valkeinen wrote:
> Hi Ivaylo,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>> leading to
>> various runtime allocation failures. The series enables daily usage of 
>> devices
>> without exausting limited resources like CMA or DMM space if GPU 
>> rendering is
>> needed.
>>
>> The first patch doesn't bring any functional changes, it just moves some
>> TILER/DMM related code to a separate function, to simplify the review 
>> of the
>> next two patches.
>>
>> The second patch allows off-CPU rendering to non-scanout buffers. 
>> Without that
>> patch, it is basically impossible to use the driver allocated GEM 
>> buffers on
>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>> want GPU
>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>> allocated.
>> CMA soon gets fragmented and we start seeing allocation failures. Such 
>> failres
>> in Xorg cannot be handeled gracefully, so the system is basically 
>> unusable.
>>
>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>> fragmented with time, leading to allocation failures.
> 
> I think this is just hacking around the problem. The problem is that 
> omapdrm is being used by some as a generic buffer allocator. Those users 

Well, the user of omap_bo interface I know is xf86-video-omap. Unless if 
by users you mean 'kernel users' which I know none.

I think that if 'we' are to teach xorg omap DDX (or any other user in 
that regard) to use GPU driver allocator for non-scanout buffers and 
omapdrm for scanout, it will become a mess. Not impossible though, just 
way more complicated than the $series. Also, why do omapdrm allow 
allocation of non-linear buffers and CPU (userspace) access to them, but 
refuses to export them to kernel drivers? Isn't that the whole point of 
DMABUF stuff? This is not consistent to me. The series fixes that 
inconsistency, nothing more.

> should be changed to use a their own allocator or a generic allocator. 

SGX driver/userspace has and uses its own allocator, however, I think 
there is more than that - what about TILER/VRFB? Do you say that SGX 
userspace shall be smart enough to requests TILER buffers from omapdrm 
when scanout buffer is requested and use its own allocator when not?

Actually I was thinking about something like that, and it is achievable 
now we have:

https://github.com/maemo-leste/sgx-ddk-um/blob/master/dbm/dbm.c (REed 
SGX 1.17 ddk gbm backend)

> And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would be 
> scanout buffers.
> 

And what about OMAP_BO_TILED_XX stuff? To me this is even more of a 
hack, but it is what it is.

Do I get it correctly that you want to get rid of omap_bo_new/_tiled and 
have only dumb buffers available in omapdrm? TBH this would be great, 
however I still don't see how a TILER/VRFB buffer would be allocated, 
given that flags in drm_mode_create_dumb is not used anywhere in the 
kernel(AFAIK). Unless all scanout buffers are allocated through 
TILER/VRFB (which is a good idea IMO).

> Or do we have a regression in the driver? My understanding is that this 
> has never really worked.
> 

There are couple of patches in omapdrm that change around BO flags and 
their meaning so I think there is a regression, as the same 
userspace/DDX on linux 5.9 results in only 2 linear buffers being 
allocated, but as SGX driver has different version as well, I can't be 
100% sure without going through a lengthy assessment of SGX 
driver/omapdrm code and patches since 5.9. Which I am not going to do as 
I don't see what the benefit will be.

Please consider this patch series as a fix to an inconsistency, as it is 
merely that, it does not really bring any new functionality in terms of 
what is allocated.

Thanks and regards,
Ivo

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

On 17/02/2022 18:21, Ivaylo Dimitrov wrote:
> Hi Tomi,
> 
> On 17.02.22 г. 15:03 ч., Tomi Valkeinen wrote:
>> Hi Ivaylo,
>>
>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>>> leading to
>>> various runtime allocation failures. The series enables daily usage 
>>> of devices
>>> without exausting limited resources like CMA or DMM space if GPU 
>>> rendering is
>>> needed.
>>>
>>> The first patch doesn't bring any functional changes, it just moves some
>>> TILER/DMM related code to a separate function, to simplify the review 
>>> of the
>>> next two patches.
>>>
>>> The second patch allows off-CPU rendering to non-scanout buffers. 
>>> Without that
>>> patch, it is basically impossible to use the driver allocated GEM 
>>> buffers on
>>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>>> want GPU
>>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>>> allocated.
>>> CMA soon gets fragmented and we start seeing allocation failures. 
>>> Such failres
>>> in Xorg cannot be handeled gracefully, so the system is basically 
>>> unusable.
>>>
>>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>>> fragmented with time, leading to allocation failures.
>>
>> I think this is just hacking around the problem. The problem is that 
>> omapdrm is being used by some as a generic buffer allocator. Those users 
> 
> Well, the user of omap_bo interface I know is xf86-video-omap. Unless if 
> by users you mean 'kernel users' which I know none.
> 
> I think that if 'we' are to teach xorg omap DDX (or any other user in 
> that regard) to use GPU driver allocator for non-scanout buffers and 
> omapdrm for scanout, it will become a mess. Not impossible though, just 
> way more complicated than the $series. Also, why do omapdrm allow 
> allocation of non-linear buffers and CPU (userspace) access to them, but 
> refuses to export them to kernel drivers? Isn't that the whole point of 
> DMABUF stuff? This is not consistent to me. The series fixes that 
> inconsistency, nothing more.
> 
>> should be changed to use a their own allocator or a generic allocator. 
> 
> SGX driver/userspace has and uses its own allocator, however, I think 
> there is more than that - what about TILER/VRFB? Do you say that SGX 
> userspace shall be smart enough to requests TILER buffers from omapdrm 
> when scanout buffer is requested and use its own allocator when not?

All I'm saying is that omapdrm should not support allocating buffers 
that are not usable by the omapdrm hardware. It doesn't make any sense.

> Actually I was thinking about something like that, and it is achievable 
> now we have:
> 
> https://github.com/maemo-leste/sgx-ddk-um/blob/master/dbm/dbm.c (REed 
> SGX 1.17 ddk gbm backend)
> 
>> And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would 
>> be scanout buffers.
>>
> 
> And what about OMAP_BO_TILED_XX stuff? To me this is even more of a 
> hack, but it is what it is.

Yes, I agree, I don't think those OMAP_BO_TILED_* values should be 
exposed to userspace. But I also agree to the "it is what it is" =).

> Do I get it correctly that you want to get rid of omap_bo_new/_tiled and 
> have only dumb buffers available in omapdrm? TBH this would be great, 
> however I still don't see how a TILER/VRFB buffer would be allocated, 
> given that flags in drm_mode_create_dumb is not used anywhere in the 
> kernel(AFAIK). Unless all scanout buffers are allocated through 
> TILER/VRFB (which is a good idea IMO).

We can't get rid of those as they're userspace API.

>> Or do we have a regression in the driver? My understanding is that 
>> this has never really worked.
>>
> 
> There are couple of patches in omapdrm that change around BO flags and 
> their meaning so I think there is a regression, as the same 
> userspace/DDX on linux 5.9 results in only 2 linear buffers being 
> allocated, but as SGX driver has different version as well, I can't be 
> 100% sure without going through a lengthy assessment of SGX 
> driver/omapdrm code and patches since 5.9. Which I am not going to do as 
> I don't see what the benefit will be.
> 
> Please consider this patch series as a fix to an inconsistency, as it is 
> merely that, it does not really bring any new functionality in terms of 
> what is allocated.

I've considered, and I think I agree. The design of omapdrm + tiler is 
broken in my opinion, but it's there, it has userspace APIs, and it's 
all old code. It's probably not worth the effort to try to clean it up, 
while still somehow keeping the old userspace working.

I've had these patchesin my work branch for a while and I haven't seen 
any issues. I'll keep them there for a bit longer and I'll look at these 
patches a bit more, but I think I'll merge them at some point.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

On 17/02/2022 18:21, Ivaylo Dimitrov wrote:
> Hi Tomi,
> 
> On 17.02.22 г. 15:03 ч., Tomi Valkeinen wrote:
>> Hi Ivaylo,
>>
>> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>>> leading to
>>> various runtime allocation failures. The series enables daily usage 
>>> of devices
>>> without exausting limited resources like CMA or DMM space if GPU 
>>> rendering is
>>> needed.
>>>
>>> The first patch doesn't bring any functional changes, it just moves some
>>> TILER/DMM related code to a separate function, to simplify the review 
>>> of the
>>> next two patches.
>>>
>>> The second patch allows off-CPU rendering to non-scanout buffers. 
>>> Without that
>>> patch, it is basically impossible to use the driver allocated GEM 
>>> buffers on
>>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>>> want GPU
>>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>>> allocated.
>>> CMA soon gets fragmented and we start seeing allocation failures. 
>>> Such failres
>>> in Xorg cannot be handeled gracefully, so the system is basically 
>>> unusable.
>>>
>>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>>> fragmented with time, leading to allocation failures.
>>
>> I think this is just hacking around the problem. The problem is that 
>> omapdrm is being used by some as a generic buffer allocator. Those users 
> 
> Well, the user of omap_bo interface I know is xf86-video-omap. Unless if 
> by users you mean 'kernel users' which I know none.
> 
> I think that if 'we' are to teach xorg omap DDX (or any other user in 
> that regard) to use GPU driver allocator for non-scanout buffers and 
> omapdrm for scanout, it will become a mess. Not impossible though, just 
> way more complicated than the $series. Also, why do omapdrm allow 
> allocation of non-linear buffers and CPU (userspace) access to them, but 
> refuses to export them to kernel drivers? Isn't that the whole point of 
> DMABUF stuff? This is not consistent to me. The series fixes that 
> inconsistency, nothing more.
> 
>> should be changed to use a their own allocator or a generic allocator. 
> 
> SGX driver/userspace has and uses its own allocator, however, I think 
> there is more than that - what about TILER/VRFB? Do you say that SGX 
> userspace shall be smart enough to requests TILER buffers from omapdrm 
> when scanout buffer is requested and use its own allocator when not?

All I'm saying is that omapdrm should not support allocating buffers 
that are not usable by the omapdrm hardware. It doesn't make any sense.

> Actually I was thinking about something like that, and it is achievable 
> now we have:
> 
> https://github.com/maemo-leste/sgx-ddk-um/blob/master/dbm/dbm.c (REed 
> SGX 1.17 ddk gbm backend)
> 
>> And we could then drop the OMAP_BO_SCANOUT flag, as all buffers would 
>> be scanout buffers.
>>
> 
> And what about OMAP_BO_TILED_XX stuff? To me this is even more of a 
> hack, but it is what it is.

Yes, I agree, I don't think those OMAP_BO_TILED_* values should be 
exposed to userspace. But I also agree to the "it is what it is" =).

> Do I get it correctly that you want to get rid of omap_bo_new/_tiled and 
> have only dumb buffers available in omapdrm? TBH this would be great, 
> however I still don't see how a TILER/VRFB buffer would be allocated, 
> given that flags in drm_mode_create_dumb is not used anywhere in the 
> kernel(AFAIK). Unless all scanout buffers are allocated through 
> TILER/VRFB (which is a good idea IMO).

We can't get rid of those as they're userspace API.

>> Or do we have a regression in the driver? My understanding is that 
>> this has never really worked.
>>
> 
> There are couple of patches in omapdrm that change around BO flags and 
> their meaning so I think there is a regression, as the same 
> userspace/DDX on linux 5.9 results in only 2 linear buffers being 
> allocated, but as SGX driver has different version as well, I can't be 
> 100% sure without going through a lengthy assessment of SGX 
> driver/omapdrm code and patches since 5.9. Which I am not going to do as 
> I don't see what the benefit will be.
> 
> Please consider this patch series as a fix to an inconsistency, as it is 
> merely that, it does not really bring any new functionality in terms of 
> what is allocated.

I've considered, and I think I agree. The design of omapdrm + tiler is 
broken in my opinion, but it's there, it has userspace APIs, and it's 
all old code. It's probably not worth the effort to try to clean it up, 
while still somehow keeping the old userspace working.

I've had these patchesin my work branch for a while and I haven't seen 
any issues. I'll keep them there for a bit longer and I'll look at these 
patches a bit more, but I think I'll merge them at some point.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

Hi,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

I have pushed this to drm-misc-next.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Tomi Valkeinen]

Hi,

On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
> This patch series fixes excessive DMM or CMA usage of GEM buffers leading to
> various runtime allocation failures. The series enables daily usage of devices
> without exausting limited resources like CMA or DMM space if GPU rendering is
> needed.
> 
> The first patch doesn't bring any functional changes, it just moves some
> TILER/DMM related code to a separate function, to simplify the review of the
> next two patches.
> 
> The second patch allows off-CPU rendering to non-scanout buffers. Without that
> patch, it is basically impossible to use the driver allocated GEM buffers on
> OMAP3 for anything else but a basic CPU rendered examples as if we want GPU
> rendering, we must allocate buffers as scanout buffers, which are CMA allocated.
> CMA soon gets fragmented and we start seeing allocation failures. Such failres
> in Xorg cannot be handeled gracefully, so the system is basically unusable.
> 
> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
> fragmented with time, leading to allocation failures.
> 
> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
> 
> Ivaylo Dimitrov (3):
>    drm: omapdrm: simplify omap_gem_pin
>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
> 
>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 +++++++++++++++++-------------
>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>   3 files changed, 116 insertions(+), 90 deletions(-)
> 

I have pushed this to drm-misc-next.

  Tomi

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

Hi,

On 28.03.22 г. 12:46 ч., Tomi Valkeinen wrote:
> Hi,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>> leading to
>> various runtime allocation failures. The series enables daily usage of 
>> devices
>> without exausting limited resources like CMA or DMM space if GPU 
>> rendering is
>> needed.
>>
>> The first patch doesn't bring any functional changes, it just moves some
>> TILER/DMM related code to a separate function, to simplify the review 
>> of the
>> next two patches.
>>
>> The second patch allows off-CPU rendering to non-scanout buffers. 
>> Without that
>> patch, it is basically impossible to use the driver allocated GEM 
>> buffers on
>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>> want GPU
>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>> allocated.
>> CMA soon gets fragmented and we start seeing allocation failures. Such 
>> failres
>> in Xorg cannot be handeled gracefully, so the system is basically 
>> unusable.
>>
>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>> fragmented with time, leading to allocation failures.
>>
>> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
>> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
>>
>> Ivaylo Dimitrov (3):
>>    drm: omapdrm: simplify omap_gem_pin
>>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
>>
>>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 
>> +++++++++++++++++-------------
>>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>>   3 files changed, 116 insertions(+), 90 deletions(-)
>>
> 
> I have pushed this to drm-misc-next.
> 

Great, next is VRFB and TE support for Droid4 panel, as soon as I find 
some spare time :)

Ivo

Re: [PATCH 0/3] drm: omapdrm: Fix excessive GEM buffers DMM/CMA usage

[Ivaylo Dimitrov]

Hi,

On 28.03.22 г. 12:46 ч., Tomi Valkeinen wrote:
> Hi,
> 
> On 19/01/2022 12:23, Ivaylo Dimitrov wrote:
>> This patch series fixes excessive DMM or CMA usage of GEM buffers 
>> leading to
>> various runtime allocation failures. The series enables daily usage of 
>> devices
>> without exausting limited resources like CMA or DMM space if GPU 
>> rendering is
>> needed.
>>
>> The first patch doesn't bring any functional changes, it just moves some
>> TILER/DMM related code to a separate function, to simplify the review 
>> of the
>> next two patches.
>>
>> The second patch allows off-CPU rendering to non-scanout buffers. 
>> Without that
>> patch, it is basically impossible to use the driver allocated GEM 
>> buffers on
>> OMAP3 for anything else but a basic CPU rendered examples as if we 
>> want GPU
>> rendering, we must allocate buffers as scanout buffers, which are CMA 
>> allocated.
>> CMA soon gets fragmented and we start seeing allocation failures. Such 
>> failres
>> in Xorg cannot be handeled gracefully, so the system is basically 
>> unusable.
>>
>> Third patch fixes similar issue on OMAP4/5, where DMM/TILER spaces get
>> fragmented with time, leading to allocation failures.
>>
>> Series were tested on Motolola Droid4 and Nokia N900, with OMAP DDX and
>> PVR EXA from https://github.com/maemo-leste/xf86-video-omap
>>
>> Ivaylo Dimitrov (3):
>>    drm: omapdrm: simplify omap_gem_pin
>>    drm: omapdrm: Support exporting of non-contiguous GEM BOs
>>    drm: omapdrm: Do no allocate non-scanout GEMs through DMM/TILER
>>
>>   drivers/gpu/drm/omapdrm/omap_gem.c        | 198 
>> +++++++++++++++++-------------
>>   drivers/gpu/drm/omapdrm/omap_gem.h        |   3 +-
>>   drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c |   5 +-
>>   3 files changed, 116 insertions(+), 90 deletions(-)
>>
> 
> I have pushed this to drm-misc-next.
> 

Great, next is VRFB and TE support for Droid4 panel, as soon as I find 
some spare time :)

Ivo