Glass to glass latency for simple camera to display streaming

NXP i.MX8M Plus
1

Hi all,
I am in an evaluation phase for an imx8mp SOM from another vendor. My user case is streaming video from MIPI-CSI image sensor to LVDS or HDMI display, with graphic overlay. the image sensor is gray scale, so no color processing is required and no other ISP processing is needed. My main requirements are low power and low latency from sensor to display. Latency of sub frame (couple of lines) is desired.
The vendor I am evaluating now can only provide a latency of 120ms in the use case. Can I achieve better latency with solidrun im8m plus SOM? What would be the latency I can achieve?

I appreciate any comment on my question

2

120ms seems high for sensor to local HDMI, but most likely this has to do with the pipeline processing. Currently the iMX8MP’s BSP is very much designed for more complex camera processing, not raw frame rate. Even then I doubt you would be able to get sub frame latency without a lot of custom processing. The entire CSI camera pipeline is interrupt driven so would be processed in that manner and you would require a polling mechanism as well as a custom display handler to properly draw sub frame damage to the display. What are the resolutions you intend to draw at?

3

Hi,
The resolution is pretty low concidering. It is less than 1080, but more than 720.
You mentioned interrupt driven mechanism, i thought the isp process is done fully by the hardware block, without cpu intervention. Is it not the case?

4

The ISP process is, but that does not include output to the display controller.

5

My other question is are you planning on using a high refresh monitor that could display the updated data at a sub-frame latency? Even if your camera is capturing at 100fps you would still be limited if your display output is only running at 60Hz.

6

yes, we are planning on using a 90 to 120Hz display. Isn’t the display driver part of the ISP?

7

The ISP is input only, and if latency is a priority you will most likely want to bypass the ISP and only use the ISI interface. All these are part of the MediaMix domain, but are distinct hardware that do different things.

8

Hi Jon,
Thank you for your help.
This sounds exactly what I want to do, to bypass the ISP. Can I do that using GStreamer, or how else would I do it?
I do need also a graphic overlay and I am looking on cairooverlay for doing that
I am using the GST command before the overlay gst-launch-1.0 v4l2src ! waylandsink

9

This will need to be configured in the device-tree and then have custom drivers implemented or modified for your sensor. There are currently no gstreamer plugins that support line by line processing that I know of.