FPGA for Real-Time Image Processing

Real time is often used as a synonym for fast. In image processing it means something else. Real time means every frame finishes inside the time allotted to it. A system that is fast on average but occasionally late is worse than one that is slow on average and never late.

In a separate article we covered which blocks make up an imaging pipeline and how their order decides image quality. This article looks at the same pipeline from the time side: do the blocks keep up, where does latency accumulate, and how does that shape the architecture.

The answer does not live in the algorithms. What decides latency is which block holds a line and which one holds an entire frame.

Two Different Constraints: Throughput and Latency

Two measures get confused with each other in a real-time pipeline.

• Throughput is how many pixels per second you can process. For 1920x1080 at 60 fps that is 124.4 million pixels per second, and it is not open to negotiation.
• Latency is the time between a pixel entering the pipeline and leaving it. Throughput is relatively easy to reach because parallelism solves it. Latency is a consequence of the architecture; once the pipeline is built you cannot tune it with a parameter.

And in a real-time system, the constraint is not average latency but worst-case latency. A frame you miss once does not come back.

Latency Classes: A Line, or a Whole Frame?

The one-sentence version of this article: every image processing block is either line latency or frame latency, and that distinction decides system latency far more than the algorithms do.

• Line latency blocks need only a neighborhood a few lines deep to produce output. Demosaic, sharpening, spatial filtering, and color conversion all sit in this class. They accumulate as many lines as the window height, then start flowing and never stop again.
• Frame latency blocks have to see an entire frame before they can produce anything. Frame buffers, temporal filters, global histograms, and blocks that rotate the image or scale it by accumulating frames all belong here.

The difference between the two is not a tuning gap, it is an order of magnitude. At 1080p60 one line takes 15.4 microseconds, so a five-line window is roughly 77 microseconds. One full frame is 16,670 microseconds. A single frame latency block costs about two hundred times more than every line latency block put together.

The practical consequence: when you add a frame latency block, know that you are making an architectural decision. Add it if you need it, but add it knowing that most of your latency budget just went there.

Where Does the Clock Rate Come From?

The clock rate of an imaging pipeline is not guessed, it is divided out.

Divide the pixels per second by the number of pixels you process per clock cycle and you get the real beat rate of the pipeline. At one pixel per cycle, you need a 124.4 MHz clock. At four pixels per cycle, 31.1 MHz is enough. In a 150 MHz clock domain, a pipeline running four pixels per cycle uses about a fifth of the clock, and the rest is headroom for stalls.

Raising the pixels per cycle widens the datapath. Every block now has to handle four pixels at once, LUT and BRAM usage goes up, and neighborhood windows get more complicated. Lowering it pushes the clock instead; timing that will not close at 124 MHz closes comfortably at 31 MHz.

If timing does not close there are three routes. Adding a register stage to the pipeline costs a few cycles of latency but does not change the block’s class, so it is the cheapest fix. Turning on the tool’s retiming usually buys a few hundred picoseconds. The third is to raise the pixels per cycle and lower the clock; this is the most effective route and also the most expensive in resources.

Backpressure and FIFO Depth

No block in a streaming pipeline has to be ready at every moment. When the receiver is not ready the stream slows down, and that slowdown propagates upstream. FIFOs between the blocks are how you manage it.

FIFO depth is not guessed, it is computed: depth must be at least the longest stall multiplied by the stream rate. Absorbing a ten-microsecond stall in a pipeline running 31.1 million beats per second needs 311 beats, which rounds up to 512.

One fact works in your favor. In a video stream, horizontal blanking is a natural buffer. No new pixels arrive during that interval, so the downstream block can finish whatever it has queued. Starting the calculation from there shrinks the FIFOs noticeably.

An oversized FIFO is not free either. It spends BRAM, and while it stays full it adds latency of its own to the chain. The right depth is not the largest one, it is the computed one.

Not the Average, the Worst Case

What an FPGA wins in image processing is usually not raw speed. It is the absence of variance.

On a CPU or GPU the operating system scheduler, cache misses, memory contention, and other processes running at the same time make every frame take slightly longer or shorter. The average looks good; the tail of the distribution does not. Missed frames hide there.

On an FPGA the pipeline is a fixed circuit. The same frame is processed in the same number of cycles every time. Unless a block behaves in a data-dependent way, the gap between average and worst case is close to zero.

Set your acceptance criteria accordingly: not average latency, but the number of frames missed over a long run.

Where Does Latency Accumulate?

Once you decide to reduce latency the first question is where to look, and the answer is rarely the place people expect.

In a video pipeline, most of the latency accumulates in the compression and decompression steps. The network and buffering layer takes a moderate share. The compute time of the image processing blocks, because they run in parallel and produce a fixed latency per frame, stays very small next to those.

Starting your optimization with the imaging pipeline is therefore usually wasted effort. Look at encoder settings first, then at buffer depths.

Measuring Latency on Hardware

Guesswork does not survive contact with a board. There are three practical ways to measure latency on real hardware.

1. Probe points: You run a counter inside the programmable logic that never stops, and place small blocks at the critical points of the pipeline that latch its value. You read the results back over AXI4-Lite. This is the most precise method, accurate to a clock cycle, but it only sees the path inside the programmable logic.
2. Embedding a timestamp into the frame: You write the current counter value into the image itself and read it on the screen at the far end. It measures end to end and covers the encoder and the network as well.
3. Capturing packet timestamps: Done on the network side. It gives a system-level picture and includes buffering effects.

One caution: these three methods measure between different boundaries. If their numbers disagree, all three are usually correct and simply looked at different points. Write down what each one measures from and to before you compare them.

A Practical Checklist

• Count your frame latency blocks. Most of your latency budget sits there, and no other optimization can compete with it.
• Divide out the clock rate. Pixels per second over pixels per cycle; the result is the real constraint on the pipeline.
• Compute FIFO depth from the longest stall. Counting horizontal blanking shrinks the buffers noticeably.
• Tie acceptance criteria to the worst case. Average latency can look fine while the missed frames hide in the tail.
• Do not optimize before you measure. The culprit is rarely the block you think of first.
• If two methods disagree, both may be right. Compare what they measure before you compare the numbers.

Conclusion

Real-time image processing is not the job of making algorithms faster. It is the job of making every block in the chain commit to a timing promise.

That promise falls into one of two classes: the block either holds a few lines or a whole frame. Once that choice is made, the clock rate, the FIFO depths, and the acceptance criteria follow from it. When it is not made deliberately, no optimization added afterwards closes a gap of two hundred times.

Contact Us: info@run-x.com | www.run-x.com