Temporal Anti-aliasing
Preamble
One of my first tasks as a graphics programmer was to improve their TAA implementation. The main issue was constant flickering on stationary camera shots. I started with zero knowledge and no one at the company had much information, the shader was almost a decade old. The first thing I did was read through the shader to try and understand where the flickering was occuring and see if I could just tweak a value to reduce it. Probably not the best first step, I quickly realised I had a lot of research to do, and that most of this job is actually research!
Quickly Explain
TAA is essentially jiggling the camera every frame, then blurring/blending them together to produce a smooth looking image. Aliased pixels are where geometry (and other) detail is unable to be represented on the low resolution screen. It lands between pixels (subpixel) and cannot be perfectly represented, causing jagged edges. By jiggling (or jittering) the camera a tiny subpixel amount these jagged edges will snap back and forth between different pixel locations. By blending these subpixel movements every frame the image will blur these edges over time, resulting in a smooth image.
The main strength of TAA over other screen space anti-aliasing techniques is that it is temporally stable, during camera motion the image should still be smooth and free of aliasing. Unlike FXAA or SMAA (although there are SMAA+TAA techniques).
Cheat Sheet
Just want the answers? Every link I think is useful and a quick glossary. In retrospect it is easy to find this info, but at the time without a working lexicon it is difficult.
Glossary
- Current (frame) - the newly rendered aliased image, probably the output of our geometry/lighting passes.
- History (frame) - a texture containing the previous frame's TAA output. A smooth image we sample to compare against our current frame.
- Reprojection - calculating where the current pixel was in the previous frame (history).
- Subpixel - the place between the discrete pixels on of the screen.
- Jitter - moving the camera's translation, usually by a subpixel amount.
- Ghosting - when TAA blends too much during camera movement, causing big smears across the screen.
- Disocclusion - when an object moves and reveals surfaces that were not previously visible and therefore would not exist in history which can cause ghosting.
- Neighbourhood - the surround pixels of the current pixel we are rendering (usually the current frame). Eg 3x3 pixels surrounding the current.
- Neighbourhood Clamping - clamping the history colour into the colour bounds (min/max) of the current neighbourhood. Reduces ghosting.
- Neighbourhood Clipping - instead of just clamping to the bounds, drag the the history colour towards the current colour until it is within the bounding box. Reduce ghosting further.
- Varience Clipping - build a more precise colour bounds to clip by using the mean and the variance of the current neighbourhood. Reduces ghosting EVEN further.
- Velocity Buffer - a texture which stores the motion vectors or velocity of each pixel eg. the movement of the pixel compared to the last frame.
Research Links
- Playdead's Temporal Reprojection Anti-Aliasing in Inside: this a great overview and implementation, but a bit stylized since the game had a very unique aesthetic.
- Unreal Engine 4's TAA presentation: more focused on refining TAA for higher quality results through better HDR management and neighbourhood clipping.
- Nvidia TAA Overview: nice overview, with a GREAT look at varience clipping!
- Alex Tardif's TAA Starter Pack : after understanding the basics this really ties it all together into a very simple and clean implementation.
- TAA and the Holy Trail: a detailed and comprehensive guide to TAA with a good amount of implementation provided.
TAA Overview
Core Steps
So I recommend reading/watching the links above in that order but the key steps are:
-
- Store the previous frame's TAA output in a history texture (at frame 0 it will just be a copy of the current frame)
- On the next frame move the camera by a subpixel amount so rendered geometry will shift slightly.
- Sample the history texture by calculating where the current frame pixel would be in history using the previous view projection matrix.
- Blend the current frame with the history frame by some factor like 10% just a lerp(historyColour, currentColour, 0.1).
- The still image should look smooth but moving the camera will cause ghosting/blurriness due to slow blending and disocclusions.
- So before blending we must first use Varience Neighbourhood Clipping to bring history to a similar colour to our current frame colour.
- This will massively reduce ghosting on camera movement but on individual object movements we need to be able to track their old pixel location. Before we did this with just camera data, but now we need the motion vector of the object itself.
- We do this by rendering any moving object to a velocity buffer which is the calculation of the current pixel postion vs the previous.
- So when reprojecting the current pixel we also check the velocity buffer which might contain extra movement information for the location of the history pixel.
That is what I consider the core of TAA, the rest is somewhat more application specific.
Increasing Quality
-
TAA in motion can look too blurry, to help this when sampling history using a bicubic Catmull-Rom filter (using 5 samples) can increase the sharpness of the image.
-
Flickering is caused by many different things, and decreasing ghosting often causes more flickering which is why this is somewhat application specific (how important is having no ghosting is vs flickering).
When you have tiny geometry on screen like a wire fence if you jitter the camera that geometry no longer exists in the next frame, but when we jitter it for the frame after that it will exist again and pop back in causing flickering. A method briefly mentioned in UE4’s talk is by recording these impulses in another buffer (or alpha channel) and effectively decreasing that pixel’s importance. I achieved this by increasing the acceptable colour bounds in varience clipping, it would allow more ghosting in these flickering spots causing them to blend nicely.
Specular flickering where each frame the specular material causes very bright highlights to jitter across the edges of geometry, similar to the tiny geometry flickering. These can have very high HDR values which is difficult to handle, doing a tonemapping/luminance filtering step to both history and current colour (as seen in Alex Tardif’s TAA) can greatly mitigate this, this was the main solution to our own TAA flickering.
End Notes
In the end the TAA I had to fix up had many issues, and unpacking them was almost impossible. In the end I did almost a full rewrite and backtracked from there to figure out what went wrong in our previous TAA. Turns out in an effort to reduce ghosting we increased the blend factor of 10% to much larger amounts when velocity increased (camera or object motion). This maybe sounds good at first glance since taking more of the current frame will reduce ghosting and result in sharper images. But we lose the main strength of TAA that it is temporally stable, most other screen-space anti-aliasing break during camera motion, not TAA. This was effecively throwing out most of our work as soon as the camera moved. There were a number of other small issues like tonemapping incorrectly and oddities with jitter calculations, but that was the most egregious.