After GPAA, Humus presents us a new anti-aliasing technique (Direct3D 10 demo) called GBAA or Geometry Buffer Anti-Aliasing.
GBAA is another anti-aliasing technique that works in the same spirit as GPAA, with the same quality, but with a substantially different method to accomplish the results. Whereas GPAA operates entirely as a post-process step (plus optionally pre-processing edges), GBAA uses a mixed method. During main scene rendering the geometry information is stored to a separate buffer, alternatively it uses available channels in an existing buffer. Anti-Aliasing is then performed in the end, in a resolve-pass if you will, as a fullscreen pass using the stored geometric information.
The geometry is represented as a two-channel edge distance. The closest direction (horizontal/vertical) distance is stored and the other is set to zero. It is possible to pack it into one channel and only store a bit indicating the major direction, but this demo uses two channels for simplicity.
GBAA allows to anti-alias geometric edges but also alpha tested edges like in the follwing pictures:
Additionally, having geometry information in a buffer allows us to anti-alias other edges than just geometric ones, most notably alpha-tested edges. The shader just needs to output the distance to the alpha-edge, something that’s easily approximated using gradient instructions.
alpha tested edges: GBAA disabled
alpha tested edges: GBAA enabled