Given a point $P = P_0·s + P_1·t + P_2·(1-s-t)$ in the triangle $(P_0, P_1, P_2)$, the pixel in the triangle is only flipped if $(\frac s2 + t)^2 < t.$ Otherwise the pixel is left alone. This formula is from a Microsoft Research paper by Charles Loop and Jim Blinn and can be derived from the definition of a quadratic segment. The $s$ and $t$ values are known as Barycentric coordinates and are a common way of defining a coordinate system on a triangle.
