Today I had a pretty unexpected problem. In a full-fledged “wtf” moment, I suddenly realized that our cubemap reflection shaders have been utterly wrong throughout the entire development cycle. I don’t mean just a little incorrect, something that is a bit hard to discern. I mean like this wrong:

Fubarred cubemap

Yeah. Don’t ask me how, or why. I can’t believe that no one has noticed this, or if they have, didn’t say anything! A bit annoyed at the interruption in my workflow (I thought I was about to whip out some animated self-illumination for our reflective metal shader), I immediately set out to fix the problem. Ehem… I mean problemssssss. With lots of s’s.

The first problem which I solved pretty quickly was that the eye vector was being normalized in the vertex shader. This caused plenty of artifacts due to the normals not being interpolated correctly across the surface.

The reflection improved a lot, but it was sideways on the floor. At first it seemed to me like the cubemap was just at the wrong orientation, and I set out trying to rotate it in code. I couldn’t get this to work for the life of me, but I suddenly realized I was only able to flip the reflection on the x and y axis relative to the surface normal. Aha! Not only was the eye vector being normalized in the vertex shader, but it was being transformed to tangent space as well. Easy fix.

Unfortunately, since nobody noticed before, the pixel shader was full of “reactionary” patchwork code that was wrong but fixed the above problems with the reflection under very specific circumstances. Once these were fixed everything seemed to work great!

Then I noticed that the fresnel effect was behaving oddly. It was locked to an axis. Doh! I forgot that I actually needed the eye vector in tangent space for the fresnel calculation. It was getting late in the day, so I created a new variable and passed both the world space and tangent space eye vectors into the pixel shader. I’m not sure if this is the best way to handle needing both (I probably could have done the whole fresnel part in world space as well), but I think it’s cheaper to calculate the two vectors in the vertex shader and pass them through to the pixel shader than to transform one vector in the pixel shader. I’ll probably take a look at this some more tomorrow.