Shinobi Shader System

Shaders for my Shinobi engine are written in Lua programming language. Why Lua? Lua with its dynamic typing and first class functions makes it easy to generate shader permutations -- a hard problem to solve in a C like language with limited preprocessor power. Secondly, a custom shader compiler can target different backends such as HLSL and GLSL automatically.

Below is a simple tonemapping shader from Shinobi which shows the basic syntax (which is actually unmodified Lua syntax). The shader is actually a shader bundle because it generates four different versions of the tonemapping shader when executed by returning a table of shaders.

import "common.lua" 

texture_2d "source_buffer" : slot(0) 

uniform_block "tonemap" : slot(3)
: float "exposure"
: float "saturation" 

function tonemap_linear(color)
return color
end 

function tonemap_exponential(color)
return float3(1.0, 1.0, 1.0) - exp(-color)
end 

function tonemap_reinhard(color)
local white = 0.8
color = color * (1 + color / (white*white)) / (1.0 + color)
return color
end 

function tonemap_filmic(color)
local A = 0.15
local B = 0.50
local C = 0.10
local D = 0.20
local E = 0.02
local F = 0.30
local W = 11.2
local exposure_bias = 2.0
local v = color * exposure_bias
local color = ((v * (A * v + C * B) + D * E) / (v * (A * v + B) + D * F)) - E / F
local white = ((W * (A * W + C * B) + D * E) / (W * (A * W + B) + D * F)) - E / F
color = color / white
return color
end 

function tonemap(func)
local color = tex_load(source_buffer, sv_screen_pos()).xyz
color = color * exposure
color = func(color)
color = pow(color, 1.0/2.0)
color = saturation(color, saturation)
out.color = float4(color, 1.0)
end 

local shaders = {}
for _,func in ipairs{"linear", "exponential", "reinhard", "filmic"} do
shaders[func] = link_shader(compile_ps(tonemap, _G["tonemap_"..func]))
end 

return shaders

Another neat feature is automatic generation of shader input and output declarations between shader stages. For example, a simple mesh shader with support for optional skinning could look something like the following. Note how the fetch_xxx() functions automatically collect attributes for shader input and output declarations and check that the shader signatures match.

import "common.lua" 

function fetch_mesh_vertex()
local position = fetch_float4("position")
local normal = fetch_float4("normal")
local tangent = fetch_float4("tangent")
local texcoord = fetch_float2("texcoord")
local v = {}
v.position = position
v.normal = (normal.xyz - float3(0.5, 0.5, 0.5)) * 2.0
v.tangent = (tangent.xyz - float3(0.5, 0.5, 0.5)) * 2.0
v.bitangent = cross(v.normal, v.tangent) * ((tangent.w - 0.5) * 2.0)
v.texcoord = texcoord
return v
end 

function fetch_skinned_vertex()
local v = fetch_mesh_vertex()
v.bone_indices = fetch_int4("bone_indices")
v.bone_weights = fetch_float4("bone_weights")
return v
end 

function vs(skinning)
local v
if skinning then
v = fetch_skinned_vertex()
v = skin_transform(v)
else
v = fetch_mesh_vertex()
end
out.sv_position = transform_vec(v.position, mvp_matrix)
out.texcoord = v.texcoord
end 

function ps()
local texcoord = fetch_float2("texcoord")
out.color = tex_sample(diffuse_map, diffuse_sampler, texcoord)
end 

return {
static_mesh = link_shader(compile_vs(vs, false), compile_ps(ps)),
skinned_mesh = link_shader(compile_vs(vs, true), compile_ps(ps)),
}

The shader compiler is written in Lua. The compiler is a Lua environment with overloaded math operators, type constructors and intrinsics loaded in. The shader code is executed in this environment which generates the output shader code in one pass. Currently the compiler has backends for HLSL and GLSL, less than 200 lines of Lua code each. The compiler itself is around 2300 lines including function library for intrinsics. The most complex part of the compiler is typechecker with does full typechecking for function and operator args so that if there is an error I get a Lua stacktrace with line numbers pointing to the original Lua shader code instead of some confusing errors in the generated HLSL/GLSL code.

Of course there are some drawbacks:

1. Because of dynamic typing you can't clearly see the types of variables by looking at the shader code. Personally this is not a big deal to me as I'm used to dynamic typing (I have written a lot of Lua code). I can always sprinkle the code with type assertions or use some form of Hungarian notation for variable names if I wanted.

2. Dynamic branching can't be expressed as Lua statements because Lua does not have a feature for overloading statement like syntax. The solution I'm currently using is to use functional (Lisp) style. For example, ifs are implemented using Lambdas like this: _if(expression, true-lambda, false-lambda). Lua has sensible scoping rules for variables and full support for lambdas so this is not that bad. A nicer C-like syntax would be possible by forking Lua codebase and adding some language extension but I haven't had time to look at this yet. Luckily the Lua interpreter code is well structured and very lightweight so making a language extension shouldn't be that hard.

3. For similar reasons logical and comparison operators are hardwired in Lua and can't be overloaded for code generation. Again this can be worked around with Lisp style syntax. For example, "mag.x > mag.y && mag.x > mag.z" becomes "_and(greater(mag.x, mag.y), greater(mag.x, mag.z))". C like syntax should be possible by hacking the Lua interpreter. The idea would be to add full metamethod support for these operators and disable short circuit evaluation of logical operators.