Shadow in "Seal Guardian"

Introduction
"Seal Guardian" uses a mix of static and dynamic shadow systems to support long range shadow to cover the whole level. "Seal Guardian" only use a single directional for the whole level, so part of the shadow information can be pre-computed. It mainly consists of 3 parts: baked static shadow on static meshes stored along with the light map, baked static shadow for dynamic objects stored along with the irradiance volume and dynamic shadow with optional ESM soft shadow.

Static shadow for static objects
During the baking process of the light map, we also compute static shadow information. We first render a shadow map for the whole level in a big render target (e.g. 8192x8192), then for each texel of light map, we can compare against its world position to the shadow map to check whether that texel is in shadow. But we are using a 1024x1024 light map for the whole scene, storing the shadow term directly will not have enough resolution. So we use distance field representation[1] to reduce storage size similar to the UDK[2]. To bake the distance field representing of the shadow term, instead of comparing a single depth value at texel world position as before, we compare several values within a 0.5m x 0.5m grid, oriented along the normal at position similar to the figure below:

Blue dots indicate the positions for sampling shadow map
to compute distance field value for the texel at red dot position.
(The gird is perpendicular to the red vertex normal of the texel.)

By doing this, we can get the shadow information around the baking texel to compute the distance field. We choose this method instead of computing the distance field from a large baked shadow texture because we want to have the shadow distance filed consistently computed in world space no matter how the mesh UV is and this can also avoid UV seam too. But this method may cause potential problem for concave mesh, but so far, for all levels in "Seal Guardian", it is not a big problem.

Static shadow only

Static shadow for dynamic objects
For dynamic objects to receive baked shadow, we baked shadow information and store it along with the irradiance volume. For each irradiance probe location, we compare it to the whole scene shadow map and get a binary shadow value. During runtime, we interpolate this binary shadow value by using the position of dynamic object and the probe location to get a smooth transition of shadow value, just like interpolating the SH coefficients of irradiance volume.

Circled objects does not have light map UV, so they are treated the same as dynamic objects and shadowed with the shadow value stored along with irradiance volume

Each small sphere is a sampling location for storing the SH coefficients and shadow value of the irradiance for dynamic objects.

Dynamic Shadow
We use standard shadow mapping algorithm with exponential shadow map(ESM)[3] to support dynamic shadow in "Seal Guardian".  However due to we need to support a variety of hardware(from iOS, Mac to PC) and minimise code complexity, we choose not to use any cascade shadow map. Instead we use a single shadow map to support dynamic shadow for a very short distance (e.g. 30m-60m) and rely on baked shadow to cover the remaining part of the scene.

Dynamic shadow mixed with static shadow

Dynamic shadow only

Shadow Quality Settings
With the above systems, we can make a few shadow quality settings:

1. mix of static shadow with dynamic ESM shadow
2. mix of static shadow with dynamic hard shadow
3. static shadow only

On iOS platform, we choose the shadow quality depends on the device capability. Besides, as we are using a forward renderer, when we are drawing objects that outside the dynamic shadow distance, those objects can use the static shadow only shader to save a bit of performance.

Soft Shadow

Hard Shadow

No Shadow

Conclusion
We have briefly describe the shadow system in "Seal Guardian", which uses distance field shadow map for static mesh shadow, interpolated static shadow value for dynamic objects and ESM dynamic shadow for a short distance. Also a few shadow quality settings can be generated with very few coding effort.

Lastly, if you are interested in "Seal Guardian", feel free to check it out and its Steam store page is live now. It will be released on 8^th Dec, 2017 on iOS/Mac/PC. Thank you.

References
[1] http://www.valvesoftware.com/publications/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf
[2] https://docs.unrealengine.com/udk/Three/DistanceFieldShadows.html
[3] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.146.177&rep=rep1&type=pdf

Light Map in "Seal Guardian"

Introduction
Light mapping is a common technique used in games for storing lighting data. "Seal Guardian" used light map in order to support large variety of hardware from iOS, Mac to PC because of its low run time cost. There are many methods to bake the light map such as photon mapping and radiosity. Our baking method is similar to radiosity hemicube[1], but we render a full cube map for each light map texel to store incoming lighting data instead.

Scene with light map

Scene without light map

Light Map Atlas
In each level, light map is built for all static meshes with a second unique UV set. We gather all those static meshes and pack them into a large light map atlas by using this method[2], others method can be chosen, we just pick a simple one.

Packing a single large light map atlas for all static mesh in the scene

Compute Light Map Texel Position
Then we render all the meshes into a RGBA32Float world position render target using the light map atlas layout created before (by a vertex shader which transform the mesh 3D world position vertex to its unique 2D light map UV). Then we query back the render target to store all the written texels which correspond to the world position of each light map texel. Those position will be used for rendering cube maps for radiosity.

Each square represent a single light map texel,
we query back those texel world space position to render cube map for radiosity

Radiosity Baking
As talked before, we use a method similar to hemicube, but rendering a full cube map instead, so we will render a cube map at each light map texel with all the post processing effect/tone mapping off and just storing the lighting data. Because our light map is intended to store the incoming indirect static lighting for each texel, we convert the incoming lighting data cube map rendered at each texel to 2nd order spherical harmonics coefficients(i.e. 4 coefficients for each RGB channels), the conversion method can be found in "Stupid Spherical Harmonics (SH) Tricks"[3]. So we will need 1 RGBA32Float(or RGBA16Float) cube map and 3 temporal RGBA32Float(or RGBA16Float) textures for each radiosity iteration.

No light map, direct lighting and emissive materials only

Lighting without albedo texture

Radiosity pass 1
In the first pass, we render all the meshes without analytical light source (e.g. directional light) into the cube map. Only the emissive material such as sky and static light placed in the scene get rendered to inject the initial lighting into the radiosity iterations. We support sphere and box shape static light which get rendered into the cube just like an emissive mesh during rendering the cube map. Once the cube map render is completed, we convert the cube map to SH coefficients and store the values. After all the texels are rendered, we will have an incoming lighting light map from emissive mesh and static light source ready for the next pass.

Light map baked with the emissive sky material

Lighting with light map using the emissive sky

Radiosity pass 2
In the second pass, we render all the meshes only with analytical light source and the SH light map from previous pass into a new cube map to calculate the first bound incoming lighting. Then convert the cube map to SH coefficients. After all the texels are rendered and converted to an SH light map, we need to sum this SH light map to the previous pass SH light map to get the accumulated lighting for pass 1 and 2 into another 3 accumulated SH light maps for our final storage (this accumulated light map is not used in the radiosity iteration, just for final radiosity output.).

Light map baked with direct lighting and emissive material

Lighting with light map using direct lighting and emissive sky

Radiosity pass >= 3
For the sub-sequence passes, we can use the SH light map from previous iteration to render the cube map and repeat the conversion to SH and accumulate SH lighting for all passes steps to get the incoming indirect lighting for each light map texels.

Final baked result, showing both direct and indrect lighting

Lighting using light map, without albedo texture

Storage Format
To store the light map data for runtime and reduce memory usage (3 SH light maps in float format, i.e. 12 values for each texels, is too much data to store...), we decompose the the incoming lighting color data to luma and chroma. We only store the luma data in SH format and compute an average chroma value by integrating the SH RGB incoming lighting data with a SH cosine transfer function along the static mesh normal direction, this will get a reflected Lambertian lighting and we use this value to compute the chroma value. By doing this, we can preserve the directional variation of indirect lighting, keeping an average color of incoming lighting and reduce the light map storage to 6 values per texels. To further reduce the memory usage, we clamp the incoming SH luma value to a predefined range so that we can store it in 8-bit texture. However, using compression like DXT will result in artifacts, so we just store the light map data in 2 RGBA8 textures.

Final light map used in run-time, storing SH luma and average chroma

Conclusion
In this post, we have briefly outlined how the light maps are created in "Seal Guardian". It is based on a modified version of radiosity hemicube and using SH as an intermediate representation for baking and reduce the storage size (by splitting the lighting data to luma and chroma.). We skipped some of the baking details like padding the lighting data for each UV shell in each radiosity iteration to avoid light leaking from empty light map texels. Also "Seal Guardian" is rendered using PBR, that means we have metallic material which doesn't work well with radiosity. Instead of converting the metallic to a diffuse material, we pre-filter all the environment probe in each radiosity pass to get the lighting for metallic material. Also, we would like to improve the light map baking in the future, like improving the baking time, fixing the compression problem, we may try BC6H (but need to find another method for iOS compression...), or using a smaller texture size for chroma light map than the luma SH light map texture...

Lastly, if you are interested in "Seal Guardian", feel free to check it out and its Steam store page is live now. It will be released on 8^th Dec, 2017 on iOS/Mac/PC. Thank you.

The yellow light bounce on the floor is done by the yellow metallic wall with pre-filtering the environment map in each radiosity pass

Showing only the indirect lighting

References
[1] https://www.siggraph.org/education/materials/HyperGraph/radiosity/overview_2.htm
[2] http://blackpawn.com/texts/lightmaps/
[3] http://www.ppsloan.org/publications/StupidSH36.pdf

"Seal Guardian" announced!!!

Finally, "Seal Guardian" is announced!!!

It has been a very long time since my last post, I was busy with making the game "Seal Guardian".

"Seal Guardian" is a hard core hack and slash action game, powered by the engine described in this blog. It took me more than 5 years to code the engine(with some help of open source libraries like Bullet physics, Lua and DLMalloc.)/gameplay and creating all the visual artwork from modelling, texturing, skinning, rigging and animation... The game will be available on 8^th December, 2017 on iOS/Mac/PC via iOS App Store/Mac App Store/Steam.

Finishing a game takes lots of effort, patient and time, especially making the whole game on your own. It contains lots of fun tasks like rendering and gameplay, but it contains even more boring tasks like game menu, localisation, UI(e.g. handling all the input UI for mouse/keyboard, touch screen, different gamepad type like PS4/XBox/MFi in different languages for different resolution), making website, prepare online store artwork like app icon, trailer video and screen shots(where different stores have different resolution requirements... :S), opening bank account(which may take more than a month for a small indie game company.). Hope I can share these in future blog posts.

Before sharing the game's postmortem and waiting for its release on 8^th December, 2017. I will write some more blog posts about the engine tech used in "Seal Guardian", e.g. light map baking processing and its storage format, how the static shadow is baked, visibility system, the cross platform rendering pipeline... So, stay tuned!

In the mean time, feel free to visit the "Seal Guardian" Steam store page and share it if you like.
Thank you very much!!! =]

Pre-Integrated Skin Shading

Introduction
Recently, I was implementing skin shading in my engine. The pre-integrated skin shading technique is chosen because it has a low performance cost and does not require another extra pass. The idea is to pre-bake the scattering effect over a ring into a texture for different curvature to look up during run-time. More information can be found in the GPU Pro 2, the SIGGRAPH slides, and also in the presentation of the game "The Oder:1886". Here is the result implemented in my engine (all screen shots are rendered with filmic tone mapping):

The head on the left is lit with Oren-Nayar shading
The head on the right is lit with pre-integrated skin

Curve Approximation for Direct Lighting
In my engine, iOS is one of my target platform, which only have 8 texture unit to use for the OpenGL ES 2.0 API. This is not enough for the pre-integrated skin look up texture because my engine have already used some slots for light map, shadow map, IBL... So I need to find an approximation to the look up texture.

Unfortunately I don't have a Mathematica license at home. So I think may be I can fit the curve manually by inspecting the shape of the curve. So, I started by plotting the graph of the equation:

Here is the shape of the red channel diffusion curve, plotting with N.L(normalized to [0, 1])and r (from 2 to 16):

My idea for approximating the curve is by finding some simple curves first and then interpolate between them like this:

For the light blue line in the above figure, a single straight line can get a close enough approximation:

curve1= saturate(1.95 * NdotL -0.96)

To approximate the dark blue line, I divide it into 2 parts: linear part and quadratic part:

curve0_linear= saturate(1.75* NdotL -0.76)

curve0_quadratic= 0.65*(NdotL^ 2)  + 0.045

blending both linear and quadratic curve will get a curve that is similar to the original function:

curve0= lerp(curve0_quadratic, curve0_linear, NdotL^2)

Now we have 2 curves that is similar to our original function at both end. By mixing them together, we can get something similar to the original function like this:

curve= lerp(curve0, curve1, 1 - (1 - curvature)^4)

By repeating the above steps for the blue and green channels, we can shade the pre-integrated skin without the look up texture. Here is the result:

Lit with a single directional light From left to right: r = 2, 4, 8, 16 Upper row: shaded with look up texture Lower row: shaded with approximated function

This is how it looks like when applying to a human head model:

From left to right: shaded with look up texture, approximated function, lambert shader Upper row: shaded with albedo texture applied Lower row: showing only lighting result

For your reference, here is the approximated function I used for the RGB channels:

NdotL = mad(NdotL, 0.5, 0.5); // map to 0 to 1 range
float curva = (1.0/mad(curvature, 0.5 - 0.0625, 0.0625) - 2.0) / (16.0 - 2.0); // curvature is within [0, 1] remap to normalized r from 2 to 16
float oneMinusCurva = 1.0 - curva;
float3 curve0;
{
    float3 rangeMin = float3(0.0, 0.3, 0.3);
    float3 rangeMax = float3(1.0, 0.7, 0.7);
    float3 offset = float3(0.0, 0.06, 0.06);
    float3 t = saturate( mad(NdotL, 1.0 / (rangeMax - rangeMin), (offset + rangeMin) / (rangeMin - rangeMax)  ) );
    float3 lowerLine = (t * t) * float3(0.65, 0.5, 0.9);
    lowerLine.r += 0.045;
    lowerLine.b *= t.b;
    float3 m = float3(1.75, 2.0, 1.97);
    float3 upperLine = mad(NdotL, m, float3(0.99, 0.99, 0.99) -m );
    upperLine = saturate(upperLine);
    float3 lerpMin = float3(0.0, 0.35, 0.35);
    float3 lerpMax = float3(1.0, 0.7 , 0.6 );
    float3 lerpT = saturate( mad(NdotL, 1.0/(lerpMax-lerpMin), lerpMin/ (lerpMin - lerpMax) ));
    curve0 = lerp(lowerLine, upperLine, lerpT * lerpT);
}
float3 curve1;
{
    float3 m = float3(1.95, 2.0, 2.0);
    float3 upperLine = mad( NdotL, m, float3(0.99, 0.99, 1.0) - m);
    curve1 = saturate(upperLine);
}
float oneMinusCurva2 = oneMinusCurva * oneMinusCurva;
float3 brdf = lerp(curve0, curve1, mad(oneMinusCurva2, -1.0 * oneMinusCurva2, 1.0) );

Curve Approximation for Indirect Lighting
In my engine, the indirect lighting is stored in spherical harmonics up to order 2. So the pre-integrated skin BRDF need to be projected into spherical harmonics coefficients and can be store in look up texture just like the presentation of "The Oder:1886" described. One thing to note about the integral range in equation (19) from the paper should be up to π instead of π/2 because to project the coefficients, we need to integrate over the whole sphere domain and the value of D(θ, r) in the lower hemi-sphere may be non zero for small r due to sub-surface scattering, which does not like the clamped cos(θ). So I compute the spherical harmonics coefficient using this:

To make the indirect lighting work on my target platform, an approximate function to this indirect lighting look up texture also need to be found. By using similar methods described above and with some trail and error. Here is my result:

Lit with both directional light and BRDF projected into SH From left to right: r = 2, 4, 8, 16 Upper row: shaded with look up texture Lower row: shaded with approximated function

And applying it to the human head model, but this time the approximation is not close enough and lose a bit of red color:

From left to right: shaded with look up texture, approximated function, lambert shader Upper row: shaded with albedo texture applied Lower row: showing only lighting result

And some code for your reference, where the ZH is the zonal harmonics coefficient:

float curva = (1.0/mad(curvature, 0.5 - 0.0625, 0.0625) - 2.0) / (16.0 - 2.0); // curvature is within [0, 1] remap to r distance 2 to 16
float oneMinusCurva = 1.0 - curva;
// ZH0
{
    float2 remappedCurva = 1.0 - saturate(curva * float2(3.0, 2.7) );
    remappedCurva *= remappedCurva;
    remappedCurva *= remappedCurva;
    float3 multiplier = float3(1.0/mad(curva, 3.2, 0.4), remappedCurva.x, remappedCurva.y);
    zh0 = mad(multiplier, float3( 0.061659, 0.00991683, 0.003783), float3(0.868938, 0.885506, 0.885400));
}
// ZH1
{
    float remappedCurva = 1.0 - saturate(curva * 2.7);    
 float3 lowerLine = mad(float3(0.197573092, 0.0117447875, 0.0040980375), (1.0f - remappedCurva * remappedCurva * remappedCurva), float3(0.7672169, 1.009236, 1.017741));
    float3 upperLine = float3(1.018366, 1.022107, 1.022232);
    zh1 = lerp(upperLine, lowerLine, oneMinusCurva * oneMinusCurva);
}

Result
Putting both the direct and indirect lighting calculation together, with a simple GGX specular, lit by 1 directional light, SH projected indirect light and pre-filtered IBL.

Heads from left to right: shaded with lambert shader, approximated function, look up texture Images from left to right: full shading, direct lighting only, indirect lighting only Upper row: shaded with albedo texture applied Lower row: showing only lighting result

With another lighting environment:

Heads from left to right: shaded with look up texture, approximated function, lambert shader Images from left to right: full shading, direct lighting only, indirect lighting only Upper row: shaded with albedo texture applied Lower row: showing only lighting result

I have uploaded the curvature map for the human head here, look up texture for the direct lighting here and indirect lighting look up texture here. The textures need to be loaded without sRGB conversion. For the indirect lighting texture, I have scale the value so that it fits into an 8-bit texture within 0 to 1 range. So a sample use of the look up textures looks like:

float3 brdf= directBRDF.Sample(samplerLinearClamp, float2(mad(NdotL, 0.5, 0.5), oneOverR)).rgb;
float3 zh0= indirectBRDF_ZH.Sample(samplerLinearClamp, float2(oneOverR, 0.25)).rgb;
float3 zh1= indirectBRDF_ZH.Sample(samplerLinearClamp, float2(oneOverR, 0.75)).rgb;
float remapMin= 0.75;
float remapMax= 1.05;
zh0= zh0 * (remapMax - remapMin) + remapMin;
zh1= zh1 * (remapMax - remapMin) + remapMin;

Conclusion
In this post, I describe a way to find an approximated function for the pre-integrated skin diffusion profile, which gives a similar result for the direct lighting function while losing a bit of red color for the indirect lighting. The down side of fitting the curve manually is when the function is changed a bit, say changing the function input from radial distance to curvature(i.e. from r to 1/r), all the approximate functions need be re-do again (or the conversion need to be done during run-time just like my code snippet above...). Also the shadow scattering described in the original paper is not implemented, so some artifact may be seen at the direct shadow boundary. Overall, the skin shading result is improved compare to shade with Lambert or Oren-Nayar under environment with a strong directional light source.

Reference
[1] SIGGRAPH 2011- Pre-Integrated Skin Shading
http://advances.realtimerendering.com/s2011/Penner%20-%20Pre-Integrated%20Skin%20Rendering%20(Siggraph%202011%20Advances%20in%20Real-Time%20Rendering%20Course).pptx
[2] GPU Pro 2- Pre-Integrated Skin Shading http://www.amazon.com/GPU-Pro-2-Wolfgang-Engel/dp/1568817185
[3]  Crafting a Next-Gen Material Pipeline for The Order: 1886 http://blog.selfshadow.com/publications/s2013-shading-course/rad/s2013_pbs_rad_notes.pdf
[4] GPU Gem 3- Advanced Techniques for Realistic Real-Time Skin Rendering http://http.developer.nvidia.com/GPUGems3/gpugems3_ch14.html
[5] Mathematica and Skin Rendering http://c0de517e.blogspot.jp/2011/09/mathematica-and-skin-rendering.html
[6] Addendum to Mathematica and Skin Rendering http://c0de517e.blogspot.jp/2011/09/mathematica-and-skin-rendering.html
[7] 3D head model by Infinite-Realities http://graphics.cs.williams.edu/data/meshes/head.zip