[Unity] Tessellation

Tessellation is the process of rendering pipelines that can divide the polygon on the GPU side.

The hull and domain shader stages are part of the tessellation pipeline of the GPU. They’re generally used to compute highly-detailed surface geometry based on lower-detail input surface geometry, which is defined as triangles or quads (et cetera). The lower-detail input primitives are called “patches,” and it’s important to note that they may not represent actual geometry that will eventually exist (although they could). Think of more like the control points of a bezier curve, except for a surface. The tessellation phase occurs after the vertex shader stage in the pipeline.

Tessellation is composed of three stages:

• Hull-Shader Stage (programmable)

The hull shader takes an input patch and produces an output patch (or patches; this is where subdivision of the patch would generally occur). Constant metadata about the patch can also be computed within the hull shader and output for processed by later stages of the pipeline.

• Tessellation Stage

The output of the hull shader runs through a (fixed function) tessellation stage which produces tiled, normalized domains of the appropriate type (e.g., quads or triangles).

• Domain-Shader Stage (programmable)

The domain shader is executed against these domains in order to compute the actual vertex position of any given point in a domain that resulted from the aforementioned tessellation. The domain shader thus outputs a vertex position.

In SurfaceShader

Declare the usage of displacement and tessellation function.

#pragma surface surf BlinnPhong addshadow fullforwardshadows vertex:disp tessellate:tessEdge nolightmap

Given three vertexes (one patch), we can compute the tessellation edge used unity built-in functions.

The output x, y, z is the divided count of each edge, and w is the divided patches count.

float4 tessEdge(appdata v0, appdata v1, appdata v2) {
  return UnityEdgeLengthBasedTessCull(v0.vertex, v1.vertex,  
  v2.vertex, _EdgeLength, _Displacement);
}

Tesselation Functions

There are several functions you can use:

• tessFixed() : The fixed amount of tessellation by giving specific _Tess
  float4 tessFixed() { return _Tess; }
• UnityDistanceBasedTess: tessellation level based on distance from the camera
• UnityEdgeLengthBasedTess: tessellation levels based on triangle edge length on the screen
• UnityEdgeLengthBasedTessCull: same as UnityEdgeLengthBasedTess but performs patch frustum culling, giving better performance. This makes the shader a bit more expensive but saves a lot of GPU work for parts of meshes that are outside of the Camera’s view.

Phong Tessellation

Phong Tessellation modifies positions of the subdivided faces so that the resulting surface follows the mesh normals a bit. It’s quite an effective way of making low-poly meshes become more smooth.

#pragma surface surf Lambert vertex:dispNone tessellate:tessEdge tessphong:_Phong nolightmap

Displacement Function

This function is for displacement mapping and is called after tessellation (in domain shader stage).

void disp(inout appdata v) {
float2 uv = v.texcoord.xy * _DispTex_ST.xy + _DispTex_ST.zw;
float d = tex2Dlod(_DispTex, float4(uv, 0, 0)).r * _Displacement;
#ifdef _NEGATIVE_ON
  v.vertex.xyz += v.normal * d;
#else
  v.vertex.xyz -= v.normal * d;
#endif
}

In Vertex and Fragment Shader

Tessellation gets more complicated in vertex and fragment. You have to write the hull and domain shader by yourself.

#pragma vertex vert
#pragma fragment frag
#pragma hull hull_shader
#pragma domain domain_shader

Tessellation Data Structure

Define the tesselation input and output structure:

struct InternalTessInterp_appdata {
  float4 vertex : INTERNALTESSPOS;
  float4 tangent : TANGENT;
  float3 normal : NORMAL;
  float2 texcoord : TEXCOORD0;
};
struct TessellationFactors{
  float edge[3] : SV_TessFactor;
  float inside : SV_InsideTessFactor;
};

Vertex Shader

In the vertex shader, just convert the appdata to tessellation input data.

InternalTessInterp_appdata vert (appdata v) {
  InternalTessInterp_appdata o;
  o.vertex = v.vertex;
  o.tangent = v.tangent;
  o.normal = v.normal;
  o.texcoord = v.texcoord;
  return o;
}

Tessellation Hull Shader

This function uses the tessellation vertexes to compute the divided edge and patch count.

TessellationFactors hull_const(InputPatch<InternalTessInterp_appdata, 3> v) {
  TessellationFactors o;
  float4 tf;
  tf = UnityEdgeLengthBasedTessCull(v[0].vertex, v[1].vertex,   
  v[2].vertex, _EdgeLength, _Displacement * 1.5f);
  o.edge[0] = tf.x;
  o.edge[1] = tf.y;
  o.edge[2] = tf.z;
  o.inside = tf.w;
  return o;
}

In the real hull shader, we need to define the setting of the hull stage and just return the output control point.

[UNITY_domain("tri")]                  //triangle
[UNITY_partitioning("fractional_odd")] //integer,fractional_even,odd
[UNITY_outputtopology("triangle_cw")]  //cw: clock wise, ccw:counter
[UNITY_patchconstantfunc("hull_const")]//patch function
[UNITY_outputcontrolpoints(3)]         //output control point count
InternalTessInterp_appdata hull_shader (InputPatch<InternalTessInterp_appdata,3> v, uint id : SV_OutputControlPointID) {
  return v[id];
}

Tessellation Domain Shader

Use the hull shader output to do the positioning of vertex and pass the data to fragment shader.

[UNITY_domain("tri")]
v2f domain_shader(TessellationFactors tessFactors, const OutputPatch<InternalTessInterp_appdata, 3> vi, float3 bary : SV_DomainLocation) {
  appdata v;
  UNITY_INITIALIZE_OUTPUT(appdata, v);
  v.vertex = vi[0].vertex * bary.x + vi[1].vertex * bary.y +  
  vi[2].vertex * bary.z;
  v.tangent = vi[0].tangent * bary.x + vi[1].tangent * bary.y +
  vi[2].tangent * bary.z;
  
  v.normal = vi[0].normal * bary.x + vi[1].normal * bary.y +  
  vi[2].normal * bary.z;
  v.texcoord = vi[0].texcoord * bary.x + vi[1].texcoord * bary.y +
  vi[2].texcoord * bary.z;
disp(v);
  v2f o = vert_to_frag_process(v);
  return o;
}

Before fragment shader, we compute the necessary data such as world tangent and normal

v2f vert_to_frag_process(appdata v) {
  v2f o;
  UNITY_INITIALIZE_OUTPUT(v2f, o);
  o.pos = UnityObjectToClipPos(v.vertex);
  o.uv_MainTex.xy = TRANSFORM_TEX(v.texcoord, _MainTex);
  float3 worldPos = mul(unity_ObjectToWorld, v.vertex);
  float3 worldNormal = UnityObjectToWorldNormal(v.normal);
  fixed3 worldTangent = UnityObjectToWorldDir(v.tangent.xyz);
  fixed tangentSign = v.tangent.w * unity_WorldTransformParams.w;
  fixed3 worldBinormal = cross(worldNormal, worldTangent) *    
  tangentSign;
  o.tSpace0 = float4(worldTangent.x, worldBinormal.x, worldNormal.x,   
  worldPos.x);
  o.tSpace1 = float4(worldTangent.y, worldBinormal.y, worldNormal.y,  
  worldPos.y);
  o.tSpace2 = float4(worldTangent.z, worldBinormal.z, worldNormal.z, 
  worldPos.z);
  return o;
}

Fragment Shader

In the fragment shader, do the lighting and coloring.

fixed4 frag(v2f i) : SV_Target{
  float3 worldPos = float3(i.tSpace0.w, i.tSpace1.w, i.tSpace2.w);
  #ifndef USING_DIRECTIONAL_LIGHT
    fixed3 lightDir = normalize(UnityWorldSpaceLightDir(worldPos));
  #else
    fixed3 lightDir = _WorldSpaceLightPos0.xyz;
  #endif
  float3 worldViewDir = normalize(UnityWorldSpaceViewDir(worldPos));
  fixed3 albedo = 0;
  half emission = 0;
  half specular = 0;
  fixed alpha = 0;
  fixed gloss = 0;
  fixed3 normal = fixed3(0, 0, 1);
  fixed4 col = tex2D(_MainTex, i.uv_MainTex) * _Color;
  albedo = col.rgb;
  specular = _Specular;
  gloss = _Glossiness;
  normal = UnpackNormal(tex2D(_NormalMap, i.uv_MainTex));
  float3 worldN;
  worldN.x = dot(i.tSpace0.xyz, normal);
  worldN.y = dot(i.tSpace1.xyz, normal);
  worldN.z = dot(i.tSpace2.xyz, normal);
  worldN = normalize(worldN);
  normal = worldN;
  half3 h = normalize(lightDir + worldViewDir);
  fixed diff = max(0, dot(normal, lightDir));
  float nh = max(0, dot(normal, h));
  float spec = pow(nh, specular * 128) * _Glossiness;
  fixed4 c;
  c.rgb = albedo * diff + _SpecColor.rgb * spec;
  c.a = col.a;
  return c;
}