Screen Effects

Contents:

1. Intro

Screen effects allow you to create nice effects by processing the rendered image. Demos:

• Try the View -> Screen Effects menu in view3dscene, after loading any 3D scene. Note that you can activate many effects at the same time.

• Try the X3D files screen_effects.x3dv, screen_effect_blood_in_the_eyes.x3dv, screen_effect_film_grain.x3dv, screen_effect_movie_texture.x3dv. You should download our complete VRML/X3D demo models and open files in screen_effects subdirectory there, to see the complete working demos with an example castle model underneath.

2. Definition

You can define your own screen effects by using the ScreenEffect node in your VRML/X3D files. Inside the ScreenEffect node you provide your own shader code to process the screen, given the current color and depth buffer contents. With the power of GLSL shading language, your possibilities are endless :). You can warp the view, apply textures in screen-space, do edge detection, color operations and so on.

ScreenEffect : X3DChildNode {
  SFNode     [in,out]      metadata    NULL      # [X3DMetadataObject]
  SFBool     [in,out]      enabled     TRUE    
  SFBool     [in,out]      needsDepth  FALSE   
  MFNode     [in,out]      shaders     []        # [X3DShaderNode]
}

A ScreenEffect is active if it's a part of normal VRML/X3D transformation hierarchy (in normal words: it's not inside a disabled child of the Switch node or such) and when the "enabled" field is TRUE. In the simple cases, you usually just add ScreenEffect node anywhere at the top level of your VRML/X3D file. If you use many ScreenEffect nodes, then their order matters: they process the rendered screen in the given order.

You have to specify a shader to process the rendered screen by the "shaders" field. This works exactly like the standard X3D "Appearance.shaders", by selecting a first supported shader. Right now our engine supports only GLSL (OpenGL shading language) shaders inside ComposedShader nodes. To learn more about GLSL and X3D, see

• The overview of shaders support in our engine.
• The GLSL function reference. Be careful: the reference linked here describes both GLSL and OpenGL API. You can ignore the functions named glXxx, they are part of the OpenGL API, and they not useful to a shader author.
• GLSL description at Wikipedia and GLSL description at Khronos wiki.

The shader inside ScreenEffect is always linked with a library of useful GLSL functions:.

• ivec2 screen_position() - position of the current pixel. This pixel's color will be set by our gl_FragColor value at exit.
• int screen_x(), int screen_y() - x and y coordinates of current pixel position, for comfort.
• vec4 screen_get_color(ivec2 position) - get previous screen color at this pixel.
• float screen_get_depth(ivec2 position) - get previous depth at this pixel (only if needsDepth was TRUE).
• uniform values int screen_width, int screen_height - screen size in pixels.
  Note: do not define these uniform variables in your shader code. Or you will get "repeated declaration" GLSL errors. That is because on OpenGLES (Android, iOS), we have to actually "glue" the common screen effect code at the beginning of your code, and it already defines these uniform variables. (And we do it the same for desktop OpenGL, for consistency.)

3. Examples

A simplest example:

ScreenEffect {
  shaders ComposedShader {
    language "GLSL"
    parts ShaderPart { type "FRAGMENT" url "data:text/plain,
ivec2 screen_position();
vec4 screen_get_color(ivec2 position);
void main (void)
{
  gl_FragColor = screen_get_color(screen_position());
}
" } } }

The above example processes the screen without making any changes. You now have the full power of GLSL to modify it to make any changes to colors, sampled positions and such. For example make colors two times smaller (darker) by just dividing by 2.0:

ivec2 screen_position();
vec4 screen_get_color(ivec2 position);
void main (void)
{
  gl_FragColor = screen_get_color(screen_position()) / 2.0;
}

Or turn the screen upside-down by changing the 2nd texture coordinate:

ivec2 screen_position();
vec4 screen_get_color(ivec2 position);
int screen_x();
int screen_y();
void main (void)
{
  gl_FragColor = screen_get_color(
    ivec2(screen_x(), screen_height - screen_y()));
}

4. Details

Details about special functions available in the ScreenEffect shader:

• Internally, we pass the screen contents (color and, optionally, depth buffer) as a texture (normal npot texture, before OpenGLES we also used texture rectangle) or a multi-sample texture. You should just use the comfortable functions screen_get_xxx to read previous screen contents, they will hide the differences for you, and your screen effects will work for all multi-sampling (anti-aliasing) configurations.

• The texture coordinates for screen_get_xxx are integers, in range [0..width - 1, 0..height - 1]. This is usually comfortable when writing screen effects shaders, for example you know that (screen_x() - 1) is "one pixel to the left".

  You can of course sample the previous screen however you like. The screen_position() (or, equivalent, ivec2(screen_x(), screen_y())) is the position of current pixel, you can use it e.g. to query previous color at this point, or query some other colors around this point (e.g. to blur the image). Note that using gl_FragCoord.st as a pixel position will work in simple cases too, but it's not advised, because it will not work intuitively when you use custom viewports with our engine. screen_position() will cooperate nicely with custom viewports.

• We also pass uniform "screen_width", "screen_height" integers to the shader. These give you the size of the screen. (On new GPUs, you could also get them with GLSL function textureSize, but it's not available on older GPUs/OpenGL versions.)

• If you set "needsDepth" to TRUE then we also pass depth buffer contents to the shader. You can query it using screen_get_depth function..

  You can query depth information at any pixel for various effects. Remember that you are not limited to querying the depth of the current pixel, you can also query the pixels around (for example, for Screen Space Ambient Occlusion). The "Flashlight" effect in view3dscene queries a couple of pixels in the middle of the screen to estimate the distance to an object in front of the camera, which in turn determines the flashlight circle size.

• Remember that you can pass other uniform values to the shader, just like with any other ComposedShader nodes. For example you can pass an additional helper texture (e.g. a headlight mask) to the shader. Or you can route the current time (from TimeSensor) to the shader, to make your effect based on time.

5. Todos

ScreenEffect under a dynamic Switch doesn't react properly — changing "Switch.whichChoice" doesn't deactivate the old effect, and doesn't activate the new effect. For now, do not place ScreenEffect under Switch that can change during the world life. If you want to (de)activate the shader dynamically (based on some events in your world), you can send events to the exposed "enabled" field.