source.md

Themes

Set your presentation theme:
Black (default) - White - League - Sky - Beige - Simple
Serif - Blood - Night - Moon - Solarized

H:

VERTEX-SHADERS IN PROCESSING

Jean Pierre Charalambos
Universidad Nacional de Colombia
Presentation best seen online
See also the source code

H:

Contents

1. Introduction
2. Shader design patterns
3. Lighting intro
4. Diffuse light
5. Specular light

H:

Intro: What is a shader?

A shader is a program that runs on the GPU (Graphics Processing Unit) and it is controlled by our application (for example a Processing sketch)

The language of the shaders in Processing is GLSL (OpenGL Shading Language)

History

Andres Colubri started his involvement with Processing back in 2007 with a couple of libraries called GLGraphics and GSVideo

In 2013, Processing 2.0 was released and incorporated most of the funcionality of GLGraphics and GSVideo, including shaders, into the core of the language

V:

Intro: The graphics pipeline

Vertex shader

Fragment shader

V:

Intro: Shaders GPU execution

The vertex shader is run on each vertex sent from the sketch:

for vertex in geometry:
    vertex_clipspace = vertex_shader(vertex)

The fragment shader is run on each pixel covered by the geometry in our sketch:

for pixel in screen:
    if covered_by_geometry(pixel):
        ouptut_color = fragment_shader(pixel)

V:

Intro: Shader variable types

*Uniform* variables are those that remain constant for each vertex in the scene, for example the _projection_ and _modelview_ matrices

*Attribute* variables are defined per each vertex, for example the _position_, _normal_, and _color_

*Varying* variables allows to relate a vertex attribute to a fragment, using interpolation

N:

• varying variables get interpolated between the vertex and the fragment shader

V:

Intro: Processing shader API: PShader

Class that encapsulates a GLSL shader program, including a vertex and a fragment shader

V:

Intro: Processing shader API: loadShader()

Loads a shader into the PShader object

Method signatures

  loadShader(fragFilename)
  loadShader(fragFilename, vertFilename)

Example

  PShader unalShader;
  void setup() {
    ...
    //when no path is specified it looks in the sketch 'data' folder
    unalShader = loadShader("unal_frag.glsl", "unal_vert.glsl");
  }

V:

Intro: Processing shader API: shader()

Applies the specified shader

Method signature

  shader(shader)

Example

  PShader simpleShader, unalShader;
  void draw() {
    ...
    shader(simpleShader);
    simpleGeometry();
    shader(unalShader);
    unalGeometry();
  }

V:

Intro: Processing shader API: resetShader()

Restores the default shaders

Method signatures

  resetShader()

Example

  PShader simpleShader;
  void draw() {
    ...
    shader(simpleShader);
    simpleGeometry();
    resetshader();
    otherGeometry();
  }

V:

Intro: Processing shader API: PShader.set()

Sets the uniform variables inside the shader to modify the effect while the program is running

Method signatures for vector uniform variables vec2, vec3 or vec4:

  .set(name, x)
  .set(name, x, y)
  .set(name, x, y, z)
  .set(name, x, y, z, w)
  .set(name, vec)

• name: of the uniform variable to modify
• x, y, z and w: 1st, snd, 3rd and 4rd vec float components resp.
• vec: PVector

V:

Intro: Processing shader API: PShader.set()

Sets the uniform variables inside the shader to modify the effect while the program is running

Method signatures for array uniform variables bool[], float[], int[]:

  .set(name, x)
  .set(name, x, y)
  .set(name, x, y, z)
  .set(name, x, y, z, w)
  .set(name, vec)

• name: of the uniform variable to modify
• x, y, z and w: 1st, snd, 3rd and 4rd vec (boolean, float or int) components resp.
• vec: boolean[], float[], int[]

V:

Intro: Processing shader API: PShader.set()

Sets the uniform variables inside the shader to modify the effect while the program is running

Method signatures for mat3 and mat4 uniform variables:

  .set(name, mat) // mat is PMatrix2D, or PMatrix3D

• name of the uniform variable to modify
• mat PMatrix3D, or PMatrix2D

V:

Intro: Shaders

Processing shader API: PShader.set()

Sets the uniform variables inside the shader to modify the effect while the program is running

Method signatures for texture uniform variables:

  .set(name, tex) // tex is a PImage

V:

Intro: Processing shader API: PShader.set()

Sets the uniform variables inside the shader to modify the effect while the program is running

Example to set mat4 uniform variables:

  PShader unalShader;
  PMatrix3D projectionModelView1, projectionModelView2;
  void draw() {
    ...
    shader(unalShader);
    unalShader.set("unalMatrix", projectionModelView1);
    unalGeometry1();
    unalShader.set("unalMatrix", projectionModelView2);
    unalGeometry2();
  }

H:

Shader design patterns

1. Data sent from the sketch to the shaders
2. Passing data among shaders
3. Consistency of geometry operations

V:

Shader design patterns

Pattern 1: Data sent from the sketch to the shaders

Processing passes data to the shaders in a context sensitive way

Specific data (attribute and uniform vars) sent to the GPU depends on the specific Processing commands issued, e.g., ```fill(rgb) -> attribute vec4 color```

Several types of shader thus arise in Processing

More details are discussed in the _Shader Programming for Computational Arts and Design - A Comparison between Creative Coding Frameworks_ [paper](http://www.scitepress.org/DigitalLibrary/PublicationsDetail.aspx?ID=ysaclbloDHk=&t=1)

V:

Shader design patterns

Pattern 1: Data sent from the sketch to the shaders

(Frequently used) Attribute variables

Processing methods	Type	Attribute	Space
vertex()	vec4	vertex (or position)	local
normal(), shape()	vec3	normal	local
vertex()	vec2	texCoord	texture
stroke(), fill()	vec4	color	--

V:

Shader design patterns

Pattern 1: Data sent from the sketch to the shaders

(Frequently used) Uniform variables

Processing methods	Type	Uniform
ortho(), perspective()	mat4	projection
applyMatrix(), translate(), rotate(), scale()	mat4	modelview
applyMatrix(), translate(), rotate(), scale()	mat3	normalMatrix

V:

Shader design patterns

Pattern 1: Data sent from the sketch to the shaders

(Frequently used) Uniform variables

Processing methods	Type	Uniform	Space
texture()	mat4	texMatrix	--
texture()	sampler2D	texture	--
texture()	vec2	texOffset	texture
lights(), ambientLight(), spotLight(), directionalLight()	vec4	lightPosition	eye

V:

Shader design patterns

Pattern 1: Data sent from the sketch to the shaders

Check the code to consult all the attribute and uniform variables sent to the shaders

V:

Shader design patterns

Pattern 2: Passing data among shaders

Uniform variables are available for both, the vertex and the fragment shader. Attribute variables are only available to the vertex shader

Passing a vertex *attribute* variable to the fragment shader thus requires relating it first to a vertex shader *varying* variable

The vertex and fragment shaders would look like the following: ```glsl // vert.glsl attribute var; varying vert_var; void main() { ... vert_var = fx(var); } ``` ```glsl // frag.glsl varying vert_var; ```

V:

Shader design patterns

Pattern 2: Passing data among shaders

(Frequently used) Varying variables

Processing methods	Type	Attribute	Type	Varying
stroke(), fill()	vec4	color	vec4	vertColor
vertex()	vec2	texCoord	vec4	vertTexCoord

V:

Shader design patterns

Pattern 3: Consistency of geometry operations

Geometry operation operands should be defined in the same coordinate system

Tip 1: ```transform * vertex // projection * modelview * vertex``` yields the vertex coordinates in [clip-space](http://visualcomputing.github.io/Transformations/#/6/8) (see also [this](http://www.songho.ca/opengl/gl_transform.html))

Tip 2: ```modelview * vertex``` yields the vertex coordinates in eye-space

Tip 3: Since the eye position is 0 in eye-space, eye-space is the usual coordinate system for geometry operations

H:

Examples

Raster

[Raster example](https://github.com/VisualComputing/Shaders/tree/gh-pages/sketches/desktop/raster)

V:

Examples

Raster

Vertex shader code:

// Pattern 1: variables are sent by processing
uniform mat4 transform;
attribute vec4 position;
attribute vec4 color;
varying vec4 vertColor;

void main() {
  // Pattern 2: data among shaders
  vertColor = color;
  // Pattern 3: consistency of geometry operations
  // gl_Position should be defined in clipspace
  gl_Position = transform * position;
}

V:

Examples

Raster

Fragment shader code:

varying vec4 vertColor;
uniform bool cmy;

void main() {
  gl_FragColor = cmy ? vec4(1-vertColor.r, 1-vertColor.g, 1-vertColor.b, vertColor.a) : vertColor;
}

V:

Examples

Raster

raster.pde excerpt:

PShader shader;
boolean cmy;

void setup() {
  //shader = loadShader("frag.glsl", "vert.glsl");
  // same as:
  shader = loadShader("frag.glsl");
  // don't forget to ask why?
  shader(shader);
}

void draw() {
  background(0);
  scene.drawAxes();
  scene.render();
}

void keyPressed() {
  if (key == 'c') {
    cmy = !cmy;
    shader.set("cmy", cmy);
  }
}

V:

Examples

Bypassing Processing matrices

Passive transformation shaders output (source code available [here](https://github.com/VisualComputing/Shaders/tree/gh-pages/sketches/desktop/PassiveTransformations))

V:

Examples

Bypassing Processing matrices

Design patterns

Pattern 1: Data sent from the sketch to the shaders

(vert.glsl excerpt)

...
uniform mat4 nub_transform;
attribute vec4 vertex;

void main() {
  gl_Position = nub_transform * vertex;
  ...
}

V:

Examples

Bypassing Processing matrices

Design patterns

(PassiveTransformations.pde excerpt)

Graph graph;
Node[] nodes;
PShader shader;

void setup() {
  graph = new Graph(g, width, height);
  graph.setMatrixHandler(new MatrixHandler() {
    @Override
    protected void _setUniforms() {
      shader(shader);
      Scene.setUniform(shader, "nub_transform", transform());
    }
  });
  ...
  //discard Processing matrices
  resetMatrix();
  shader = loadShader("frag.glsl", "vert.glsl");
}

void draw() {
  background(0);
  // sets up the initial nub matrices according to user interaction
  graph.preDraw();
  graph.render();
}

H:

Light shaders

Simple lighting models

Simple lighting models of a 3D scene involves at least:

1. (optionally) Taking into account ambient light
2. Placing one or more light sources in the space
3. Defining their parameters, such as type (point, spotlight) and color (diffuse, specular)

Assumption: light source emits light equally in all directions

H:

Light shaders: diffuse light

Lighting parameters

Diffuse light: `$I = direction \bullet normal$`

V:

Light shaders: diffuse light

Per-Vertex vs Per-Fragment computations

Diffuse light: `$I = direction \bullet normal$`

V:

Light shaders: diffuse light

Per-Vertex computations

Per vertex diffuse light shader output (source code available [here](https://github.com/codeanticode/pshader-tutorials/blob/master/intro/Ex_06_1_light/))

V:

Light shaders: per-vertex diffuse light

Design patterns

Pattern 1: Data sent from the sketch to the shaders

//excerpt from lightvert.glsl
uniform mat4 modelview;
uniform mat3 normalMatrix;
uniform vec4 lightPosition;

attribute vec4 position;
attribute vec4 color;
attribute vec3 normal;

V:

Light shaders: per-vertex diffuse light

Design patterns

Observation about the normal matrix

Let $M$ be $ModelView(4;4)$ (i.e., it is formed by deleting row and column 4 from the ModelView)

Multiplying the input normal vector by the normalMatrix, i.e., $({M^{-1})}^T$, yields its coordinates in the eye-space

V:

Light shaders: per-vertex diffuse light

Design patterns

Observation about the normal matrix

Why not use modelview matrix, instead of normalMatrix ($({M^{-1})}^T$)?

`$N * modelview$` when the matrix contains a non-uniform scale

V:

Light shaders: per-vertex diffuse light

Design patterns

Pattern 2: Passing data among shaders

Pattern 3: Consistency of geometry operations

//excerpt from lightvert.glsl
uniform vec4 lightPosition;
varying vec4 vertColor;

void main() {
  ...
  vec3 ecPosition = vec3(modelview * position);//eye coordinate system
  vec3 ecNormal = normalize(normalMatrix * normal);//eye coordinate system
  vec3 direction = normalize(lightPosition.xyz - ecPosition);//Pattern 3   
  float intensity = max(0.0, dot(direction, ecNormal));//Pattern 3
  vertColor = vec4(intensity, intensity, intensity, 1) * color;
}

V:

Light shaders: per-vertex diffuse light

Design patterns

Pattern 2: Passing data among shaders

//lightfrag.glsl
varying vec4 vertColor;

void main() {
  gl_FragColor = vertColor;
}

V:

Light shaders: per-pixel diffuse light

Design patterns

Per pixel diffuse light shader output (source code available [here](https://github.com/codeanticode/pshader-tutorials/blob/master/intro/Ex_06_2_pixlight/))

V:

Light shaders: per-pixel diffuse light

Design patterns

Pattern 1: Data sent from the sketch to the shaders

//excerpt from pixlightvert.glsl
uniform mat4 modelview;
uniform mat3 normalMatrix;
uniform vec4 lightPosition;

attribute vec4 position;
attribute vec4 color;
attribute vec3 normal;

V:

Light shaders: per-pixel diffuse light

Design patterns

Pattern 2: Passing data among shaders

Pattern 3: Consistency of geometry operations

//excerpt from pixlightvert.glsl
varying vec4 vertColor;
varying vec3 ecNormal;
varying vec3 lightDir;

void main() {
  ...
  vec3 ecPosition = vec3(modelview * position);
  ecNormal = normalize(normalMatrix * normal);
  lightDir = normalize(lightPosition.xyz - ecPosition);//Pattern 3
  vertColor = color;
}

V:

Light shaders: per-pixel diffuse light

Design patterns

Pattern 2: Passing data among shaders

Pattern 3: Consistency of geometry operations

//pixlightfrag.glsl
varying vec4 vertColor;
varying vec3 ecNormal;
varying vec3 lightDir;

void main() {  
  vec3 direction = normalize(lightDir);
  vec3 normal = normalize(ecNormal);
  float intensity = max(0.0, dot(direction, normal));//Pattern 3
  gl_FragColor = vec4(intensity, intensity, intensity, 1) * vertColor;
}

H:

Light shaders: specular light (Phong model)

Lighting parameters

Specular light: `$I = direction_{reflected} \bullet observer$`

V:

Light shaders: specular light

Per-Vertex vs Per-Fragment computations

Specular light: `$I = direction_{reflected} \bullet observer$`

V:

Light shaders: specular light

Per-Vertex computations

Per vertex specular light shader output (source code available [here](https://github.com/VisualComputing/Shaders/tree/gh-pages/sketches/desktop/Specular))

V:

Light shaders: per-vertex specular light

Design patterns

Identifying the per vertex specular shader design patterns is left as an excercise to the reader

V:

Light shaders: per-pixel specular light

Design patterns

Per pixel specular light shader output (source code available [here](https://github.com/VisualComputing/Shaders/tree/gh-pages/sketches/desktop/PixSpecular))

V:

Light shaders: per-pixel specular light

Design patterns

Identifying the per pixel specular shader design patterns is left as an excercise to the reader

V:

Light shaders

Suggested workshop

Simple lighting and material

Tasks

1. Add ambient light
2. Add light attenuation
3. Add fog
4. Combine all the simple lighting models (ambient, diffuse and specular) using per-vertex and per-pixel shaders
5. Use up to 8 lights in the model, relating each light source with a nub node
6. Implement advanced lighting models such as Warn lights, normal mapping and shadow mapping.

H:

References

• OpenGL Shading Language
• Data Type (GLSL)
• The Book of Shaders, by Patricio Gonzalez Vivo
• Processing shaders tutorial
• Tutorial source code
• Shader Programming for Computational Arts and Design - A Comparison between Creative Coding Frameworks
• ShaderBase: A Processing Tool for Shaders in Computational Arts and Design (source code available here)