# User Interaction in WebGL

#### Duncan Murdoch

#### September 19, 2015

## Introduction

This document describes how to use embedded Javascript to control a WebGL display in an HTML document. For more general information, see rgl Overview.

We start with two simple examples. The next section gives reference information.

Consider the simple plot of the iris data. We insert a code chunk with label `plot3d` (which will be used below).

    with(iris, plot3d(Sepal.Length, Sepal.Width, Petal.Length,
                      type="s", col=as.numeric(Species)))
    subid <- currentSubscene3d()

You must enable Javascript to view this page properly.

We might like a button on the web page to cause a change to the display, e.g. a rotation of the plot. First we add buttons, with the “onclick” event set to a function described below:

    <button type="button" onclick="rotate(10)">Forward</button>
    <button type="button" onclick="rotate(-10)">Backward</button>

which produces these buttons:

We stored the subscene number that is currently active in `subid` in the code chunk above, and use it as ```r subid``` in the script below.

The `rotate()` function makes use of the global `<prefix>rgl` object. The `knitr` WebGL support sets the prefix to the code chunk label, so the global is called `plot3drgl`:

    <script type="text/javascript">
    var rotate = function(angle) {
      plot3drgl.userMatrix[`r subid`].rotate(angle, 0,1,0);
      plot3drgl.drawScene();
    };
    </script>

## Autogenerated controls

We can also change the contents of the plot using `toggleButton`. For example, we can redo the previous plot, but with the three species as separate “spheres” objects and buttons to toggle them:

    sphereid <- with(subset(iris, Species == "setosa"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                      col=as.numeric(Species),
                      radius = 0.211))
    with(subset(iris, Species == "versicolor"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                   col=as.numeric(Species),
                   radius = 0.211))
    with(subset(iris, Species == "virginica"),
         spheres3d(Sepal.Length, Sepal.Width, Petal.Length,
                   col=as.numeric(Species),
                   radius = 0.211))
    aspect3d(1,1,1)
    decorate3d()
    subid <- currentSubscene3d()

You must enable Javascript to view this page properly.

    toggleButton(sphereid, label = "setosa", prefix = "toggle", subscene = subid)

    toggleButton(sphereid+1, label = "versicolor", prefix = "toggle", subscene = subid)

    toggleButton(sphereid+2, label = "virginica", prefix = "toggle", subscene = subid)

Note that we need to use `results="asis"` for the button code.
Normally we would also use `echo=FALSE`, though I didn’t do so above; then the buttons will end up side-by-side:

An alternate control to achieve the same thing is `subsetSlider`.

You must enable Javascript to view this page properly.

    subsetSlider(subsets = list(setosa = sphereid,
                      versicolor = sphereid + 1,
                      virginica = sphereid + 2,
                      all = sphereid + 0:2),
                 prefixes = "slider", subscenes = subid,
                 init = 3)

all

There are several other functions to generate the Javascript code for controls. `par3dinterpSetter` generates a function that approximates the result of `par3dinterp`. `propertySetter` is a more general function to set the value of properties of the scene. Both generate Javascript functions, but not the controls to use them; for that, use `propertySlider` or your own custom code.

For example, the following code (similar to the `play3d` example) rotates the scene in a complex way.

You must enable Javascript to view this page properly.

    M <- r3dDefaults$userMatrix
    fn <- par3dinterp(time = (0:2)*0.75, userMatrix = list(M,
                                         rotate3d(M, pi/2, 1, 0, 0),
                                         rotate3d(M, pi/2, 0, 1, 0) ) )
    propertySlider(setter = par3dinterpSetter(fn, 0, 1.5, steps=15,
                           prefix = "userMatrix",
                           subscene = subid),
               step = 0.01)

0

Some things to note: The generated Javascript slider has 150 increments, so that motion appears smooth. However, storing 150 `userMatrix` values would take up a lot of space, so we use interpolation in the Javascript code. However, the Javascript code can only do linear interpolation, not the more complex spline-based SO(3) interpolation done by `par3dinterp`. Because of this, we need to output 15 steps from `par3dinterpSetter` so that the distortions of linear interpolation are not visible.

Another function that auto-generates Javascript code is `clipplaneSlider`. This function allows the user to control the location of a clipping plane by moving a slider. Both it and `par3dinterpSetter` are implemented using the more general `propertySlider`, which allows control of multiple objects in multiple scenes, but which does require knowledge of the internal representation of the scene in its Javascript implementation.

Less general than `propertySetter` is `vertexSetter`. This function sets attributes of individual vertices in a scene. For example, to set the x-coordinate of the closest point in the setosa group, and modify its colour from black to white,

You must enable Javascript to view this page properly.

    setosa <- subset(iris, Species == "setosa")
    which <- which.min(setosa$Sepal.Width)
    init <- setosa$Sepal.Length[which]
    propertySlider(
        vertexSetter(values = matrix(c(init,8,0,1,0,1,0,1), nrow=2),
                     attributes=c("x", "r", "g", "b"),
                 vertices = which, objid = sphereid,
                 prefix = "vertex"),
        step=0.01)

1

A related function is `ageSetter`, though it uses a very different specification of the attributes. It is used when the slider controls the “age” of the scene, and attributes of vertices change with their age. For example, to show a point moving along a curve, we could do the following.
Note that we need to specify multiple colours in the plot so that the colour is not fixed, and can be controlled by the slider. We also give two `ageSetter` calls in a list; the `propertySlider` will control both of them simultaneously.

    time <- 0:500
    xyz <- cbind(cos(time/20), sin(time/10), time)
    lineid <- plot3d(xyz, type="l", col = c("black", "black"))["data"]
    sphereid <- spheres3d(xyz[1, , drop=FALSE], radius = 8, col = "red")

You must enable Javascript to view this page properly.

    propertySlider( list(ageSetter(births = time, ages = c(0, 0, 50),
                  colors = c("gray", "red", "gray"),
                  objids = lineid, prefixes = "curve"),
                 ageSetter(births = 0, ages = time,
                      vertices = xyz, objids = sphereid,
                      prefixes = "curve")),
                 maxS = max(time) + 50)

0

The final function of this type is `matrixSetter`, for setting up multiple controls to modify a matrix, typically `userMatrix`. This is used when complex manipulation of a matrix requires several controls.

## User Defined Mouse Controls

`rgl` allows user defined mouse controls. For these to work within WebGL, you will need to write a Javascript version as well as the R version. This isn’t easy: R provides a lot of support functions which are not easily available in Javascript.

TODO: write a simple mouse control.

## Reference for `rglClass`

In writing the `writeWebGL()` function, I haven’t tried to prevent access to anything. On the other hand, I haven’t provided access to everything. The parts documented here should remain relatively stable (unless indicated otherwise). Users may also consult the source to `writeWebGL`, but should be aware that anything that isn’t documented here is subject to change without notice.

`rglClass`

As documented in `writeWebGL`, the call

    writeWebGL(..., prefix = "<prefix>")

will create a global object on the output page with name `<prefix>rgl` and Javascript class `rglClass`. This class has a large number of properties and methods, some of which are designed to be available for use by other code on the web page.

Most of the properties are stored as Javascript `Array` objects, indexed by the `rgl` id of the subscene to which they apply. There are also Javascript methods attached to the `rglClass` class.

### Methods

`drawScene()`

After any change that will affect the display, code should call `<prefix>rgl.drawScene()` to redraw the scene.

`inSubscene()`, `addToSubscene()`, `delFromSubscene()`

These methods each take two arguments: `id` and `subscene`, which should be the `rgl` ids of an object and a subscene. `inSubscene` tests whether `id` is already included in the subscene, and the others add it or delete it from the subscene.

`getSubsceneEntries()`

This function takes a subscene id as argument, and returns an `Array` containing all of the ids displayed in that subscene.

`setSubsceneEntries()`

This takes an `Array` of ids and a subscene id as arguments, and sets the contents of the subscene to the ids.

### Properties

`FOV`, `listeners`, `userMatrix`, `zoom`

These correspond to the `par3d` properties with the same names.

- `FOV` and `zoom` are arrays of numbers.
- `userMatrix` is an array of `CanvasMatrix4` objects (documented in the file `system.file("WebGL/CanvasMatrix.src.js")`.
- Each `listeners` item is itself an array of subscene ids that “listen” to mouse actions, i.e. `listeners[19]` would contain all subscene ids that respond to mouse actions in subscene 19.

`viewport`

This property also corresponds to the `par3d` property, but should be considered to be read-only.

`drawFns`, `clipFns`

These two arrays contain the code to display each object in the scene. The functions in the `drawFns` array are called for each object each time it is displayed. The `clipFns` functions are called when objects being clipped are drawn.

`values`, `offsets`

Most of the data about each object in a scene is contained in the `values` property. This is an array, indexed by object id. The individual entries are numeric arrays. Though they are singly-indexed, the entries are meant to be interpreted as matrices stored by row. The first 3 columns are generally the coordinates of a vertex, and remaining columns correspond to other values from `rgl.attrib`.

The `offsets` property gives the (0-based) offset of the first column for a particular attribute in a named object. Not all columns will be present in every object; if not present, the corresponding `offsets` entry will be `-1`. The entries are

Name | Meaning
--- | ---
vofs | Offset to 3 columns of vertex data in XYZ order
cofs | Offset to 4 columns of colour data in RGBA order
nofs | Offset to 3 columns of normal data
radofs | Offset to 1 column of sphere radius data
oofs | Offset to 2 columns of text or sprite origin data
tofs | Offset to 2 columns of texture coordinates
stride | Total number of columns in `values`

For example, to find the blue colour entry for vertex `i` in an object, one would first check if `offsets["cofs"]` was `-1`, indicating that no colour information was present. If not, the entry could be found using

    values[offsets["stride"]*(i-1) + offsets["cofs"] + 2]

This assumes `i` is specified using 1-based vertex counting as in R, and writes `values` and `offsets` instead of the fully specified `<prefix>rgl.values` and `<prefix>rgl.offsets` for clarity.

Changes to `values` need to be pushed to the graphics system to be reflected in the scene; see the calls to `gl.bindBuffer` and `gl.bufferData` in the source to `propertySlider` for details.

## Index

The following functions and `rglClass` properties and methods are described in this document:

FOV | drawFns | offsets | setSubsceneEntries | viewport
addToSubscene | drawScene | par3dinterpSetter | subsetSlider | zoom
ageSetter | getSubsceneEntries | propertySetter | toggleButton |
clipFns | inSubscene | propertySlider | userMatrix |
clipplaneSlider | listeners | propertySlider | values |
delFromSubscene | matrixSetter | rglClass | vertexSetter |