The scene_rdl2 library

RDL2 is the data format that MoonRay uses internally to store the scene it is rendering. RDL stands for “Render Data Layer”, and you may not be surprised to learn that RDL2 is the second such data layer developed at DWA.

The scene_rdl2 library contains the implementation of RDL2 : it is in the scene_rdl2 git repo under lib/scene/rdl2. The scene_rdl2 API is used to write translators to RDL2 format, Arras/MoonRay clients, and implementations of scene object classes. This document covers the main concepts, classes and functions in the API : more detail can be found in the header files themselves.

The class used to represent a scene is scene_rdl2::rdl2::SceneContext. For the remainder of this document, I will omit the namespace prefix “scene_rdl2::rdl2::”

MoonRay itself stores a SceneContext as part of the RenderContext class. You can also use scene_rdl2 on its own to create, examine and modify RDLA and RDLB files. Since RDLA is a text format, it can also be edited directly, but scene_rdl2 is the only way to work with RDLB.

SceneContext

A SceneContext contains a set of SceneObjects, representing cameras, geometry materials, lights (and so on…) within the scene. It’s easier to try out examples if you can load and save SceneContexts. This is the code to load from an RDLA or RDLB file:

#include <scene_rdl2/scene/rdl2/rdl2.h>
SceneContext context;
context.readSceneFromFile("myScene.rdla",context);

and to save:

context.writeSceneToFile(context,"anotherScene.rdlb");

These functions automatically determine the file type (RDLA or RDLB) from the extension. They will only work if the environment variable RDL2_DSO_PATH is set correctly (more on this later…)

Each SceneObject has a unique name, which you can use to access it:

SceneObject* shot_camera = context.getObject("shot_camera");

If the requested object doesn’t exist, getObject will throw except::KeyError. You can check for existence with bool sceneObjectExists(const std::string& name). Iterate through all SceneObjects like this:

for (auto iter = context.beginSceneObject(); 
           iter != context.endSceneObject(); ++iter) {
           SceneObject* sceneObj =  iter->second;
           ...
}

To add a new SceneObject to the SceneContext, use:

SceneObject* createSceneObject(const std::string& className, 
                                const std::string& objectName);

Classes are discussed in the next section. If an object with the class and name you specify already exists, this function will simply return it. If an object with the same name but different class exists, scene_rdl2 will throw an exception.

Each SceneObject has a set of named attributes. You can access attribute values directly by name:

template <typename T> const T& get(const std::string& name) const;
template <typename T> void set(const std::string& name, const T& value);
void set(const std::string& name, SceneObject* value);
void resetToDefault(const std::string& name);
void resetAllToDefault();

There are more efficient ways to get and set attributes, discussed below.

Code that sets attributes on a scene object must be surrounded by calls to SceneObject::beginUpdate() and SceneObject::endUpdate(). If you try to set attributes without doing this, scene_rdl2 will throw an exception.

SceneClass

Every SceneObject belongs to a SceneClass that defines the object’s attributes and behavior during rendering. Most of the SceneClasses are defined in shared libraries that scene_rdl2 loads on demand. The path that scene_rdl2 uses to look for these libraries defaults to the colon-separated string in the environment variable RDL2_DSO_PATH, although you can set it to something else by calling SceneContext::setDsoPath(). Two versions of each library are generated by the MoonRay build – the “proxy” versions of the libraries contain the attribute information needed by scene_rdl2, but not the code needed to render the objects – meaning that they do not need to link with the rest of MoonRay. To tell scene_rdl2 to use the proxy libraries do this:

context.setProxyModeEnabled(true);

The LUA syntax of RDLA files doesn’t distinguish between an undeclared variable and a SceneClass that cannot be found on the rdl2 dso path. Using a class that cannot be found produces this error message:

ERROR: RDLA Error: /tmp/banana.rdla:1: global variable 'BananaMaterial' was never declared (assign a value to it before using it), or if 'BananaMaterial' is a constructor for a DSO type, the DSO could not be found

SceneContext provides the following functions for SceneClasses:

const SceneClass* getSceneClass(const std::string& name) const;
bool sceneClassExists(const std::string& name) const;
SceneClassConstIterator beginSceneClass() const;
SceneClassConstIterator endSceneClass() const;

These only look for SceneClasses that have been loaded (for example, because they are used by a scene file that has been loaded). You can ask a scene context to load every SceneClass it can find on the rdl2 dso path by calling SceneContext::loadAllSceneClasses(). To find and load a single scene class by name, call SceneContext::createSceneClass(name).

SceneClass objects define the attribute names, types and layout for a class of scene objects. Individual attributes are represented by the Attribute class.

const SceneClass& SceneObject::getSceneClass() const;

Attribute* SceneClass::getAttribute(const std::string& name);
AttributeConstIterator SceneClass::beginAttributes() const;
AttributeConstIterator SceneClass::endAttributes() const;

const std::string& Attribute::getName() const;
AttributeType Attribute::getType() const;
template <typename T> const T& Attribute::getDefaultValue() const;

These are the possible attribute types:

Each of these has a vector version : TYPE_BOOL_VECTOR, etc… These are all std::vectors of the underlying type, except TYPE_BOOL_VECTOR, which is std::deque<bool>.

TYPE_SCENE_OBJECT_INDEXABLE is an alternative to TYPE_SCENE_OBJECT_VECTOR with fast hash lookup of index from value.

Each Attribute also has a set of flags:

bool isBindable() const;
bool isBlurrable() const;
bool isEnumerable() const;
bool isFilename() const;

Binding and blurring are discussed below. The filename flag is mainly for the benefit of client code – for example, to enable a file browser when editing the attribute.

Enumerable attributes should be TYPE_INT, and they have a list of valid Int values, each with an associated descriptive string. scene_rdl2 has no direct support for setting enumerable attributes directly from the string description.

bool isValidEnumValue(Int enumValue) const;
EnumValueConstIterator beginEnumValues() const;
EnumValueConstIterator endEnumValues() const;

Attribute objects can also hold arbitrary string metadata. The most common use for this is to associate a documentation string with the attribute, under the key “comment”.

const std::string& getMetadata(const std::string& key) const;

SceneObject subclasses

Each dso that defines a scene class contains a C++ subclass of SceneObject. These SceneObject subclasses form a hierarchy that provides attribute inheritance. For example, PerspectiveCamera.so contains the C++ class PerspectiveCamera:

PerspectiveCamera’s parent class is Camera (in the scene_rdl2 library) Camera’s parent class is Node (representing scene objects with a 3D transform) Node’s parent class is SceneObject.

Each SceneObject subclass has static members of type AttributeKey<T> corresponding to the attributes it adds – where T is the C++ type of the attribute. For example, PerspectiveCamera defines the static member

AttributeKey<Float> attrFocalKey;

for the attribute named “focal”. AttributeKey directly stores the data offset of the attribute within a scene object of this class, and therefore can be used to get and set attributes much more efficiently than lookup by name. Code like this:

float focal = shot_camera.get(PerspectiveCamera::attrFocalKey);

uses the AttributeKey overload of the get function, and is much faster than:

float focal = shot_camera.get<float>("focal");

The first version has the added advantage that it will automatically use the correct C++ type for the attribute, and fail at compile time if conversion is not possible.

In practice, outside the implementation of a specific scene class, you are more likely to want to access attributes common to an entire sub-type of classes. For example, many scene classes inherit from Node, and they all have a “node_xform” attribute that you can access like this:

Mat4d xform = any_node_object.get(Node::sNodeXformKey)

SceneObject subclasses often also provide utility functions. For example, Camera provides setNear (which simply sets the Camera::sNear attribute) and computeProjectionMatrix (which is overridden by Camera subclasses like PerspectiveCamera).

For built-in types like Node and Camera, SceneObject provides a fast dynamic cast that avoids use of RTTI:

if (an_object->isA<Camera>()) {
     Camera* a_camera = an_object->asA<Camera>();
    a_camera->setNear(0.001);
}

isA() and asA() do not work to cast to types defined in external DSOs.

This is the full hierarchy defined in scene_rdl2:

SceneObject
    DisplayFilter
    GeometrySet
        ShadowReceiverSet
    LightFilter
    LightFilterSet
    LightSet
        ShadowSet
    Metadata
    Node
        Camera
        EnvMap
        Geometry
        Joint
        Light
    RenderOutput
    SceneVariables
    Shader
        Map
        NormalMap
        RootShader
            Displacement
            Material
    TraceSet
        Layer
    UserData

SceneObject-valued attributes

Attributes with type TYPE_SCENE_OBJECT, TYPE_SCENE_OBJECT_VECTOR or TYPE_SCENE_OBJECT_INDEXABLE hold references to scene objects. An example is the attribute “geometry” on MeshLight, referencing the geometry object that is to act as a light source.

Attributes defined with one of these types can specify an “interface set’ restricting the type of objects that they can refer to. SceneObjectInterface is an int acting as a bitset, with these values:

INTERFACE_GENERIC               INTERFACE_GEOMETRYSET
INTERFACE_LAYER                 INTERFACE_LIGHTSET 
INTERFACE_NODE                  INTERFACE_CAMERA
INTERFACE_ENVMAP                INTERFACE_GEOMETRY 
INTERFACE_LIGHT                 INTERFACE_SHADER
INTERFACE_DISPLACEMENT          INTERFACE_MAP 
INTERFACE_MATERIAL              INTERFACE_USERDATA
INTERFACE_METADATA              INTERFACE_LIGHTFILTER
INTERFACE_TRACESET              INTERFACE_JOINT
INTERFACE_LIGHTFILTERSET        INTERFACE_SHADOWSET
INTERFACE_NORMALMAP             INTERFACE_DISPLAYFILTER
INTERFACE_SHADOWRECEIVERSET     INTERFACE_ROOTSHADER
INTERFACE_VOLUMESHADER          INTERFACE_RENDEROUTPUT
INTERFACE_DWABASELAYERABLE      INTERFACE_DWABASEHAIRLAYERABLE

which correspond roughly to the SceneObject base classes in scene_rdl2, with a couple of additions. Each SceneObject subclass declares itself to belong to one or more of these interfaces, and each SceneObject-valued attribute can list the interfaces it allows.

SceneObject-valued attributes are always allowed to hold a null reference, meaning “no object”, and that is always, in practice, the default value for these attributes.

Attribute binding

Attributes with the bindable flag set can be bound to Map objects. Then the result of evaluating the map at each sample is used to drive the attribute value. Map objects include ImageMap, used to drive an attribute from an image texture, and procedural maps like CheckerboardMap.

These are the main SceneObject functions to get and set bindings:

template <typename T> SceneObject* getBinding(AttributeKey<T> key);
template <typename T> void setBinding(AttributeKey<T> key, 
                                      SceneObject* sceneObject);

There are a few other variants of these in defined in SceneObject.h, including functions to set by attribute name.

Attribute bindings are in addition to attribute values : a bound attribute has both a value and an object that it is bound to. The implementation of scene object containing the bound attribute has access to both of these : strictly speaking, it can interpret them as it likes. However the usual procedure is to evaluate the map at a sample and multiply that result by the value. The MoonRay shading API (lib/rendering/shading) has convenient functions to perform the usual evaluation steps.

Map objects always produce a single RGB color output : there is no mechanism supporting multiple different named outputs from a single map, or alternative types. Maps can be bound to Vec2 and Vec3 attributes : r,g,b are interpreted as elements 0,1,2. Float attributes can also be bound : the float value is taken as the average of r,g and b.

One thing to be careful of when translating from other representations to RDL2 is that the default value of many attributes is 0 (or (0,0,0)) – which may not be what you want, since it usually acts as a multiplier. You may have to update the value to 1 or (1,1,1) after setting a binding.

Blurrable attributes

Attributes with the blurrable flag set have two values, indexed by TIMESTEP_BEGIN (= 0) and TIMESTEP_END (= 1). MoonRay will linearly interpolate/extrapolate the two values as required by the SceneVariable motion step and frame settings, which are not discussed in detail here.

enum AttributeTimestep {
    TIMESTEP_BEGIN = 0,
    TIMESTEP_END   = 1,
    NUM_TIMESTEPS  = 2 // not a valid timestep
};
template <typename T> const T& get(AttributeKey<T> key,
                                   AttributeTimestep timestep) const;
template <typename T> void set(AttributeKey<T> key, const T& value, 
                               AttributeTimestep timestep);

Reading and writing

The functions readSceneFromFile and writeSceneToFile, mentioned in the first section, determine file type (RDLA or RDLB) from the file extension (.rdla or .rdlb).

writeSceneToFile() has an additional mode, used if the filepath you specify has no extension. If will then create both a .rdla and a .rdlb file, putting smaller attribute data into the .rdla file and larger attribute data into to the .rdlb. This can be very useful if you are debugging a scene that contains large geometry objects. Vector attributes that contain more than 12 elements are considered to be “large”.

The full set of arguments is:

void writeSceneToFile(const SceneContext& context, 
                      const std::string& filePath,
                      bool deltaEncoding, 
                      bool skipDefaults, 
                      size_t elemsPerLine);

deltaEncoding causes the function to only write out attributes that have changed since the last call to commitAllChanges(). This flag is used by Arras clients to send incremental changes to a MoonRay render process.

skipDefaults doesn’t write out values that are at their default value

elemsPerLine only applies to RDLA format : if greater than zero it limits the number of array elements written on a single line.

The lower-level implementations used by these functions are in the AsciiReader, AsciiWriter, BinaryReader and BinaryWriter classes. You can use these directly to read/write from standard C++ streams or strings.

Python API

scene_rdl2 has Python bindings created with boost::python. Usage generally follows the C++ APIs.

import scene_rdl2

context = scene_rdl2.SceneContext()
context.setProxyModeEnabled(True)
context.loadAllSceneClasses()

camera = context.createSceneObject("PerspectiveCamera", "shot_cam")
camera.set("focal", 24.98)
camera.set("mb_shutter_open", -0.6)
camera.set("mb_shutter_close", 0)

scenevars = context.getSceneVariables()
scenevars.set("camera", camera)

scene_rdl2.writeSceneToFile(context, "example.rdla")

Writing new scene classes

The base SceneObject subclasses (e.g. Node, Camera) provide some examples of how to write new scene classes. There are many other examples in the MoonRay code.

Generally, you need to declare a static AttributeKey member for each new attribute. This code is from PerspectiveCamera:

using namespace scene_rdl2;

RDL2_DSO_ATTR_DECLARE

    rdl2::AttributeKey<rdl2::Float> attrFocalKey;
    rdl2::AttributeKey<rdl2::Int>   attrStereoView;
    ...

RDL2_DSO_ATTR_DEFINE(<baseclass>) is followed by the body of the declare() function, which creates the attributes and adds them to the class:

RDL2_DSO_ATTR_DEFINE(rdl2::Camera)

    attrFocalKey = sceneClass.declareAttribute<rdl2::Float>("focal", 30.0f, rdl2::FLAGS_BLURRABLE);

    attrStereoView = sceneClass.declareAttribute<rdl2::Int>("stereo_view", 0, rdl2::FLAGS_ENUMERABLE, rdl2::INTERFACE_GENERIC, { "stereo view" });
    sceneClass.setMetadata(attrStereoView, "label", "stereo view");
    sceneClass.setEnumValue(attrStereoView, 0, "center view");
    sceneClass.setEnumValue(attrStereoView, 1, "left view");
    sceneClass.setEnumValue(attrStereoView, 2, "right view");

    ...
RDL2_DSO_ATTR_END

You can also define attribute groups : these are used by applications to organise attributes in their UI:

sceneClass.setGroup("Frustum", attrFocalKey);
sceneClass.setGroup("Stereo", attrStereoView);