Support for CPU-side Styling

[j9liu]

Background

Our philosophy with cesium-native integrations has always been to bring in 3D Tiles, with the experience being as close as possible to working with any other meshes or objects in the runtime engine. When it came to support for metadata styling in Unreal, we looked to implement this with Unreal's built-in material system. That way, people take advantage of their existing expertise to write complex material graphs.

However, this means that styling inherits the limitations of the material systems. For instance:

• Materials or shaders cannot be generated at runtime. They must be created in-Editor, anticipating all possible cases that a material should handle or in which it should change.
• Properties must be passed in as textures. Custom vertex attributes are typically not supported (defeating the purpose of property attributes)
• Some property types, such as strings and arrays, are not easily supported (and may not be reasonably possible).

Additionally, we are seeing use cases that push the limitations of this system. Notably, the use of UINT64 identifiers to categorize elements in in large design models is becoming increasingly relevant for Unreal. Users want to highlight elements based on these category identifiers, or otherwise toggle the visibility of them. Some examples of how this is accomplished in CesiumJS are:

// Hide specific categories by their int64 element IDs
const hiddenCategories = new Set([
  BigInt("2199023255776"),  // OST_Walls
  BigInt("2199023255724"),  // OST_Ceilings
  BigInt("2199023255990"),  // OST_Floors
  BigInt("2199023255765"),  // OST_Stairs
  BigInt("2199023255795"),  // OST_Roofs
]);

// Apply style that hides features in hidden categories
tileset.style = new Cesium.Cesium3DTileStyle();
tileset.style.show = {
  evaluate(feature) {
    return !hiddenCategories.has(feature.getProperty("category"));
  },
};

// Highlight specific categories by their int64 element IDs
const highlightedCategories = new Set([
  BigInt("2199023255776"),  // OST_Walls
  BigInt("2199023255724"),  // OST_Ceilings
  BigInt("2199023255990"),  // OST_Floors
  BigInt("2199023255765"),  // OST_Stairs
  BigInt("2199023255795"),  // OST_Roofs
]);

// Apply style that colors matching features yellow
tileset.style = new Cesium.Cesium3DTileStyle();
tileset.style.color = {
  evaluateColor(feature, result) {
    if (highlightedCategories.has(feature.getProperty("category"))) {
      return Cesium.Color.YELLOW.withAlpha(1.0, result);
    }
    return Cesium.Color.WHITE.withAlpha(1.0, result);
  },
};

Although we could hack uint64_ts into a material (apparently supported in Shader Model 6!), it would still not provide a comparable user experience. We would still need to handle a dynamic list of categories, no matter if it's to highlight / hide/ show. One could pass the IDs of the affected categories to through a texture, but to act conditionally on them would require something like:

int categoryID;
Texture2D hiddenCategories;
uint width, height;

hiddenCategories.GetDimensions(width, height);
for (uint w = 0; w < width; w++) {
   for (uint h = 0; h < height; h++) {
       if (categoryID == hiddenCategories.Load(w, h)) {
          outColor.a = 0.f;
          break;
       }
   }
}

Even so, dynamic for loop lengths are not supported in HLSL. And even if they were, performance could be questionable given the sheer number of categories at play for a given tileset.

Possible Solutions

CesiumJS already implements a form of CPU styling. Given some style expression (as in the examples above), it evaluates a show/hide condition, or a highlight color one, for all the features in a tileset. The results are saved and applied to the features in the shader.

We may want to do something similar in cesium-native. Evaluate conditions per-feature, then pass the computed color value to the shader. It's questionable whether we want to implement full Javascript-like styling (e.g. CesiumGS/cesium-unreal#1392), but this has also come up for #1380.

Although it may take some time to evaluate the styling conditions across the entire tileset, the one-time computation of complex conditions for consistent framerate afterwards seems like a worthy tradeoff, and it allows for flexibility/dynamism that is difficult to achieve with in-game materials.

[j9liu]

I've been thinking about the above example, as well as the one in Step 3 from the "Working with the Asset Elements API" tutorial:

let selectedElementId = null; // BigInt or null
let hoveredElementId = null;  // BigInt or null

function applyStyle() {
  const style = new Cesium.Cesium3DTileStyle();
  style.color = {
    evaluateColor: function (feature, result) {
      const id = feature.getProperty("element");

      // Hidden elements (2D cleanup): fully transparent
      if (hiddenElementIds.has(id)) {
        return Cesium.Color.clone(Cesium.Color.TRANSPARENT, result);
      }
      // Selected element: yellow highlight
      if (id === selectedElementId) {
        return Cesium.Color.clone(Cesium.Color.YELLOW, result);
      }
      // Hovered element: cyan highlight
      if (id === hoveredElementId) {
        return Cesium.Color.clone(Cesium.Color.CYAN.withAlpha(0.7), result);
      }
      // Query match: green highlight
      if (matchedElementIds.size > 0 && matchedElementIds.has(id)) {
        return Cesium.Color.clone(Cesium.Color.LIME, result);
      }
      // Faded when a selection or query is active
      if (selectedElementId || matchedElementIds.size > 0) {
        return Cesium.Color.clone(Cesium.Color.WHITE.withAlpha(0.02), result);
      }
      // Default: full color
      return Cesium.Color.clone(Cesium.Color.WHITE, result);
    },
  };
  tileset.style = style;
}

...and trying to mock up something similar to the example. Here's some thoughts so far!

Native-based Styling Interface

Similar to our vector styling implementation (#1430), we could create a new interface that allows for some override of a function that resolves to a color for a given feature:

class Style {
   virtual CesiumUtility::Color evaluateColor() = 0;
}

From a public API view in cesium-native, this would be used like:

Cesium3DTilesSelection::Tileset tileset;
tileset.style = new Style();
// if changes need to be picked up...
tileset.refreshStyle();

In #1430 we expose a list of feature IDs as well as the CesiumGltf::Model to resolve them to the model's metadata. However, I kinda like the idea of creating a new Feature class that resolves the metadata properties for the user so it's less troublesome to retrieve:

struct Feature {
   std::map<std::string, CesiumUtility::JsonValue> properties;
}

Of course, there's a drawback if the feature has 200+ properties and a user only needs one. But, from a code perspective it would be a hassle to setup some CesiumGltf::PropertyTableView to retrieve properties from the given feature.

Otherwise, we'd need to do some extra legwork on the Unreal/Unity side, which does already abstract away the C++ typing woes for us, but makes things a little annoying. So similar to what #1430 does:

// in cesium-native
class StylingProvider {
   virtual CesiumUtility::Color evaluateColor(std::vector<int64_t> featureIds, const CesiumGltf::ModelMetadata& metadata) = 0;
}

// in cesium-unreal
class BlueprintStylingProvider { 
  UFUNCTION(BlueprintCallable, BlueprintNativeEvent)
  void OnStylingBegin(const FCesiumModelMetadata& InModelMetadata);

  UFUNCTION(BlueprintCallable, BlueprintNativeEvent)
  FColor OnStyleFeature(int64 InFeatureId);
}

Then putting onus on the user to retrieve the properties they want specifically:

Runtime Consumption

Once the colors are evaluated per-feature, we need some way to actually make them affect the rendering of the model. This is where I imagine taking inspiration from the existing glTF modifier pipeline (#1231), where models can be reloaded with arbitrary modifications.

1. User sets a style on a tileset, triggering per-tile modification.
2. TileRenderContent generates some map of feature IDs to styled colors, e.g., std::map<int64_t, CesiumUtility::Color>.
   a. This implies some gathering of feature IDs in the model beforehand. This could be done once and cached for future styling updates.
   b. Then, call evaluateColor per-feature.
3. Once this map is generated, fire some event / set some state to prompt the runtime side to refresh in some way.

In Unreal, we could consume this in a similar way to what we already do with CesiumFeaturesMetadataComponent. We can load these colors to a texture parameter in the material, then do lookup in the material and output it in the styling layer. It's essentially skipping over whatever material logic the user would have had to add themselves to resolve the same colors 🤷