Skip to content

Commit c7df70e

Browse files
authored
feat(geometry): content-dedup identical representation items (skip re-meshing) (#1130)
Boolean-heavy models exported without IfcMappedItem (structural-steel detailers emitting every plate/bolt as its own item) re-mesh + re-CSG thousands of byte-identical solids; with the exact pure-Rust kernel (#1024) this dominated load time and caused the #1109-class "hangs at 95%" reports. A 128-bit structural hash of each resolved representation-item subtree keys a shared cache of the LOCAL, void-free, colour-free item mesh. A hit skips meshing+CSG; geometry_id (colour/palette/texture), voids and placement stay per-instance, so a hit is byte-identical to a fresh build — no styled-item gate needed. Key folds tessellation quality + unit scale + RTC so a setTessellationQuality change is never served a stale mesh (#976). Cache is Arc<Mutex<_>> shared across one model's routers (native: rayon pool; wasm: per-worker, persisted on IfcAPI and cleared on model swap). Measured (serial, exact kernel): 19.5 MB steel 182s->24s (7.6x, 0/15291 mismatches); 35 MB architectural w/ 82 voided hosts 4.78s->4.49s (0/6574). Committed CI guards + gated A/B harness; geometry 479 + processing 62 suites green.
1 parent 9fca359 commit c7df70e

8 files changed

Lines changed: 754 additions & 32 deletions

File tree

rust/geometry/src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ pub use profile::{Profile2D, Profile2DWithVoids, ProfileType, VoidInfo};
126126
pub use profile_extractor::{extract_profiles, ExtractedProfile};
127127
pub use profiles::ProfileProcessor;
128128
pub use router::{
129-
ClassificationStats, GeometryProcessor, GeometryRouter, HostOpeningDiagnostic,
129+
ClassificationStats, GeometryProcessor, GeometryRouter, HostOpeningDiagnostic, ItemDedupCache,
130130
OpeningDiagnostic, OpeningKindDiag,
131131
};
132132
pub use tessellation::{scale_segments, TessellationQuality};
Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
// This Source Code Form is subject to the terms of the Mozilla Public
2+
// License, v. 2.0. If a copy of the MPL was not distributed with this
3+
// file, You can obtain one at https://mozilla.org/MPL/2.0/.
4+
5+
//! Structural content hash of an IFC representation ITEM subtree, for geometry
6+
//! deduplication of the meshing + CSG compute.
7+
//!
8+
//! Tekla (and other steel detailers) export thousands of geometrically identical
9+
//! parts — connection plates, bolts — each with its OWN representation item
10+
//! rather than sharing one via `IfcMappedItem`. The Manifold kernel chewed
11+
//! through the redundant booleans fast; the exact pure-Rust kernel (#1024) is
12+
//! ~20-40× slower per cut, so re-meshing+re-CSG'ing the duplicates dominates load
13+
//! time (a 19.5 MB Tekla model: 83% of 15k items are byte-duplicates).
14+
//!
15+
//! This hashes the FULLY RESOLVED item subtree (entity references followed to
16+
//! their values), so two geometrically identical items with different entity
17+
//! numbers map to the SAME key. It deliberately covers ONLY geometry-defining
18+
//! structure: colour/style (`IfcStyledItem` points INTO the item from outside,
19+
//! so it is never in the closure), the per-instance `geometry_id`, voids and
20+
//! placement all live OUTSIDE the item and stay per-instance — the cache holds a
21+
//! colour-free local mesh that every instance reuses with its own attributes.
22+
//!
23+
//! The hash is 128-bit over the COMPLETE structure (every attribute value,
24+
//! recursively), unlike the sampled 64-bit mesh hash that collided in #833. The
25+
//! collision probability across a model's items is ~1e-30, so no post-mesh
26+
//! equality fallback is needed. Deterministic (integer splitmix64, no float
27+
//! ordering beyond the bit pattern), so native x86_64/aarch64 and wasm32 produce
28+
//! identical keys.
29+
30+
use ifc_lite_core::{AttributeValue, EntityDecoder};
31+
use rustc_hash::FxHashMap;
32+
33+
/// Defensive recursion bound. IFC geometry is a DAG (item → solids → profiles →
34+
/// points); deeply NESTED `IfcBooleanResult` chains are the realistic deep case,
35+
/// so this is set well above any plausible cut chain. Beyond it the hash falls
36+
/// back to an entity-id-distinct value (see `sig_entity`) so over-depth subtrees
37+
/// can never COLLIDE — they simply stop deduping rather than risk a false merge.
38+
const MAX_DEPTH: u32 = 256;
39+
40+
/// Sentinel written into the memo while an entity's hash is being computed, so a
41+
/// (malformed) cycle resolves to a fixed value instead of recursing forever.
42+
const CYCLE_SENTINEL: u128 = 0xC1C1_C1C1_C1C1_C1C1_C1C1_C1C1_C1C1_C1C1;
43+
44+
#[inline]
45+
fn mix64(mut x: u64) -> u64 {
46+
// splitmix64 finalizer — strong avalanche, same as `geom_hash::mix64`.
47+
x = (x ^ (x >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
48+
x = (x ^ (x >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
49+
x ^ (x >> 31)
50+
}
51+
52+
/// Fold a 64-bit value into a 128-bit running state across two independent lanes.
53+
#[inline]
54+
fn fold(state: u128, v: u64) -> u128 {
55+
let lo = state as u64;
56+
let hi = (state >> 64) as u64;
57+
let lo2 = mix64(lo.wrapping_add(v).wrapping_mul(0x9E37_79B9_7F4A_7C15));
58+
let hi2 = mix64(
59+
hi.rotate_left(23) ^ v.wrapping_mul(0xC2B2_AE3D_27D4_EB4F).wrapping_add(0x1656_67B1),
60+
);
61+
((hi2 as u128) << 64) | (lo2 as u128)
62+
}
63+
64+
#[inline]
65+
fn fold_bytes(mut state: u128, bytes: &[u8]) -> u128 {
66+
state = fold(state, bytes.len() as u64);
67+
let mut chunks = bytes.chunks_exact(8);
68+
for c in &mut chunks {
69+
state = fold(state, u64::from_le_bytes(c.try_into().unwrap()));
70+
}
71+
let rem = chunks.remainder();
72+
if !rem.is_empty() {
73+
let mut buf = [0u8; 8];
74+
buf[..rem.len()].copy_from_slice(rem);
75+
state = fold(state, u64::from_le_bytes(buf));
76+
}
77+
state
78+
}
79+
80+
/// 128-bit structural hash of the representation item rooted at `root_id`. `memo`
81+
/// caches per-entity hashes so shared sub-entities (a profile reused by many
82+
/// solids, the representation context) are visited once; it keys on entity ids,
83+
/// so it must belong to ONE model (the `GeometryRouter` owns one per loaded
84+
/// file).
85+
pub fn item_signature(decoder: &mut EntityDecoder, root_id: u32, memo: &mut FxHashMap<u32, u128>) -> u128 {
86+
sig_entity(decoder, root_id, memo, 0)
87+
}
88+
89+
/// Combine the pure structural item hash with the router parameters that change
90+
/// the MESHED output but live outside the IFC structure — tessellation quality
91+
/// (curved profiles tessellate finer at higher quality), unit scale, and RTC
92+
/// offset. Without this, a cache shared across routers — or one that outlives a
93+
/// `setTessellationQuality` change on the same worker — would serve a mesh built
94+
/// under different parameters (e.g. #976: every quality level returns the
95+
/// first-cached triangle count).
96+
pub fn key_with_params(structural: u128, quality_index: u8, unit_scale: f64, rtc: (f64, f64, f64)) -> u128 {
97+
let mut s = fold(structural, quality_index as u64);
98+
s = fold(s, unit_scale.to_bits());
99+
s = fold(s, rtc.0.to_bits());
100+
s = fold(s, rtc.1.to_bits());
101+
fold(s, rtc.2.to_bits())
102+
}
103+
104+
fn sig_entity(decoder: &mut EntityDecoder, id: u32, memo: &mut FxHashMap<u32, u128>, depth: u32) -> u128 {
105+
if let Some(&s) = memo.get(&id) {
106+
return s;
107+
}
108+
if depth > MAX_DEPTH {
109+
// Fold the entity id so two DIFFERENT over-depth subtrees get DIFFERENT
110+
// values — they stop deduping (id breaks renumbering-invariance) but can
111+
// never false-merge, which matters far more than deduping a pathological
112+
// boolean chain.
113+
return fold(0xDEAD_BEEF_DEAD_BEEF, id as u64);
114+
}
115+
memo.insert(id, CYCLE_SENTINEL); // break cycles (DAG ⇒ unreachable in practice)
116+
let entity = match decoder.decode_by_id(id) {
117+
Ok(e) => e,
118+
Err(_) => {
119+
// Unresolvable reference: a fixed sentinel (NOT the id, so structurally
120+
// identical-but-renumbered files still collide).
121+
let s = fold(0, 0x00BA_D0BA_D0BA_D000);
122+
memo.insert(id, s);
123+
return s;
124+
}
125+
};
126+
// Hash the stable type NAME (IfcType isn't a primitive-castable enum).
127+
let mut acc = fold_bytes(fold(0, 0x5EED_5EED), entity.ifc_type.as_str().as_bytes());
128+
for attr in &entity.attributes {
129+
acc = hash_attr(decoder, attr, acc, memo, depth);
130+
}
131+
memo.insert(id, acc);
132+
acc
133+
}
134+
135+
fn hash_attr(
136+
decoder: &mut EntityDecoder,
137+
attr: &AttributeValue,
138+
acc: u128,
139+
memo: &mut FxHashMap<u32, u128>,
140+
depth: u32,
141+
) -> u128 {
142+
match attr {
143+
AttributeValue::EntityRef(r) => {
144+
let child = sig_entity(decoder, *r, memo, depth + 1);
145+
// Fold both lanes of the child hash, tagged.
146+
fold(fold(fold(acc, 1), child as u64), (child >> 64) as u64)
147+
}
148+
AttributeValue::String(s) => fold_bytes(fold(acc, 2), s.as_bytes()),
149+
AttributeValue::Integer(i) => fold(fold(acc, 3), *i as u64),
150+
AttributeValue::Float(f) => fold(fold(acc, 4), f.to_bits()),
151+
AttributeValue::Enum(e) => fold_bytes(fold(acc, 5), e.as_bytes()),
152+
AttributeValue::List(items) => {
153+
let mut a = fold(fold(acc, 6), items.len() as u64);
154+
for it in items {
155+
a = hash_attr(decoder, it, a, memo, depth);
156+
}
157+
a
158+
}
159+
AttributeValue::Null => fold(acc, 8),
160+
AttributeValue::Derived => fold(acc, 9),
161+
}
162+
}

rust/geometry/src/router/mod.rs

Lines changed: 87 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
99
mod caching;
1010
mod clipping;
11+
mod content_hash;
1112
mod layers;
1213
mod processing;
1314
mod transforms;
@@ -33,7 +34,7 @@ use nalgebra::Matrix4;
3334
use rustc_hash::FxHashMap;
3435
use std::cell::RefCell;
3536
use std::collections::HashMap;
36-
use std::sync::Arc;
37+
use std::sync::{Arc, Mutex};
3738

3839
/// Geometry processor trait
3940
/// Each processor handles one type of IFC representation
@@ -57,6 +58,14 @@ pub trait GeometryProcessor {
5758
fn supported_types(&self) -> Vec<IfcType>;
5859
}
5960

61+
/// Shared content-dedup cache: maps a 128-bit structural item hash to the
62+
/// LOCAL (pre-placement, void-free, colour-free) item mesh. Build ONE per loaded
63+
/// model with [`GeometryRouter::new_dedup_cache`] and inject it into every
64+
/// per-element / per-batch router via
65+
/// [`GeometryRouter::enable_content_dedup_shared`] so byte-identical geometry is
66+
/// meshed once regardless of how the work is partitioned across threads/batches.
67+
pub type ItemDedupCache = Arc<Mutex<FxHashMap<u128, Arc<Mesh>>>>;
68+
6069
/// Geometry router - routes entities to processors
6170
pub struct GeometryRouter {
6271
schema: IfcSchema,
@@ -68,6 +77,25 @@ pub struct GeometryRouter {
6877
/// Buildings with repeated floors have 99% identical geometry
6978
/// Key: Hash of mesh content, Value: Processed mesh
7079
geometry_hash_cache: RefCell<FxHashMap<u64, Arc<Mesh>>>,
80+
/// SHARED content-dedup of LOCAL (pre-placement, void-free) representation-ITEM
81+
/// meshes, keyed by a 128-bit structural hash of the item subtree
82+
/// (`content_hash::item_signature`). Skips the meshing + CSG for byte-identical
83+
/// geometry the exporter failed to share via `IfcMappedItem` (Tekla connection
84+
/// plates/bolts). The cached mesh is COLOUR-FREE; the per-instance
85+
/// `geometry_id` (colour/palette/texture), voids and placement are applied by
86+
/// the caller, so reuse never changes an instance's appearance.
87+
///
88+
/// `Arc<Mutex<_>>` so ONE cache outlives any single router and is shared across
89+
/// the native rayon pool's per-element routers AND a wasm worker's per-batch
90+
/// routers (re-injected each batch). A hit skips the expensive build entirely,
91+
/// so the lock is held only for a map get/clone (hit) or insert (miss); the
92+
/// build runs outside it. `None` ⇒ dedup disabled (e.g. `new()` in tests).
93+
item_dedup_cache: Option<ItemDedupCache>,
94+
/// Per-router memo for the per-item structural hash (shared sub-entities hashed
95+
/// once). Keyed by entity id ⇒ valid for one loaded model. Kept LOCAL (not
96+
/// shared) so the recursive DAG walk never contends the shared cache's lock;
97+
/// recomputing it per router is cheap next to meshing.
98+
content_sig_memo: RefCell<FxHashMap<u32, u128>>,
7199
/// Unit scale factor (e.g., 0.001 for millimeters -> meters)
72100
/// Applied to all mesh positions after processing
73101
unit_scale: f64,
@@ -204,6 +232,8 @@ impl GeometryRouter {
204232
processors: HashMap::new(),
205233
mapped_item_cache: RefCell::new(FxHashMap::default()),
206234
geometry_hash_cache: RefCell::new(FxHashMap::default()),
235+
item_dedup_cache: None, // armed by `with_units` / `enable_content_dedup_shared`
236+
content_sig_memo: RefCell::new(FxHashMap::default()),
207237
unit_scale: 1.0, // Default to base meters
208238
rtc_offset: (0.0, 0.0, 0.0), // Default to no offset
209239
material_layer_index: None,
@@ -250,21 +280,25 @@ impl GeometryRouter {
250280
where
251281
T: AsRef<[u8]> + ?Sized,
252282
{
253-
let content = content.as_ref();
254-
let mut scanner = ifc_lite_core::EntityScanner::new(content);
255-
let mut scale = 1.0;
283+
let scale = Self::scan_unit_scale(content.as_ref(), decoder);
284+
let mut router = Self::with_scale(scale);
285+
router.arm_content_dedup();
286+
router
287+
}
256288

257-
// Scan through file to find IFCPROJECT
289+
/// Scan to the first `IFCPROJECT` and extract its length-unit scale (e.g.
290+
/// `0.001` for millimetres → metres); `1.0` if none is found.
291+
fn scan_unit_scale(content: &[u8], decoder: &mut EntityDecoder) -> f64 {
292+
let mut scanner = ifc_lite_core::EntityScanner::new(content);
258293
while let Some((id, type_name, _, _)) = scanner.next_entity() {
259294
if type_name == "IFCPROJECT" {
260295
if let Ok(s) = ifc_lite_core::extract_length_unit_scale(decoder, id) {
261-
scale = s;
296+
return s;
262297
}
263298
break;
264299
}
265300
}
266-
267-
Self::with_scale(scale)
301+
1.0
268302
}
269303

270304
/// Create router with unit scale extracted from IFC file AND RTC offset for large coordinates
@@ -282,21 +316,10 @@ impl GeometryRouter {
282316
where
283317
T: AsRef<[u8]> + ?Sized,
284318
{
285-
let content = content.as_ref();
286-
let mut scanner = ifc_lite_core::EntityScanner::new(content);
287-
let mut scale = 1.0;
288-
289-
// Scan through file to find IFCPROJECT
290-
while let Some((id, type_name, _, _)) = scanner.next_entity() {
291-
if type_name == "IFCPROJECT" {
292-
if let Ok(s) = ifc_lite_core::extract_length_unit_scale(decoder, id) {
293-
scale = s;
294-
}
295-
break;
296-
}
297-
}
298-
299-
Self::with_scale_and_rtc(scale, rtc_offset)
319+
let scale = Self::scan_unit_scale(content.as_ref(), decoder);
320+
let mut router = Self::with_scale_and_rtc(scale, rtc_offset);
321+
router.arm_content_dedup();
322+
router
300323
}
301324

302325
/// Create router with pre-calculated unit scale
@@ -306,6 +329,47 @@ impl GeometryRouter {
306329
router
307330
}
308331

332+
/// Arm content-dedup with a NEW empty cache. Used by the model constructors
333+
/// (`with_units*`) where this router owns the only reference; multi-router
334+
/// callers (native pool, wasm batches) should build ONE shared cache via
335+
/// [`Self::new_dedup_cache`] and inject it into every router with
336+
/// [`Self::enable_content_dedup_shared`] so the cache persists across them.
337+
fn arm_content_dedup(&mut self) {
338+
self.item_dedup_cache = Some(Self::new_dedup_cache());
339+
}
340+
341+
/// A fresh empty shared item-dedup cache, to be cloned into every per-element /
342+
/// per-batch router of ONE loaded model so they all dedup against it. Keep one
343+
/// per model: the key is a per-model entity-structure hash, and the cached
344+
/// meshes bake in this model's unit scale / tessellation quality.
345+
pub fn new_dedup_cache() -> ItemDedupCache {
346+
Arc::new(Mutex::new(FxHashMap::default()))
347+
}
348+
349+
/// Inject a shared item-dedup cache (see [`Self::new_dedup_cache`]) into this
350+
/// router. All routers given the SAME `Arc` dedup against one cache, so
351+
/// byte-identical geometry is meshed once across the whole model regardless of
352+
/// how elements are partitioned across threads or batches.
353+
pub fn enable_content_dedup_shared(&mut self, cache: ItemDedupCache) {
354+
self.item_dedup_cache = Some(cache);
355+
}
356+
357+
/// Disable content-dedup (drops the cache reference so `item_dedup_key`
358+
/// returns `None` and meshing is never skipped). Test/bench helper for an A/B
359+
/// against the deduped path.
360+
pub fn disable_content_dedup(&mut self) {
361+
self.item_dedup_cache = None;
362+
}
363+
364+
/// Number of unique item meshes cached by content-dedup so far — the reuse the
365+
/// pipeline recovered (vs. the meshed-item count). Diagnostics.
366+
pub fn dedup_unique_count(&self) -> usize {
367+
self.item_dedup_cache
368+
.as_ref()
369+
.map(|c| c.lock().expect("dedup cache poisoned").len())
370+
.unwrap_or(0)
371+
}
372+
309373
/// Create router with RTC offset for large coordinate handling
310374
/// Use this for georeferenced models (e.g., Swiss UTM coordinates)
311375
pub fn with_rtc(rtc_offset: (f64, f64, f64)) -> Self {

0 commit comments

Comments
 (0)