📄 全文中文译稿已上线:论文库译稿页(保留全部图表,含工程向译者解读),需要逐节细节请直接读译稿。

R2SGEN:从真实数据到仿真场景的约束求解生成框架——OOPSLA 2026论文解读

论文信息

项目 内容
标题 Real-to-Sim Generation: Synthesizing Scenario Programs from Real-World Data via Constraint Solving
作者 Peishan Huang, Wenmeng Zhang, Yusen Chen, Zhenbang Chen
机构 National University of Defense Technology, Changsha, China
会议 OOPSLA 2026 (SPLASH)
DOI 10.1145/3839527
arXiv 2609.23342
页数 31页, 4图, 4表
语言 Programming Languages (cs.PL)

核心创新

R2SGEN解决了合成数据生成中的sim-to-real gap问题:从真实世界数据反向合成结构化场景程序。关键创新是解耦合成策略:将离散结构搜索与连续几何求解分离,用轻量级原子SMT约束替代整体SMT编码。

一句话总结:给定一个真实驾驶场景的坐标快照,自动合成可泛化的Scenic场景程序,通过约束求解保证物理合理性。

问题背景

合成数据的困境

方法 优势 问题
神经网络生成 高逼真度 黑箱,无法精确控制语义元素
LLM生成 自然语言→代码 幻觉问题,无法形式化验证
手写Scenic 精确可控 需专家,高成本,低效率
R2SGEN 真实→仿真 物理合理+可泛化+形式验证

从坐标到关系的挑战

1
2
3
4
5
6
7
8
9
10
给定真实场景:
ego: (0, 0, 0°)
car1: (3, 8, 0°)
car2: (3, 5, 0°)
car3: (0, 5, 0°)

目标程序:
car3 = new Car ahead of ego by Range(3,8)
car2 = new Car right of car3 by Range(1,6)
car1 = new Car ahead of car2 by Range(1,6)

挑战1: 验证程序正确性(关系描述是否满足坐标?)
挑战2: 搜索空间组合爆炸(O(N!)实体排列)

R2SGEN框架

整体架构

解耦合成策略

核心思想:将问题分解为离散和连续两部分。

离散部分:结构搜索

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
from enum import Enum
from typing import List, Tuple, Optional
from dataclasses import dataclass

class RelationOperator(Enum):
"""Scenic DSL中的空间关系算子"""
AHEAD_OF = "ahead of"
BEHIND = "behind"
LEFT_OF = "left of"
RIGHT_OF = "right of"
AT = "at" # 绝对位置(最简单)

class FacingOperator(Enum):
"""朝向关系算子"""
AWAY_FROM = "away from"
TOWARD = "toward"
SAME_DIRECTION = "same direction as"
OPPOSITE = "opposite to"

@dataclass
class EntityDeclaration:
"""实体声明"""
entity_id: str # car1, car2, ...
entity_type: str # Car, Pedestrian, ...
reference_entity: str # 参照实体
relation: RelationOperator # 空间关系
facing: FacingOperator # 朝向关系
distance_range: Tuple[float, float] # 距离范围

@dataclass
class SceneProgram:
"""场景程序"""
declarations: List[EntityDeclaration]

def to_scenic_code(self) -> str:
"""生成Scenic代码"""
lines = ["ego = new Car on 0@0, facing 0 deg"]
for d in self.declarations:
line = f"{d.entity_id} = new {d.entity_type} "
line += f"{d.relation.value} {d.reference_entity} "
line += f"by Range({d.distance_range[0]}, {d.distance_range[1]}), "
line += f"facing {d.facing.value} {d.reference_entity}"
lines.append(line)
return "\n".join(lines)


class StructuralSearch:
"""
R2SGEN离散结构搜索

搜索空间:
1. 实体声明顺序(O(N!)排列)
2. 参照实体选择(每个实体可引用任意已声明实体)
3. 关系算子选择(ahead/behind/left/right)

通过两种剪枝策略加速:
- BFS + Common Prefix Abstraction
- DFS + Branch-and-Bound
"""

def __init__(self, entities: List[dict]):
"""
Args:
entities: 真实场景实体列表
[{'id': 'car1', 'type': 'Car', 'pos': (3,8), 'heading': 0}, ...]
"""
self.entities = entities
self.n = len(entities)
self.best_program = None
self.best_cost = float('inf')

def search_bfs(self) -> Optional[SceneProgram]:
"""BFS + Common Prefix Abstraction Pruning"""
from collections import deque

# 搜索状态:(声明序列, 已用实体, 当前成本)
initial_state = ([], set(), 0)
queue = deque([initial_state])

while queue:
declarations, used, cost = queue.popleft()

# 剪枝:成本已超过最优
if cost >= self.best_cost:
continue

# 终止条件:所有实体已声明
if len(declarations) == self.n:
if cost < self.best_cost:
self.best_cost = cost
self.best_program = SceneProgram(declarations)
continue

# Common Prefix Abstraction 剪枝
prefix = tuple(d.entity_id for d in declarations)
if self._is_dominated_prefix(prefix, cost):
continue

# 扩展:为下一个实体选择参照和关系
for entity in self.entities:
if entity['id'] in used:
continue

for ref in [d.entity_id for d in declarations] + ['ego']:
for rel in RelationOperator:
for face in FacingOperator:
new_decl = EntityDeclaration(
entity_id=entity['id'],
entity_type=entity['type'],
reference_entity=ref,
relation=rel,
facing=face,
distance_range=(1, 10)
)

# SMT验证(连续部分)
if self._verify_smt(new_decl, declarations):
new_cost = cost + self._cost(new_decl)
new_used = used | {entity['id']}
queue.append(
(declarations + [new_decl], new_used, new_cost)
)

# 记录前缀以供剪枝
self._record_prefix(prefix, cost)

return self.best_program

def search_dfs(self) -> Optional[SceneProgram]:
"""DFS + Branch-and-Bound Pruning"""

def dfs_recursive(declarations, used, cost):
if cost >= self.best_cost:
return # Branch-and-Bound 剪枝

if len(declarations) == self.n:
if cost < self.best_cost:
self.best_cost = cost
self.best_program = SceneProgram(declarations)
return

for entity in self.entities:
if entity['id'] in used:
continue

for ref in [d.entity_id for d in declarations] + ['ego']:
for rel in RelationOperator:
for face in FacingOperator:
new_decl = EntityDeclaration(
entity_id=entity['id'],
entity_type=entity['type'],
reference_entity=ref,
relation=rel,
facing=face,
distance_range=(1, 10)
)

if self._verify_smt(new_decl, declarations):
new_cost = cost + self._cost(new_decl)
dfs_recursive(
declarations + [new_decl],
used | {entity['id']},
new_cost
)

dfs_recursive([], set(), 0)
return self.best_program

def _verify_smt(self, new_decl, existing) -> bool:
"""SMT约束验证(简化版)"""
# 实际使用Z3 SMT solver
# 这里简化为几何验证
entity = next(e for e in self.entities
if e['id'] == new_decl.entity_id)
ref_entity = next(
(e for e in self.entities if e['id'] == new_decl.reference_entity),
{'pos': (0, 0), 'heading': 0} # ego
)

dx = entity['pos'][0] - ref_entity['pos'][0]
dy = entity['pos'][1] - ref_entity['pos'][1]
dist = (dx**2 + dy**2) ** 0.5

# 验证距离范围
if not (1 <= dist <= 10):
return False

# 验证关系算子
if new_decl.relation == RelationOperator.AHEAD_OF:
if dy <= 0: # ahead = y方向前方
return False
elif new_decl.relation == RelationOperator.BEHIND:
if dy >= 0:
return False
elif new_decl.relation == RelationOperator.LEFT_OF:
if dx >= 0:
return False
elif new_decl.relation == RelationOperator.RIGHT_OF:
if dx <= 0:
return False

return True

def _cost(self, decl: EntityDeclaration) -> float:
"""程序成本(更简洁=更低成本)"""
cost = 0
# 绝对坐标成本最高
if decl.relation == RelationOperator.AT:
cost += 10
# 关系描述成本较低
else:
cost += 1
# 参照ego比参照其他实体成本低
if decl.reference_entity == 'ego':
cost += 0.5
else:
cost += 1.0
return cost

def _is_dominated_prefix(self, prefix, cost):
"""Common Prefix Abstraction 剪枝"""
# 如果相同前缀已被更低成本探索过,剪枝
return self._prefix_cache.get(prefix, float('inf')) <= cost

def _record_prefix(self, prefix, cost):
if not hasattr(self, '_prefix_cache'):
self._prefix_cache = {}
if prefix not in self._prefix_cache or cost < self._prefix_cache[prefix]:
self._prefix_cache[prefix] = cost


# 测试
if __name__ == "__main__":
print("=" * 60)
print("R2SGEN 场景程序合成 - 测试")
print("=" * 60)

# nuScenes示例场景
entities = [
{'id': 'car1', 'type': 'Car', 'pos': (3, 8), 'heading': 0},
{'id': 'car2', 'type': 'Car', 'pos': (3, 5), 'heading': 0},
{'id': 'car3', 'type': 'Car', 'pos': (0, 5), 'heading': 0},
]

# BFS搜索
search = StructuralSearch(entities)

print("\nBFS搜索...")
program = search.search_bfs()
if program:
print(f"最优成本: {search.best_cost:.2f}")
print(f"\n生成的Scenic程序:")
print(program.to_scenic_code())

# DFS搜索
search2 = StructuralSearch(entities)
print("\nDFS搜索...")
program2 = search2.search_dfs()
if program2:
print(f"最优成本: {search2.best_cost:.2f}")
print(f"\n生成的Scenic程序:")
print(program2.to_scenic_code())

SMT约束验证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
from typing import List, Tuple

class SMTConstraintVerifier:
"""
SMT约束验证器(连续几何部分)

将Scenic DSL的几何语义编码为SMT约束
使用Z3 solver验证程序一致性

约束类型:
1. 距离约束:|pos_a - pos_b| in [min, max]
2. 方向约束:heading_a - heading_b in [angle_min, angle_max]
3. 可见性约束:ego can see entity
4. 碰撞约束:entities don't overlap
"""

def __init__(self):
# 简化版:直接几何验证
# 实际使用Z3 SMT solver
pass

def verify_distance(self, pos_a: Tuple[float, float],
pos_b: Tuple[float, float],
dist_range: Tuple[float, float]) -> bool:
"""验证距离约束"""
dist = ((pos_a[0]-pos_b[0])**2 + (pos_a[1]-pos_b[1])**2)**0.5
return dist_range[0] <= dist <= dist_range[1]

def verify_direction(self, heading_a: float, heading_b: float,
angle_range: Tuple[float, float]) -> bool:
"""验证方向约束"""
diff = (heading_a - heading_b + 180) % 360 - 180
return angle_range[0] <= diff <= angle_range[1]

def verify_relation(self, pos_entity, pos_ref, relation,
dist_range=(1, 10)) -> bool:
"""验证空间关系"""
dx = pos_entity[0] - pos_ref[0]
dy = pos_entity[1] - pos_ref[1]
dist = (dx**2 + dy**2)**0.5

if not self.verify_distance(pos_entity, pos_ref, dist_range):
return False

if relation == "ahead of":
return dy > 0 # 前方
elif relation == "behind":
return dy < 0 # 后方
elif relation == "left of":
return dx < 0 # 左侧
elif relation == "right of":
return dx > 0 # 右侧
return True

def verify_no_collision(self, positions: List[Tuple[float, float]],
min_distance: float = 2.0) -> bool:
"""验证无碰撞"""
for i in range(len(positions)):
for j in range(i+1, len(positions)):
dist = ((positions[i][0]-positions[j][0])**2 +
(positions[i][1]-positions[j][1])**2)**0.5
if dist < min_distance:
return False
return True


# 测试
if __name__ == "__main__":
verifier = SMTConstraintVerifier()

# 测试关系验证
print("关系验证测试:")
print(f" (3,8) ahead of (0,0): {verifier.verify_relation((3,8), (0,0), 'ahead of')}")
print(f" (3,5) right of (0,5): {verifier.verify_relation((3,5), (0,5), 'right of')}")
print(f" (0,5) ahead of (0,0): {verifier.verify_relation((0,5), (0,0), 'ahead of')}")

实验结果

nuScenes评估

指标 R2SGEN (BFS) R2SGEN (DFS) LLM基线
场景一致性 ✅ 100% ✅ 100% ⚠️ 部分
程序成本 最低 中等 最高
搜索效率 高(CPA剪枝) 高(B&B剪枝) N/A
可扩展性 中等 高 低
物理合理性 ✅ 保证 ✅ 保证 ❌ 不保证

搜索策略互补性

策略 优势 劣势 适用场景
BFS + CPA 最优解保证 内存消耗大 小规模场景(≤5实体)
DFS + B&B 内存高效 可能非最优 大规模场景(>5实体)

复杂度分析

实体数 排列数 单一SMT(失败) R2SGEN(成功)
3 6 超时 <1s
5 120 超时 <5s
8 40320 超时 <30s
10 3628800 超时 <120s

IMS座舱数据合成启示

1. 座舱场景约束定义

R2SGEN的思路可直接迁移到座舱DMS数据合成:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# 座舱场景约束示例
cabin_constraints = {
"空间约束": {
"驾驶员位置": "fixed at driver seat",
"乘客位置": "Range(0.3, 0.8) m from seat back",
"安全带路径": "across shoulder and lap",
"手部可达范围": "within arm reach sphere R=0.6m"
},
"物理约束": {
"光照": "Range(10, 10000) lux",
"材质反射率": "Range(0.1, 0.9)",
"遮挡关系": "driver body occludes rear seat"
},
"行为约束": {
"头部姿态": "pitch: [-30, 30]°, yaw: [-60, 60]°",
"眼动模式": "saccade + fixation + smooth pursuit",
"手势区域": "center console / steering wheel / window"
},
"时间约束": {
"行为持续时间": "fixation: 0.2-2s, saccade: 20-80ms",
"行为转移概率": "P(phone→driving) = 0.15"
}
}

2. 与NVIDIA Isaac Sim的集成

组件 R2SGEN角色 Isaac Sim角色
场景生成 从真实数据合成程序 执行程序渲染
参数控制 约束求解保证合理性 物理引擎执行
多样性 域随机化参数范围 材质/光照随机
标注 自动语义标注 自动传感器标注

3. 优先级排序

任务 优先级 预期收益
座舱约束定义 🔴 高 基础框架
Scenic座舱DSL扩展 🟡 中 可编程生成
与Isaac Sim集成 🟡 中 端到端管道
真实→仿真验证 🟢 低 闭环验证

总结

R2SGEN代表了合成数据生成从”随机生成”到”真实→仿真反推”的范式转变。核心贡献:

  1. 解耦合成:离散结构搜索+连续SMT求解,解决组合爆炸
  2. 形式验证:SMT约束保证生成的程序与真实场景一致
  3. 两种搜索策略:BFS+CPA和DFS+B&B互补,覆盖不同规模场景
  4. 可泛化:从单一场景合成可变程序,生成多样化变体

对IMS的价值: 可用于从真实驾驶日志反推座舱场景程序,在Isaac Sim中生成多样化训练数据(不同光照、遮挡、行为变体),解决DMS训练数据稀缺问题。

信息来源: arXiv:2609.23342 | OOPSLA 2026 | DOI: 10.1145/3839527