Touch What You Say: Contact-Local Tactile Grounding for Physical-Property Instructions

Le Wang, Kelan Peng, Zhang Chenyu, Wenjuan Han


Abstract
While vision-language-action (VLA) models excel at robotic manipulation, they frequently fall short when interpreting physical-property language, where adjectives such as fragile or coarse-grained specify constraints that are often visually ambiguous and should modulate behavior primarily during active contact. We therefore propose Contact-Local Semantic Actuation (CLSA), an architecture that extracts semantically structured tactile representations through a language-guided bottleneck and injects them through a contact-gated residual path, confining physical adjustments to contact-relevant stages without perturbing stable nominal motion. To probe instruction-conditioned behavior under matched contexts, we introduce the Action-Property-Constraint (APC) hierarchy. On the language side, APC contrasts action-only (L0), property-aware (L1), and constraint-explicit (L2) prompts; on the trajectory side, offline evaluation uses the corresponding demonstrated behavior strata defined by the data protocol. Across matched offline comparisons, a 140-trial comparative real-robot benchmark, and a separate 240-rollout APC sweep on four tactile-sensitive tasks, CLSA improves instruction sensitivity while maintaining action accuracy. We also report visually ambiguous black-bag trials within the comparative benchmark as a diagnostic of touch-based material recognition after contact.
Anthology ID:
2026.emnlp-main.1120
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
25635–25648
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1120/
DOI:
Bibkey:
Cite (ACL):
Le Wang, Kelan Peng, Zhang Chenyu, and Wenjuan Han. 2026. Touch What You Say: Contact-Local Tactile Grounding for Physical-Property Instructions. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 25635–25648, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Touch What You Say: Contact-Local Tactile Grounding for Physical-Property Instructions (Wang et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1120.pdf