Skip to content

From 'Here' to 'There': Exploring Proximity Semantics in Multimodal Data Exploration

Home / Papers / From 'Here' to 'There': Exploring Proximity Semantics in Multimodal Data Exploration

From ‘Here’ to ‘There’: Exploring Proximity Semantics in Multimodal Data Exploration

Authors: Dennis Bromley, Diana Wang, Vidya Setlur Date: 2026-05-04 Paper ID: openalex:2605.02261

Summary

This paper explores how users interact with time-series and geospatial data using multimodal inputs, specifically focusing on the intersection of sketching, natural language, and visual annotations. The authors introduce a hybrid architecture that combines geometric sketch matching with visual language models to interpret these interleaved query methods. Through a preliminary study, they identify ‘Proximity Semantics’ (PS), a phenomenon where the spatial and temporal closeness of multimodal elements serves as a critical mechanism for disambiguating analytical intent. The work establishes PS as a foundational lens for the design of future multimodal data exploration systems.

Key Contributions

  • Introduces a multimodal research probe integrating sketching, natural language, and visual annotations for time-series and geospatial data exploration.
  • Proposes ‘Proximity Semantics’ (PS) as a conceptual framework for deictic disambiguation in multimodal human-computer interaction.
  • Demonstrates through user study that participants naturally rely on relative placement of inputs to resolve ambiguity in analytical queries.

Open Questions & Future Work

Key Concepts

  • proximity-semantics: A form of deictic disambiguation where the meaning of multimodal query elements is determined by their relative spatial and temporal proximity in an interaction space.

Archivist Review

The concept ‘Proximity Semantics’ was approved as it introduces a novel theoretical lens for multimodal human-computer interaction that is likely to be reused in interface design research. The open question regarding its generalizability was refined and approved as it identifies a critical gap in maturing this framework from a preliminary observation to a universal design principle. No datasets were approved as none were mentioned.

Approved Concepts

  • Proximity Semantics: The paper introduces Proximity Semantics as a core conceptual lens for understanding how users disambiguate multimodal inputs through their relative placement.

Approved Open Questions

  • Generalizability of Proximity Semantics: Validating PS as a robust analytical lens is necessary for designing consistent, user-centric query interfaces that transcend individual experimental platforms.

Metadata & Links

url
https://arxiv.org/abs/2605.02261
paper_id
2605.02261
paper_source
openalex
domain
nlp
tags
nlp
architectures
datasets
concept_slugs
proximity-semantics
dataset_slugs
skill
TimeSeriesSkill
created_at
2026-05-07T06:05:08Z