SymAntGrounding Long-Horizon Action Anticipation with Adaptable Symbolic Priors

WACV 2027
Fnu Aryan1 Yaqi Xie1 Joseph Campbell3 Woojun Kim1 Kwonjoon Lee2 Hossein Nourkhiz Mahjoub2 Ehsan Moradi-Pari2 Katia Sycara1 Simon Stepputtis4
1Carnegie Mellon University 2Honda Research Institute USA 3Purdue University 4Virginia Tech
Paper (arXiv, soon) Code BibTeX
SymAnt at a glance: from two observed frames (cut onion, put in bowl), a Mamba video encoder and object detection feed a joint scene- and knowledge-graph search whose symbolic prior drives iterative action refinement to predict add dressing and mix salad.
TL;DR: SymAnt searches a joint scene and knowledge graph to build a symbolic prior, then refines it into a long sequence of future actions. It anticipates accurately from only 5–10% of a video, with 3M trainable parameters, and its predictions can be steered at inference by editing graph nodes.

Abstract

As robots and intelligent systems increasingly interact with humans, they must anticipate long sequences of future actions from only brief observations. While purely neural models typically require extensive observations to predict future actions and remain difficult to interpret and adjust, symbolic priors offer a structured alternative. To this end, we introduce SymAnt, a novel neuro-symbolic approach that unifies the expressiveness of neural networks with the structured reasoning of scene and knowledge graphs. Given a short video, SymAnt performs a joint-search over both graphs to construct a symbolic context, enabling unified reasoning over spatial relationships, object attributes, affordances, and associated actions. This symbolic context serves as a prior for an iterative refinement process that progressively generates and improves the sequence of future actions. By grounding predictions in symbolic context, SymAnt anticipates long-horizon actions from short observation windows in a parameter-efficient manner, while enabling users to directly edit the underlying knowledge to adjust predictions without further training. Through our experiments, we demonstrate the effectiveness of SymAnt, particularly when only short observation contexts are available, across five diverse datasets: Breakfast, 50 Salads, EPIC-KITCHENS, EGTEA Gaze+, and Assembly101.

Method

SymAnt architecture: a Mamba video encoder, a temporal scene graph merged with a knowledge graph, joint-graph search producing a symbolic prior, and an iterative action decoder.
SymAnt overview. Objects detected in the short observation form a temporal scene graph (SG), which is merged with a knowledge graph (KG) built from the training split. A joint-graph search produces a symbolic context that initializes an iterative action decoder conditioned on Mamba-encoded video features.
1. Temporal scene graphGrounding DINO detects objects in each observed frame; a relation classifier links them, and the graph is updated over time so stale relations decay.
2. Joint-graph searchThe scene graph is linked to a knowledge graph of objects, affordances, and actions. Propagation and importance networks expand only the relevant nodes into a context embedding.
3. Iterative refinementThe symbolic context, not random noise, initializes a transformer decoder that refines the future action sequence over several steps, conditioned on the video.

Results

With short observations (α = 5–10% of the video), SymAnt is best in every setting on Breakfast and 50 Salads, by 3.2% MoC on average over the strongest baseline (NeSCA) and 5–10% over LLM baselines. The table shows the longest horizon (β = 0.5, predicting the next half of the video).

Method (MoC, β = 0.5)Breakfast α=0.05Breakfast α=0.150 Salads α=0.0550 Salads α=0.1
FUTR5.5811.718.6311.36
GPT-o15.15–11.1812.29
NeSCA5.8811.949.3813.96
SymAnt (ours)8.1512.9214.3318.83

On egocentric EPIC-KITCHENS, SymAnt outperforms the LLM-based AntGPT and PALM and approaches the much larger ICVL and INSIGHT, with 3M trainable parameters.

EPIC-KITCHENS (mAP)ParamsAllFreq.Rare
DiffAnt80M38.155.031.0
AntGPT7.5B40.158.831.9
PALM7.5B40.459.330.3
SymAnt (ours)3M42.559.332.3
ICVL17.5B43.361.633.8
INSIGHT8.5B45.262.436.0

Parameter counts are trainable parameters; SymAnt's count excludes the frozen object detector, ViT node encoder, and I3D backbone. Full results, including EGTEA Gaze+ and Assembly101, are in the paper.

Steering Predictions by Editing the Graph

Because predictions are anchored in an explicit graph, activating or deactivating graph nodes at inference steers the anticipated actions without retraining.

Bar chart: anticipation accuracy for Cut Tomato, Cut Lettuce, and Serve Salad without edits, with the node removed, and with the node activated.
50 Salads, on samples where the target action occurs: accuracy without edits, after deactivating the action's node, and after activating it.
Real-world kitchen examples: a tomato already cut in a bowl, and a scene without a knife, with the relevant parts of the expanded joint graph and the predicted actions.
Real-world kitchen. Left: the tomato is already cut and in a bowl, so no tomato actions are predicted. Right: no knife is present, so no cutting action is predicted. Scene-graph nodes in green, knowledge-graph nodes in blue.
50 Salads examples with ground truth, prediction, and prediction after editing the tomato nodes of the joint graph.
50 Salads. (a) The tomato is occluded and missed by the detector; activating its nodes recovers the cut-tomato actions. (b) Deactivating them removes those actions. (c) A failure case where the visual stream overrides the edit.

BibTeX

@inproceedings{aryan2027symant,
  title     = {SymAnt: Grounding Long-Horizon Action Anticipation with Adaptable Symbolic Priors},
  author    = {Aryan, Fnu and Xie, Yaqi and Campbell, Joseph and Kim, Woojun and Lee, Kwonjoon and
               Nourkhiz Mahjoub, Hossein and Moradi-Pari, Ehsan and Sycara, Katia and Stepputtis, Simon},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}