TII Releases Falcon Perception: A 0.6B-Parameter Early-Fusion Transformer for Open-Vocabulary Grounding and Segmentation from Natural Language Prompts

In the current landscape of computer vision, the standard operating procedure involves a modular ‘Lego-brick’ approach: a pre-trained vision encoder for feature extraction, paired with a separate decoder for task prediction. While effective, this architectural separation complicates scaling and hinders the interaction between language and vision.

Technology Innovation Institute (TII) Research team is challenging this paradigm falcon perceptionA 600M-parameter integrated compact transformer. By processing image patches and text tokens from the first layer in a shared parameter space, the TII research team has developed a rapid fusion Stack that handles perception and task modeling with extreme efficiency.

Architecture: a single stack for each method

The core design of Falcon Perception is built on the hypothesis that a single transformer can simultaneously learn visual representations and perform task-specific generation^.

HYBRID ATTENTION AND GGROPE

Unlike standard language models that use strict causal masking, Falcon Perception employs a hybrid meditation strategy^{. Image tokens pay indirect attention to each other to form global visual context, while text and task tokens pay attention to all preceding tokens (causal masking) to enable autoregressive prediction..}

To maintain 2D spatial relationships in a flattened sequence, the research team uses 3D Rotary Positional Embeddings. It decomposes the head dimension using a sequential component and a spatial component. Golden Gate Rope (Grop). GGROPE allows attention heads to be placed at relative positions with arbitrary angles, making the model robust to rotation and aspect ratio variations.

minimum sequence logic

The basic architectural sequence is as follows chain of perception Format:

[Image] [Text] <coord> <size> <seg> ... <eos>^.

This ensures that the model resolves spatial ambiguity (position and shape) as a conditioning signal before generating the final segmentation mask.^.

Engineering for scale: muons, flexattention, and raster ordering

The TII research team introduced several optimizations to stabilize training and maximize GPU utilization for these heterogeneous sequences.

• Moon Adaptation: The research team reports that employment moon adapter Reduced training loss and improved performance on benchmarks compared to standard AdamW for specific heads (coordinates, shapes, and partitions).
• FlexAttention and Sequence Packing: To process images at native resolution without wasting calculations on padding, the model uses scatter and pack strategy. Valid patches are packed into fixed-length blocks, and flexattention Used to restrict self-focus within the boundaries of each image sample.
• Raster ordering: Falcon Perception predicts when multiple objects are present raster order (Top to bottom, left to right). It was found that it converges faster and produces less coordination loss than random or size-based ordering.

Training Method: Distillation up to 685GT

uses the model multi-distillation For starters, distilling knowledge DINOv3(ViT-H) For local facilities and SigLIP2(So400m) For language-aligned features^{. After initialization, the model goes through a Three-Step Perception Training Pipeline a total of approximately 685 gigatoken (GT): :}

1. List in reference (450 GT): Learning to ‘list’ a view list to create a global context.
2. Working Alignment (225 GT): Transition to independent-query tasks using query masking To ensure that the model makes each query based solely on the image.
3. Long-Reference Finetuning (10 GT): Minor optimization for extreme density, increasing mask limit to 600 per expression.

During these steps, task-specific numbering is used:

<image>expr1<present><coord><size><seg> <eoq>expr2<absent> <eoq> <eos>^.

<present> And <absent> Tokens force the model to make a binary decision on the existence of an object before localization^.

PBench: Profiling capabilities beyond saturated baselines

To measure progress, the TII research team introduced pbenchA benchmark that organizes samples into five levels of semantic complexity to model failure modes.

Main results: Falcon Perception vs SAM 3 (macro-f1)

Falcon Perception significantly outperforms SAM3 in complex semantic tasks, specifically showing a +21.9 point gain On Spatial Understanding (Level 3)^.

FalconOCR: 300M Document Expert

The TII team also pioneered this early-fusion recipe FalconOCRa compact 300M-parameter The model was started from scratch to prioritize fine-grained glyph recognition. FalconOCR is competitive with several large proprietary and modular OCR systems:

• OlmoCR: get 80.3% accuracyMatches or exceeds Gemini 3 Pro (80.2%) and GPT 5.2 (69.8%).
• OmniDocBench: reaches an overall score of 88.64Ahead of GPT 5.2 (86.56) and Mistral OCR 3 (85.20), though it lags behind the top modular pipeline PaddleOCR VL 1.5 (94.37).

key takeaways

• Unified Early-Fusion Architecture: Falcon Perception replaces modular encoder-decoder pipelines with a single dense transformer that processes image patches and text tokens in a shared parameter space from the first layer. It uses a hybrid attention mask – bidirectional for visual tokens and causal for action tokens – to act simultaneously as a vision encoder and an autoregressive decoder.
• Chain-of-Perception Sequence: Sorts the model instance partitions into a structured sequence. $(\lang coordinate\rangle \rightarrow \lang shape\rangle \rightarrow \lang seg\rangle)$Which forces it to resolve spatial position and shape as a conditioning signal before generating a pixel-level mask.
• SPECIFIC HEAD AND GGROPE: To handle dense spatial data, the model uses high-dimensional coordinate mapping and Fourier Feature Encoder for Golden Gate Rope (GGROPE) to enable isotropic 2D spatial attention. The muon optimizer is employed for these particular heads to balance the learning rate against the pre-trained backbone.
• semantic performance benefits: On the new PBench benchmark, which addresses semantic abilities (levels 0-4), the 600M model shows significant gains over SAM3 in complex categories, including a +13.4 point gain in OCR-guided queries and a +21.9 point gain in spatial understanding.
• High Efficiency OCR Extension: The architecture is limited to Falcon OCR, a 300M-parameter model that achieves 80.3% on OLMOCR and 88.64 on OmniDocBench. It matches or exceeds the accuracy of much larger systems like Gemini 3 Pro and GPT 5.2 while maintaining high throughput for large-scale document processing.

check it out paper, model weight, repo And technical details. Also, feel free to follow us Twitter And don’t forget to join us 120k+ ml subreddit and subscribe our newsletter. wait! Are you on Telegram? Now you can also connect with us on Telegram.