This project develops a unified framework for reliable multimodal document understanding in low-resource and multilingual settings It consists of four complementary parts that progressively test different capabilities: understanding the structure and content of individual forms, splitting and ordering multi-document packets, verifying whether document bundles satisfy application requirements, and following explicit field-level instructions and constraints. Across all four parts, the goal is to isolate failures in visual recognition, layout understanding, cross-page reasoning, and instruction following, and to evaluate how reliably multimodal models can support realistic administrative workflows.
BaFCo focuses on fine-grained understanding of complex Bangla government forms. It evaluates whether multimodal models can identify document elements, understand key-value relationships, and extract important information from dense, multi-page forms containing both printed and handwritten content. A paper on this work was published at ECCV 2026.
Khondo studies document packets in which multiple scanned forms are merged together. It evaluates whether models can separate the packet into its constituent forms, restore the correct page order, and identify each form type, with page ordering emerging as a particularly difficult challenge.
Khondo v2 moves from document organization to packet verification. It asks whether a model can determine if an application bundle contains all required documents and excludes incorrect ones, requiring reasoning over the full packet against application-specific requirements.
Doc-IFEval evaluates whether vision-language models can complete Bangla and English forms while following explicit constraints. It tests formatting, consistency, and dependencies across fields and pages, while separating failures in document recognition from failures in instruction following.
Team:
BaFCO & Khondo: Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan, Ezharuddin Jubaer, Sumaiya Karim Katha, Armun Alam, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman
DoC-IFEval: Moshiur Rahman, Md Mubtasim Ahasan, Sami Ibn Rashid, Syed Mohaimin Ul Hoque, AKM Mahbubur Rahman, Amin Ahasan Ali
