Recognizing a formal language does not imply the ability to construct the computation that decides it. We operationalize this distinction as an executable benchmark for language-model reasoning: given a formal-language specification, an LLM must synthesize a deterministic or nondeterministic Turing machine that decides the language. We introduce TMEval, a benchmark of 253 language specifications across 7 structural categories spanning context-free and non-context-free phenomena. Each language includes 600 oracle-labeled membership tests, and every output is evaluated as an executable artifact rather than a textual response. Our pipeline enforces a strict output contract, validates machine construction, executes each buildable machine, and assigns one of four outcomes: static_fail, build_fail, semantic_mismatch, or exact_pass. Across eight LLMs, 2,024 model–language candidates, and up to 1,214,400 behavioral evaluations, only 11.12% exactly pass all tests. More than half fail before execution, over one-third implement the wrong language, and 75 languages remain unsolved by all systems. Failure depends less on the context-free/non-context-free boundary than on the operational structure required to realize the specification. TMEval reveals a substantial gap between formal-language recognition and machine-checked symbolic construction. TMEval is available at https://turing-machineeval.github.io/.

Artificial intelligence has long been evaluated by its ability to perform computation and structured reasoning. While recent large language models (LLMs) demonstrate strong capabilities in language generation, code synthesis, and structured problem solving, it remains unclear whether they can construct formal computational mechanisms from high-level specifications. In this work, we investigate whether modern LLMs can synthesize pushdown automata (PDAs) from language descriptions expressed in natural language or mathematical form. Because PDA construction requires explicit reasoning about states, transitions, and stack-based memory, it provides a controlled test of structured symbolic reasoning. We introduce the task of pushdown automaton synthesis and develop an automated evaluation framework that verifies generated PDAs through structural buildability checks and execution-based validation. Using a benchmark of context-free languages with labeled test cases, we evaluate contemporary LLMs and analyze their ability to construct and execute formal computational machines. Our results show that current state-of-the-art LLMs struggle to reliably construct executable PDAs, indicating that synthesizing computational mechanisms requiring coordinated reasoning over state transitions and stack-based memory manipulation remains challenging for present-day models. All code and data are available at: https://github.com/Dip-to/Pushdown-Automaton-Synthesis-Dataset.git

Large language models (LLMs) are no longer just language generators—they are increasingly used to simulate human behavior, perspectives, and demographic variation across social domains, from public opinion surveys to experimental research. Amid this shift, the use of LLMs to simulate historical narratives has emerged as a timely frontier. It is crucial to scrutinize the asymmetries these models embed when framing, interpreting, and retelling the past. Building on prior work that defines caricature as the combination of individuation and exaggeration, we analyze LLM-generated responses across 197 historically significant events—each featuring a directly and an indirectly affected persona. We find that LLMs reliably distinguish persona-based responses from neutral baselines, and that directly affected personas consistently exhibit higher exaggeration—amplifying identity-specific portrayals. Beyond lexical patterns, personas often frame the same event in conflicting ways—especially in military, political, and morally charged contexts. Grammatical analysis further reveals that direct personas adopt more passive constructions in institutional contexts, but shift to active framing when emotional immediacy is foregrounded. Our findings show how subtle asymmetries in tone, stance, and emphasis—not overt toxicity—can quietly, yet systematically, distort how history is told and remembered.

Despite Bengali being the sixth most spoken language in the world, handwritten text recognition (HTR) systems for Bengali remain severely underdeveloped. The complexity of Bengali script—featuring conjuncts, diacritics, and highly variable handwriting styles—combined with a scarcity of annotated datasets makes this task particularly challenging. We present GraDeT-HTR, a resource-efficient Bengali handwritten text recognition system based on a Grapheme-aware Decoder-only Transformer architecture. To address the unique challenges of Bengali script, we augment the performance of a decoder-only transformer by integrating a grapheme-based tokenizer and demonstrate that it significantly improves recognition accuracy compared to conventional subword tokenizers. Our model is pretrained on large-scale synthetic data and fine-tuned on real human-annotated samples, achieving state-of-the-art performance on multiple benchmark datasets.