Large language models (LLMs) are increasingly used for code generation, but their ability to program specialized cryptographic systems remains poorly understood. Cheon-Kim-Kim-Song (CKKS), a fully homomorphic encryption (FHE) scheme supporting approximate arithmetic over packed real- and complex-valued vectors, is particularly relevant to machine-learning workloads. Prior work has not systematically examined the CKKS-specific challenges of LLM-generated encrypted programs. To address this gap, we introduce FHE-CodeEval, a 46-case benchmark that evaluates three core challenges: packing, nonlinear approximation, and level/scale management, as well as how models coordinate decisions across them. We further introduce communication-aware efficiency, an FHE-specific metric comparing generated programs with compiler baselines using server-side runtime and ciphertext footprint.

We evaluate eight recent proprietary and open-weight LLMs on FHE-CodeEval. Across all 46 cases, the best observed scores reach 51.7% Pass@1 and 69.6% Pass@5 with iterative execution feedback, but only 7.8% Pass@1 and 13.0% Pass@5 in the one-shot setting. For efficiency, even the best-performing model achieves a geometric-mean communication-aware efficiency of only 0.44 over its valid rollouts in the compiler-comparable subset. These results reveal a substantial gap between functional correctness and efficiency in LLM-generated CKKS programs. Code is available at https://github.com/qiuchuy/FHE-CodeEval.

The Model Context Protocol (MCP) enables large language models (LLMs) to access external resources on demand. While commonly assumed to enhance performance, how LLMs actually leverage this capability remains poorly understood. We introduce MCPGAUGE, the first comprehensive evaluation framework for probing LLM–MCP interactions along four key dimensions: proactivity (self-initiated tool use), compliance (adherence to tool-use instructions), effectiveness (task performance post-integration), and overhead (computational cost incurred). MCPGAUGE comprises a 160-prompt suite and 25 datasets spanning knowledge comprehension, general reasoning, and code generation. Our large-scale evaluation, spanning six commercial LLMs, 30 MCP tool suites, and both one- and two-turn interaction settings—comprises around 20,000 API calls and over USD 6,000 in computational cost. This comprehensive study reveals four key findings that challenge prevailing assumptions about the effectiveness of MCP integration. These insights highlight critical limitations in current AI–tool integration and position MCPGAUGE as a principled benchmark for advancing controllable, tool-augmented LLMs.