Large language models (LLMs) are strong sequence predictors, but standard context accumulation offers no explicit mechanism for revising the memory carried between predictions. We propose LLM-as-RNN, an inference-time framework for online learning with frozen LLMs through a bounded natural-language memory state. At each timestep, the model predicts from the current observation and its memory, receives feedback, and rewrites the state to correct errors and retain task-relevant patterns without updating model parameters. Implemented as a structured system prompt, this state makes the adaptation process inspectable through human-readable memory traces. We evaluate LLM-as-RNN on sequential prediction benchmarks in healthcare, meteorology, and finance across Llama, Gemma, and GPT-oss model families. Comparisons with static-context and feedback-aligned baselines examine predictive performance, feedback sources, memory budgets, and inference cost. Our results demonstrate that feedback-driven memory rewriting supports online error correction and sequential prediction under a fixed memory budget, with performance varying across tasks and backbones.