Large language models have become the dominant workload in modern GPU clusters, driving unprecedented demand for compute during both training and inference. Yet GPU clusters typically operate at only 30–50% utilization, largely because schedulers rely on user-provided runtime estimates that are notoriously inaccurate. Running thousands of LLM inference requests through shared datacenters further compounds this underutilization. Accurate runtime prediction is a key enabler of LLM efficiency: it supports shortest-job-first scheduling, backfilling, dynamic resource allocation, and cost-aware instance selection, while providing NLP practitioners with reliable completion estimates for training runs and serving latency guarantees. However, LLM workloads pose unique prediction challenges: attention complexity scales quadratically with sequence length, autoregressive decoding produces variable-length outputs, and 3D parallelism creates intricate performance interactions. This survey systematically reviews runtime prediction methods with emphasis on LLM training and inference workloads, presenting a feature taxonomy, analyzing design choices of existing prediction methods, surveying public datasets and highlighting their limitations. We identify applications critical to LLM efficiency and key open challenges including generalization to rapidly evolving model architectures and hardwares.