RAG-based question-answering (QA) in specialist domains faces a cold-start problem: no evaluative benchmark and no labeled data for post-training. We present DoRA (Domain-oriented RAG Assessment), a benchmark construction and evaluation framework that needs only a small set of specialist-domain documents, integrating and expert-validating otherwise established components for this cold-start setting. DoRA generates synthetic QA training and evaluation sets with auditable evidence across five domain-specific intents. To mitigate same-pipeline circularity, its training and test splits use different LLM families (Claude Sonnet for training; GPT-4o for test) drawn from disjoint seed-document corpora. Instantiated on 40 English-language defense documents, DoRA yields approximately 6.6K curated instances. Against 7 LLM baselines over a 1,259-sample benchmark, a LoRA-adapted Llama-3.1-8B trained on the synthetic split improves all 6 coverage and faithfulness metrics—each gain significant under a paired bootstrap test—and reduces hallucination by more than half under the default GTE retrieval setting, with gains persisting across alternative retrievers, prompting-based baselines, and matched answer-length budgets. Defense experts validate the synthetic QA, calibrate the LLM judge, and supply a fully human-written test set. We position DoRA as a practical framework for specialist-domain RAG under domain shift, with defense as a high-stakes case study.