Reinforcement Learning from Human Feedback (RLHF) often suffers from sparse reward signals, making credit assignment to tokens challenging. We propose Shapley Credit Assignment Rewards (SCAR), a method that uses Shapley values from cooperative game theory to distribute sequence-level rewards across tokens or text spans according to their marginal contributions. SCAR provides dense rewards without requiring auxiliary critique models or fine-grained human annotations, while offering a principled game-theoretic foundation for credit assignment. We show theoretically that SCAR preserves the original optimal policy, and empirically demonstrate across sentiment control, summarization, and instruction tuning tasks that it converges faster and achieves higher final rewards than standard RLHF and attention-based dense reward baselines. To improve scalability on long sequences, we further introduce an Owen value–inspired adaptive text segmentation strategy. Our findings suggest that SCAR provides a more effective and theoretically sound method for credit assignment in RLHF, leading to more efficient alignment of LLMs.