Identifying the speaker of each dialogue instance is a key step in manga understanding, supporting applications such as translation, summarization, and voice generation. In zero-shot settings, the task is challenging because each new manga title introduces an unseen cast and provides no speaker annotations, predefined character identities, or per-title model adaptation. Existing methods, including iterative multimodal fusion, still organize speaker reasoning around manga-specific visual evidence such as characters’ appearance and local text-character relationships. However, such evidence is highly variable in manga, where speakers may be off-panel, character appearance changes across poses and styles, and local visual relations can be misleading. We propose Speaker-Seeded Identity Consolidation (SSIC), which reframes zero-shot manga speaker attribution as book-level identity consolidation driven by recurring dialogue consistency rather than local visual-region propagation. SSIC induces local anonymous speaker hypotheses from ordered dialogue context, aligns them into cross-window speaker seeds, constructs a global speaker identity registry, and performs registry-constrained attribution. In the main controlled evaluation on Manga109Dialog, SSIC achieves 74.62% speaker attribution accuracy, improving over iterative multimodal fusion by 7.02 points. The gains are largest on hard cases where local text-character relationships are unreliable, and ablations confirm that cross-window speaker seeds are central to registry construction.