Fusing charted times series with clinical notes has been found useful for predictive analytics. However, the incompleteness of clinical notes can undermine prediction quality. We thus propose a novel conditional diffusion-based model called CODE to recover the missing semantics of clinical notes, which integrates EHR time series into diffusion. CODE learns to align a diffusion process with unknown levels of note incompleteness via contrastive learning, by which missing semantics is recovered from the denoising process. We evaluate the effectiveness of CODE by conducting experiments on real-world EHR data for three prediction tasks. Empirical results show that CODE with the capability to recover the missing semantics can outperform state-of-the-art baselines.