Spanish sources have claimed that Ivanovic is high on the list of targets of Jose Mourinho as the Portuguese coach looks to fine-tune his squad to give Madrid their 10th Champions League trophy.
The 28-year-old, who was a starter in Serbia's 2-0 defeat against Spain on Saturday, stated that he is aware of Los Blancos' interest, but stressed that he is happy with life at Stamford Bridge.
"I know what has been written about my future and that Madrid are following me, but now is not the time to talk about this," Ivanovic told Marca.
"I am focused on the national team now and I am very happy at Chelsea.
"I won the Champions League just one week ago and to think about changing clubs now is complicated. I am very happy in London."
Ivanovic, who missed the Champions League final through suspension, joined Chelsea from Lokomotiv Moscow in 2008 in a reported €12 million (£9m) deal.