Coming on in place of Lucas Vazquez, Rodrygo scored in the 89th minute with what was only Madrid's second shot on target. Inter had peppered its opponents' goal but been denied by a string of fantastic saves from Thibaut Courtois and some wayward finishing.
For Inter, it was their first Champions League game at the San Siro, back in front of its over 37,000 fans.
Inter and Madrid met in last season’s group stage, with the Spanish team winning both encounters — 3-2 at home and 2-0 at San Siro.
Although the Milan club had had won only three of its last 16 Champions League matches, it dominated from the start on Wednesday and almost took an early lead.
Edin Džeko’s angled shot was parried by Courtois and Milan Škriniar’s follow-up was deflected wide. The Inter defender headed the resulting corner narrowly off target.
The Madrid goalkeeper also did well to palm away a header from Lautaro Martínez in the 19th minute.
Meanwhile, Madrid managed to make way into the Inter penalty area seldom and even in the rare occasions it did, the danger was calmly handled by the Nerazzurri defense.
The visitors didn’t have a shot on target in the first half, and the closest they came was when Eder Militao headed a corner wide of the right post.
Moments later Inter had a double chance as first Martínez headed wide of the left upright and then shortly afterwards Marcelo Brozovic's shot went narrowly past the same post.
Inter should have done better nine minutes into the second half. Courtois again did well to palm away Dzeko’s glancing header, Ivan Perišić missed the target from point-blank range and — after several nervy seconds — Madrid eventually managed to clear the danger.
At the other end, Madrid finally had its first shot on target in the 57th but Dani Carvajal’s attempt was at a comfortable height for Inter goalkeeper Samir Handanovič.
Madrid enjoyed more possession in the second half but did not appear a threat.
But as the final moments neared, Ancelotti's spirited side snatched all three points right as Eduardo Camavinga got into the box and flicked the ball on for Rodrygo to volley into the bottom right corner. The goal sparked scenes of wild celebration on the Madrid bench.
With inputs from the Associated Press