Official results showed Maithripala Sirisena, a former ally of the incumbent, had won 51.3% of the vote.
Mr Rajapaksa, in office since 2005, said on Twitter he looked forward to a peaceful transition of power.
His supporters credit him with ending the civil war and boosting the economy, but critics say he had become increasingly authoritarian and corrupt.
Mr Sirisena had already received promises of support from Tamil and Muslim leaders before the election.
But the result shows he also picked up a significant portion of the majority Sinhalese vote, most of whom solidly supported Mr Rajapaksa in previous elections.
Mr Sirisena was surrounded by supporters and photographers as he left his offices earlier. He is due to be sworn in as president later.
Maithripala Sirisena was a surprise opponent when he defected to run against his former friend
His supporters were quick to celebrate the unlikely victory
Yogita Limaye, BBC News, Colombo
Less than two months ago, when the president called snap elections, this outcome was unimaginable. There was no big opposition candidate in sight. Then suddenly Mr Sirisena, then the health minister, switched sides, propped up by heavyweight politicians and later supported by many minority groups.
Mr Sirisena has a quiet demeanour, unlike the flamboyant Mr Rajapaksa. His manifesto contained a schedule of when he would deliver on his agenda. The main pledge he made was that he would end what he called the unlimited powers of the president. Given that he has been backed by a variety of political parties and groups, one of his main challenges will be to live up to the promises he has made to all of them.
The incumbent was seeking a third term in office after he changed the constitution to scrap the two-term limit.
But before the results were announced, Mr Rajakpaksa’s press officer said the president "concedes defeat and will ensure a smooth transition of power bowing to the wishes of the people".
Both Mr Rajapaksa and Mr Sirisena are Sinhalese, the majority ethnic group in Sri Lanka.
They were allies until November when Mr Sirisena, the health minister in Mr Rajapaksa’s government, announced his surprise candidacy.
High Tamil turnout
There was a strong turnout in the elections and no reports of major disruptions to voting
Turnout in many areas was above 70%, roughly in line with previous elections, with no reports of major incidents disrupting the voting process.
In Jaffna and Trincomalee, two of the main Tamil strongholds, turnout was higher than previous national elections.
The build-up to Sri Lankan elections is usually blighted by dozens of deaths, but this year just one election-related death was reported.
Mr Rajapaksa was last elected in 2010 when he defeated his former army chief Sarath Fonseka, who was later jailed on charges of implicating the government in war crimes.
But his critics say he became increasingly authoritarian and failed to tackle the legacy of Sri Lanka’s civil war, which left the Tamil areas in the north impoverished and embittered.