The pair met for around 15 minutes at the Vatican on Saturday and agreed to meet again later the same day for further talks, a spokesman for Zelenskiy said.
Photos released by Zelenskiy's office showed the two men sitting opposite each other in the middle of a marbled hall, with none of their aides in the vicinity.
Zelenskiy described it as a very symbolic meeting and one that "could be historic" if they are able to reach results together.
He also thanked Trump for the discussion.
The meeting, their first since an angry encounter in the Oval Office in Washington in February, comes at a critical time in negotiations aimed at bringing an end to fighting between Ukraine and Russia.
"President Trump and President Zelenskiy met privately today and had a very productive discussion. More details about the meeting will follow," White House communications director Steven Cheung said.
Trump, who has been pressing both sides to agree a ceasefire, said on Friday that there had been productive talks between his envoy and the Russian leadership, and called for a high-level meeting between Kyiv and Moscow to close a deal.
Trump had previously warned his administration would walk away from its efforts to achieve a peace if the two sides do not agree a deal soon.
After a round of shuttle diplomacy this week, differences have emerged between the position of the Trump White House on peace talks and the stance of Ukraine and its European allies, according to documents from the talks obtained by Reuters.
Washington is proposing a legal recognition that Crimea, the Ukrainian peninsula annexed by Moscow in 2014, is Russian territory, something that Kyiv and its allies in Europe say is a red line they will not cross.
Another photo released by Zelenskiy's office showed Trump and Zelenskiy standing in a huddle, along with British Prime Minister Keir Starmer, and French President Emmanuel Macron.
Asked about a photograph shared by the Ukrainian presidential press office that depicted the four leaders standing together in Rome, the French presidency said: "I confirm the meeting, which was positive."
There are also differences on how quickly sanctions on Russia would be lifted if a peace deal was signed, what kind of security guarantees Ukraine would have, and how Ukraine would be financially compensated.
At the Oval Office meeting in February, as Trump and Vice President JD Vance accused the Ukrainian leader of being ungrateful for US support, a reporter who was present from a conservative US news network accused Zelenskiy of disrespecting the occasion by not wearing a suit.
Zelenskiy, since the start of Russia's full-scale invasion in 2022, has eschewed suits in favour of military-style attire, saying it is his way of showing solidarity with his countryman fighting to defend Ukraine.
In Rome on Saturday, Zelenskiy again decided against a suit, and instead wore a dark shirt, buttoned up to the neck with no tie, and wore a dark military-style jacket over the top of that.Â
