He made the announcement at a press conference in Washington DC, where he had earlier met US Secretary of State Hilary Clinton. There has been ongoing tension in the Labor Party in recent weeks over the leadership.
Ms Gillard ousted Mr Rudd as prime minister in June 2010.
Amid the leadership wrangling she had faced calls from backbenchers to sack Mr Rudd - who said her failure to defend him from these attacks had helped him come to a decision.
"The simple truth is that I cannot continue to serve as foreign minister if I don't have Prime Minister Gillard's support," he said late Tuesday night at the press conference in the US, which was broadcast live in Australia.
I therefore believe the only honourable course of action is for me to resign."
He did not say if he would be challenging Ms Gillard for the leadership, but confirmed that he would leave the US the next day, arriving home in Brisbane on Friday morning.
He said he would not be involved in a "stealth attack on a sitting prime minister".
"We all know that what happened then was wrong and it must never happen again," he said, referring to how he was ousted.
Mr Rudd said he would consult his family and community before making an announcement about his future. He would do so before parliament returned next week, he said.
'A soap opera'
In recent days speculation had been mounting that Mr Rudd's supporters were planning a leadership challenge.
The ruling Labor Party government is struggling to retain voter support and Ms Gillard has been slipping in popularity polls. She leads a minority government which relies on independents for its wafer-thin majority.
She was grilled by Australian media last week over allegations that her staff worked on a "victory speech" two weeks before she ousted Mr Rudd.
She refused to say for certain if she knew the speech was being prepared in advance, but stressed that she did not commission it.
After the interview, polls published in The Australian newspaper showed that she had lost ground to opposition leader Tony Abbott. While support for him rose three points to 40%, hers dropped to 37%.
At the press conference, Mr Rudd appeared to cast doubt on Ms Gillard's ability to defeat opposition leader Tony Abbott in the next election, due in 2013, asking colleagues who was best placed to lead the party to victory.
"Mr Abbott, I believe, does not have the temperament or the experience to ever be elected to hold the high office of prime minister of Australia,'' he said.
"But at present, and for a long time now, he has been on track just to do that."
Mr Abbott, for his part, said the ''instability'' was ''damaging Australia'' and the government showing itself ''unworthy to continue in office''.
- Nicolas Sarkozy says France has too many foreigners
- EU summit: All but two leaders sign fiscal treaty
- Turkey: Ten wounded in Istanbul police bus blast
- Strauss-Kahn freed by police in prostitution inquiry
- Greece bailout: Large protests expected against cuts
- Strauss-Kahn 'a suspect' in prostitution ring inquiry
- Pay-per-Neet' scheme aims to help teenagers find work
- Eurozone ministers back 130bn-euro bailout for Greece
- Greek bailout 'elements in place', says France's Baroin
- Robbery at Ancient Olympia museum
- Rupert Murdoch in UK to tackle Sun arrests fallout
- German President Wulff quits in home loan scandal
- Greece bailout: Eurozone calls for tighter oversight
- Amanda Knox and Raffaele Sollecito murder acquittals challenged
- Eurozone crisis: Greece 'can't take any more cuts'
- US firm Monsanto 'guilty' in France poisoning case
- Wu Yin's case has raised debate about local loan laws
- New doubt over Greek bailout plan
- Greece bailout: Coalition 'to finalise' austerity deal
- Romania PM Emil Boc resigns after austerity protests
- Cold weather death toll passes 100 in Ukraine
- Deaths in Egypt's Suez after Port Said football unrest
- West attempts to persuade Russia over Syria resolution
- Canadian firm Osisko halts Argentina mining project
- Europe freeze: Heavy snow across the continent