The 5ft 2in singer, whose debut album Rockferry made her an overnight sensation in 2008, found her sudden success difficult to deal with.
But falling in love with Ospreys and Wales scrum half Mike Phillips - one of the standout players in last year's Lions tour of South Africa - has turned her life around
Now Duffy, 26, is back with her second album, called Endlessly, and a man to protect her from the pressures of fame.
Duffy says: "Becoming really famous can be intense but Mike is 6ft 3in and over 16 stone - I need some solid arms around me when I feel vulnerable."
We meet at a London hotel and it is clear straight away just how happy Duffy is.
When she speaks about her man - labelled the best-looking scrum half in the game - she colours up a bit.
Duffy confesses: "Mike has had a huge influence on me - look, I'm blushing just at the mention of his name, I will have to fan my face with my hand to calm down.
"Last week I was in America and I was asked out by a radio DJ. It's the first time I've actually admitted that, yes, I am in love.
"The man said to me, 'I am falling for you. Are you available? Are you a single lady?' And for the first time ever I went, 'No, I'm NOT single.'
"It was an out-loud acknowledgement that I am officially off the market."
Duffy started dating 28-year-old Mike in September last year. He asked her out by text after getting her number from his agent.
They were soon compared to two other stars, singer Charlotte Church and rugby ace Gavin Henson - who have since split up - and dubbed "the new Welsh Posh and Becks".
Duffy says: "That name makes us laugh as we just enjoy being with each other and ignore the fame side of it all - we just have lots of fun together."
Rockferry was a top three hit in 19 countries and has so far sold more than 6.5 million copies, making Duffy a multi-millionairess.
In March 2008 she became one of only nine solo female singers in history to hold the No1 spot in the UK singles and album charts simultaneously, with Mercy and Rockferry.
Born Aimee Ann Duffy, the soul singer spent more than a decade trying to make it in music, but she says fame made her miserable and lonely.
"I was living out of a suitcase and always flying here and there," she recalls. "It was everything I'd ever dreamed of but there were times when I found it hard.
"I love my music and performing but some of the other demands terrified me. I am just a normal girl from Wales and always will be."
One time in particular, she found herself feeling extremely homesick in America.
She says: "I locked myself in my room and cried and I'm not usually that kind of girl. I am quite tough and can cope with almost anything but that time was horrible. I just wanted my mum with me. I remember feeling as though things were going to fall apart.
"I don't know if it was borderline breakdown - but it was a hard time."
Duffy adds: "I'm never going to complain about what I have achieved and I am eternally grateful for my success but I think the workload at times was too much for one human being.
"But you learn to switch off and the more famous you get, you develop a coping mechanism - your boundaries close in to protect yourself. That's why programmes such as The X Factor are great for singers as long as they have a game plan and know what they're letting themselves in for.
Overnight success might be everything you've dreamed of but there are times when it will hit you hard.
"Anyone on a show like X Factor has to be prepared for what happens next."
It was only when she managed to take a break following a relentless promotional campaign for Rockferry that Duffy began to enjoy life again. She met Mike and now, in the run-up to the release of Endlessly, she has finally learned to relax and take all the pressure in her stride.
And when things do get tough, she has Mike to turn to.
Duffy says she wrote the title track on Endlessly at a time when she longed to have someone of her own. She says: "I have learned a lot about myself through my new songs - they're like therapy and I realise I was lonely because I was needing a man, someone to protect me and call my own.
"I wrote Endlessly when I was dreaming of meeting someone. I genuinely wanted someone to put their arms around me.
"I knew I was missing something in my life - then I met Mike and knew he was the one from our first date."
When it comes to marriage, Duffy takes a traditional approach and will wait to be asked. She says: "I don't like looking too far ahead as I'm a girl who lives for now but I am very old-fashioned and I wouldn't want to presume.
I think the guy has to be the one to ask. I have certain views about those things, and so does Mike, who is also old-fashioned.
"He gets annoyed if anyone hits on me and just wants to protect me."
Duffy recently turned down a role in a major Hollywood movie and instead opted to play a sexy seductress in low-budget Welsh language film Patagonia. She says: "I'm not saying who the director was for the first film but he was a big name.
"Yet I made the right decision - Patagonia was great for me, as was its director Marc Evans.
"I just wanted to have fun and put myself in the arms of someone else and not have to worry.
"It's so different to making music, which gives me sleepless nights. With the film I could let the director worry about everything. I trusted him totally.
"But I'm not sure whether I'll do another role. My mind is just focused on my music. I'm so excited to be back."
See SFTW tomorrow for Duffy on new album Endlessly, out Nov 29.