updated 10/30/2013 AT 2:00 PM ET
•originally published 10/30/2013 AT 3:00 PM ET
Now the singer, 24, wishes to set the record straight, telling PEOPLE: “It’s ridiculous!”
“We’re not saying anything for two or three weeks and I’m a drug addict!”
What upset him even more is that his girlfriend, Blanda Eggenschwiler, was also dragged into the rumors.
“It’s obviously not true. I’ve never touched heroin in my life. Neither has my girlfriend,” he said. “It’s hilarious to me, but also frustrating as well. It pissed me off when it’s about my girl and even about myself.”
At one point, he said, people even approached him personally about it.
“I was walking down the street and people are asking me if they can give me a hug and how rehab is,” says Joe. “I’m like, ‘You tell me because I’ve never been.’ ”
So why didn’t he speak out right away to correct the impression?
After Nick Jonas, 21, first confessed to older brothers Joe and Kevin that he had concerns about the band – they then unanimously decided to call it quits – Joe says: “We all had to deal with it in our own way. I needed time to clear my head.”