updated 04/11/2011 AT 7:10 AM ET
•originally published 04/11/2011 AT 1:15 PM ET
“I think Holly [Madison] has expressed publicly that it would be a little strange if she was there and I respect that,” the Playboy founder told PEOPLE at his 85th birthday party at Las Vegas’s Palms Casino Resort.
As portrayed on E!’s Girls Next Door, Madison hoped to marry Hefner but eventually moved on and relocated to Las Vegas. Hefner, meanwhile, moved on to 24-year-old Crystal Harris, whom he will wed on June 18. The feud between Madison and Harris has been well documented.