The actor's Rottweiler mix Zora (named after author Zora Neale Hurston) had been Jackson's constant companion since he adopted her from the ASPCA in 2003.
"She was everything to us," the 30 Rock star, 37, tells PEOPLE. "She was New York to us. She was in every dressing room I ever had, she was kind of the supermodel of the dog park, she was really beautiful. She was everything."
But in March, Zora suddenly became ill, her body "riddled with cancer of all kinds," so Jackson and his husband, Monte Lapka, prepared a wake of sorts to celebrate Zora while she was still living.
"We brought all of her friends to the house, and all of her people she loved," he says. "It was really beautiful."
Then, with Zora's heart beating over his arm, Jackson parted with his beloved pet and put her to sleep. "It was the hardest thing we've ever gone through," he says. "We almost couldn't stand up for two days, it was so devastating."
While Jackson and Lapka figured they wouldn't be ready for another dog right away, they didn't expect to get an email from pal Bernadette Peters – "no words, just a photo" of a dog in need of a home.
They might have been reluctant, but the couple adopted the dog, named Brillo, a month ago and have been working to get him acclimated to his new life.
"He has issues," the "Before You" singer says, "but who doesn't?"