A group of street artists fighting to preserve the warehouse complex known as the "5 Pointz" suffered a defeat in court Tuesday.
A federal judge in Brooklyn lifted a temporary restraining order that was preventing the building's owner from tearing it down.
The structure in Long Island City has been called the "graffiti mecca of the world."
The owner wants to replace it with a residential development.
The plan includes two rental towers, a park and 12,000 square feet of artist's space and galleries.
The Court Square Diner has been in the area for 23 years and has seen an increase in business from tourists coming to 5 Pointz.
Nick Kanelos, the diner's owner, is optimistic that new development on the site won't have a negative impact.
"Business-wise for me, either way, it's going to be good because more residential," Kanelos said. "It could also be more commercial, so you never know what could happen."
No date has been set to begin demolition.