As part of our series on the Landmarks Preservation Commission’s 50th anniversary, we visited a homeowner who is restoring a townhouse in a historic district and learned there's been lots of ups and downs. NY1's Cheryl Wills filed this report.

You may not have heard the name John Dwight–but you’ve surely heard of his invention: Arm & Hammer Baking Soda. In the 19th century Dwight made a fortune off of the product which can be found in nearly every kitchen across the country. In 1890, he built an elaborate five story corner House which was then called Mount Morris Park West in Harlem. The green space is now called Marcus Garvey Park, but when renowned English Poet James Fenton purchased the property in 2010 to prevent a developer from gutting it, it was a rotted shell.

“We had to go around by flashlight – the cellar was under 3 inches of water – homeless guys living in here selling angel dust – graffiti on the walls – it was unbelievable," Fenton says.

Fenton and his partner were eager to start rebuilding the dilapidated townhouse, but there was a catch: It’s landmarked and part of the Mount Morris Park Historic District – which means when it comes to the exterior – you can look but don’t touch.

"With the exterior you can change practically nothing but the big thing is the windows, you really enter into a negotiation in order to say what can replace the existing windows?” Fenton says.

This mansion has 55 windows dating back to the 19th century and during the first winter that Fenton lived here. He says the cold was unbearable so he had to get approval of the Landmarks Commission to change the crumbling windows. So far, he’s replaced 45 of them.

“I’ve lost count of the number of people who are specialists in dealing with this particular problem,” Fenton says.

But the head of the Landmarks Preservation Commission says the agency tries to work with homeowners so they are not overwhelmed by the process.

"What we’ve done is we created guidelines and rules which inform homeowners as to how to make changes to their properties and do repair and renovation in a way that can take place fairly and efficiently at the agency. In fact, we have various processes and turn arounds that sometimes are two days to thirty days," says Landmarks Preservation Commission Chairwoman Meerackshi Sririvasan.

The original door to the property has been through so many changes since it was built in the late 19th century. During the 1930s it became an art school for the WPA. During the 1960s, it became a black synagogue. Now, a painstaking restoration is underway to return it to its original glory.

"I think there aren’t all that many people who would do what we’ve done,” Fenton says.

What Fenton and his partner are doing is nothing short of heroic. 

The owners have opened up staircases that had been sealed shut for decades. They have hired specialists to chip away at layers of paint, only to find mahogany underneath. The costs, as you might imagine, are astronomical—but Fenton says its worth every penny.

"You could immediately see that it had great style and the most striking thing to me was it had four oval rooms. A house with one oval room would be very exciting but four oval rooms is quite extraordinary," Fenton says.

The extraordinary views from the balcony overlook upper Manhattan. The owners say it will take at least three more years before renovations are complete. So in 2018, this historic corner house will once again become the crown jewel of this landmarked district.