Douglas Durst loves to tell the story of his grandfather, an immigrant who started with a pushcart, developed a children's clothing business and then built a real estate company. Now almost 100 years later, it's still a family owned company. NY1's Budd Mishkin filed the following "One On 1" report.

Douglas Durst is the chairman of the Durst organization, one of the most powerful companies in Manhattan real estate for almost a century. Durst's grandfather Joseph started the company. His father Seymour then ran it. After he died in 1995, Douglas took over the reins. He is known for developing eco friendly buildings, like the Conde Nast Building at 4 Times Square and the Bank of America Building at 1 Bryant Park. But in the past year, Durst has given up the title of co-president.

"I told people I was retiring. That was only half-true. I like to say that so I can say I'm retiring to spend less time with my family," says Durst.

Durst displays a self deprecating sense of humor. But the overall approach is low key. Unlike Donald Trump, you will almost never find him on Page Six. And though he understands that appearances at functions -- like a recent benefit for the Martha Stewart Center for Living -- are part of the job, he's known for not sticking around too long.

But don't confuse reticence or shyness for a lack of opinions and the resolve to state those opinions. His father opposed the original World Trade Center, and in 2007 Durst came out against the plans for the Freedom Tower, now known as 1 World Trade Center.

But this past summer, the Port Authority awarded the Durst Organization the right to take over leasing, management and operation of 1 World Trade Center, but why the change?

"I was opposed to the timing, and now they're pretty much being constructed to the timing that I thought was appropriate. The building will not open for tenants until 2013 or 14," explains Durst.

There is an aspect of history repeating itself here. Durst's Conde Nast Building was instrumental in the redevelopment of Times Square, but the company had initial objections to that project, too.

"The city was subsidizing the construction of office buildings, and we weren't receiving the subsidies -- big, big objection," recalls Durst. "We think that if there's subsidies being handed out, we should get them, and not other developers. I'm sure other developers feel the same way about us."

Not surprisingly, Durst heard a lot of real estate talk when he was young. What he didn't hear was a maternal voice. When he was only five years old, Durst's mother committed suicide.

"I always wonder what would've happened, had she lived. And so I'm sure it had a huge affect on me and all my siblings," says Durst. "Having raised three children of my own, I'm amazed my father was able to raise four children by himself. I just find that an astounding accomplishment."

Durst eventually went off to college at Berkeley in the mid 1960s. At the time, he had no intention of joining the family business. He wanted to work in the foreign service. His part-time job in college also involved serving.

"I worked at Peppy's Pizza Parlor, which was interesting 'cause all the Hells Angels hung out there. It was a little intimidating," recalls Durst.

He eventually returned to New York, joined the family business and looked to develop a building. He decided on a building on the Upper West Side after rejecting SoHo.

"I decided, nobody would ever wanna live down there. It's all trucks and commercial and dirty streets," says Durst. "So we went up to West 78th Street and Columbus which, we were about five years too early, and would have been right on time in SoHo. I always use that as an example of my real estate prowess."

That failed venture spurred Durst to move to Newfoundland with his wife and young family. Durst says they lived there for almost a year, until the wood fired hot water heater blew up.

"I was severely injured; nobody else was, luckily. So my thoughts of escaping New York -- I decided I was safer in New York, so I've stayed in New York ever since," says Durst.

The prospect of an accident is a reality that everyone in real estate has to consider and occasionally confront. The construction of the Bank of America Building was marred by some incidents of flying glass. And in 1998, a scaffolding accident at 4 Times Square resulted in one death and several injuries.

"It's painful, but it was a very serious problem with scaffolders still up in the air, and we had to deal with taking it down," says Durst.

Another painful issue for Durst is the subject of his brother Robert, who was twice suspected of murder but not charged, then acquitted of murder charges in a third case in 2004, and finally jailed on a parole violation in 2008.

"I have no direct contact, nor does anybody in my family have any contact, with my brother. And, unfortunately, there is no haven from what's going on with my brother, as it's in the papers quite frequently. But it's not something that we discuss and not something that we have any involvement in," says Durst. "There is no way to describe, to describe what my family and I, my entire family, extended family, go through because of this."

Durst is married with three grown children. Away from real estate he's a competitive bridge player. And though the family business was not his initial choice long ago, he is proud of the legacy and his role in it, standing on the shoulders of his father and grandfather.

"If you're in Liberty State Park and you look north and you see -- because that's due south of where we are -- you see this building, you see 4 Times Square and the Epic, and it's changed the skyline," says Durst. "It's quite a thrill to see that."