Construction workers rallied in Lower Manhattan to get rebuilding underway at the World Trade Center, while the Port Authority and the site lease holder file a claim in court to get insurance companies to pay up.

Carpenters, plumbers, painters, iron workers and other union members rallied across from the site this afternoon. They're calling on six insurance companies to stop withholding hundreds of millions of dollars in claims, money that if paid could pave the way for building at the site.

"The question everywhere I go is, ÎWhat's going on with Ground Zero?’ It's gotten to the point where it's embarrassing that there isn't any progress," said carpenter Tom O'Connor.

"Build this now. That's all I've got to say: build it now," said iron worker Billy Foster.

WTC site leaseholder Larry Silverstein and the Port Authority are trying to put some legal muscle into that effort, filing a lawsuit Monday against the insurance companies. The suit alleges the insurers have been balking at continuing payouts since Silverstein and the PA agreed to divide rebuilding efforts back in April.

The developer and the Port Authority say the insurers are using that deal as a technicality to withhold payment. In a statement, Silverstein accused the insurers of foot dragging and said: “We expect a quick resolution that will force these insurers to finally pay what they owe."

Union workers at Monday’s rally say they'd obviously like to get back to work for financial reasons, adding that after nearly five years of looking at the site in its current empty state, they'd also like to get back to work for emotional reasons.

“Everybody here understands what happened on 9/11 and now is the time for us to recover, now is the time for us to heal our wounds,” said Anthony Williamson of the Laborers International Union. “When those buildings are up, then we'll feel like we've done something. This city needs to get back on its feet, and the only time this city is going to get back on its feet is if the buildings are being built. They already destroyed us. We need it to be built now."

NY1 reached out to all of the insurance companies that are being sued, and all of them refused comment because they say the matter involves ongoing litigation.

News of the lawsuit comes just days after it was revealed that construction at the World Trade Center site isn't just slow, it's actually going backwards.

The Freedom Tower cornerstone was removed from the site Friday, two years after it was installed because of new design plans.