Developers of a planned Islamic center and mosque near the World Trade Center site said Friday they never planned to meet with Governor David Paterson to talk about relocating.

The governor offered to help find them an alternate location from Park Place in Downtown Manhattan, but as of late Friday no meeting was officially scheduled.

Leaders of the center, like co-founder Daisy Khan, said they are not backing down. Khan told the Associated Press on Friday that they are consulting with Muslim leaders across the United States as the plan moves forward.

Khan said there is no pressure to change locations from political leaders who have supported the project so far, and that she realized the controversy is affecting Muslims nationwide.

As Muslims gathered at the current Park Place building for Friday prayers, a group gathered outside to show support for the proposal, saying the cultural center will promote moderate Islam.

However, protesters will rally at the Park Place building against the mosque proposal on Sunday morning. Construction workers who have pledged not to work on the site will join firefighters, veterans, some families of September 11th victims, first responders and local residents.

Opponents argue the center is too close to the World Trade Center site and is insensitive to the families of the victims of the terror attacks.

Paterson, however, said the debate has gotten so out of control that he would continue his push to get together with the developers.

"They have a clear right to put the mosque wherever they want. That's why my appeal comes to them to have a dialogue about how we might be able to lessen the strain on the people who live in that area," said the governor.

The governor floated the idea of a land swap, but as of Friday no mosque organizers had responded positively to the idea.

"Land transfers, swaps, these things happen all the time. As a matter of fact, there is an attempt to relocate a Greek Orthodox church that was destroyed on 9/11," said Paterson.

Legal experts told NY1 that land transfers are only common in the private context, such as when a developer seeks air rights owned by a religious organization.

"It's hard to see what justification the government could offer for engaging in this land swap in this case, other than to try to rid the Ground Zero area of a mosque. And that's a problematic public purpose," said Cardozo Law Professor Stewart Sterk.

Also on Friday, Muslims in Jackson Heights, Queens who spoke with NY1 said that they would prefer the cultural center and mosque to stay in its original proposed location, but some said a compromise should be reached.

"Yeah, they can make some kind of compromise, but I don't really see a problem there, you know?" said one Queens Muslim. "But if I was in the position of the victims, then I think I would think differently."

"I feel like by moving it away, you're sending a very wrong message to people of other faiths also," said another. "That if we don't like you being there, move."

Meanwhile, the imam behind the center said Friday that extremism is a security threat to both the West and the Muslim world.

Feisal Abdul Rauf made the comments to the Associated Press in Bahrain, after leading prayers at a neighborhood mosque as he begins a two week tour of the Middle East.

He said he wants to draw attention to the common challenges to battle radical religious beliefs.

Rauf refused to discuss the firestorm over plans for the Islamic center in Lower Manhattan.

The trip, which includes stops in Qatar and the United Arab Emirates, is being funded by the U.S. State Department as part of a program to promote interfaith tolerance.