The Port Authority of New York and New Jersey announced on Friday a proposal that would raise tolls on all of its Hudson River crossings by $4 for E-ZPass users and $7 for cash tolls this year.

Tolls for cars using E-ZPass would increase from $6 to $10 round-trip for off-peak hours, and from $8 to $12 in peak hours.

Drivers paying cash tolls would have to pay an additional $3 service surcharge.

An additional $2 increase would be implemented in 2014.

Truck tolls would increase by $6 this year, with an additional two-dollar increase in 2014.

The Port Authority says the increase is needed due to its largest need to overhaul its infrastructure in its 90-year history, three consecutive years of zero-percent growth in operating expenses, a more than $2 billion loss in revenue, its spending $11 billion on rebuilding the World Trade Center site and a $6 billion rise in post-September 11th security costs.

Discounts for drivers on the Staten Island Bridges Plan, commuters with energy-efficient vehicles and car-poolers will be preserved until 2014.

Friday afternoon, Governor Andrew Cuomo and New Jersey Governor Chris Christie, who have the power to veto the hikes, released a joint statement saying they would review the plans but had strong concerns.

Their statement read in part, "The Port Authority is facing financial issues but so are families in the states of New York and New Jersey, and the answer cannot always be an indiscriminate and exorbitant increase in the cost to the taxpayer, or in this case, toll payer. As families must carefully and effectively manage their finances at this difficult time, so must government."

Other officials say that the Port Authority desperately needs the hikes so it can take care of its own infrastructure.

The agency said it has had to cut $5 billion in projects since 2008, and that the hikes would fund a $33 billion, 10-year capital plan that would generate 167,000 jobs.

Port Authority officials also claimed that if no action was taken, 240 projects would be at risk, 3,900 construction workers would lose their jobs and $438 million in investments would be lost.

"It's tough medicine, but it's necessary to keep all the bridges, tunnels, airports, ports, transit system and buildings overseas in good working order," said Noah Budnick of Transportation Alternatives.

Partnership for New York City President and CEO Kathryn Wylde agreed, saying in a statement, “No one likes higher tolls, but without them, key infrastructure projects to improve how New Yorkers get around will come to an abrupt stop.”

"For 10 percent, 15 percent, it's understandable, but 50 percent, that's way too much," said a commuter.

"It's ridiculous. I think New York overprices everything. Look at the gas, I'm about to put $100 in my tank," said another.

"The money I live on, Social Security, is barely enough money to cover my bills. So the tolls, what am I going to do? How am I going to get my medication, see my doctors, take care of my health?" said Staten Island resident Ed Cox, whose doctor is in Manhattan. "I have no choice -- either to pay, or to not pay and get sick and die."

PATH riders were also feeling the pinch.

"When I first started using the PATH, it was 35 cents each way. So I've seen the creep-up throughout the years. Better service in some areas, but not too good in others," said a commuter.

“I’m surprised it’s going to be $2.75 because that’ll be more actually than New York City subway which is $2.25. So it adds up after time,” said another PATH rider.

Over time, higher fares may also make some people think twice about coming to play in the city, and commutes to work will become more costly.

"It’s going to be a real problem. Driving a Prius gives us a discount on the tunnel though, so that certainly helps," said a driver.

The Port Authority will hold eight public hearings on the proposal on August 16, three days before its Board of Commissioners considers the final toll/fare plan on August 19.

Sources: Port Authority Wants $150M From 9/11 Memorial Officials

Meanwhile, sources said Friday that the Port Authority is demanding $150 million from the National September 11th Memorial to help cover construction costs at the World Trade Center site.

The memorial, which is scheduled to open on the 10th anniversary of the attacks next month, reportedly said it doesn't have to pay for the work done by the Authority.

Sources say the dispute is over who pays for underground infrastructure work at the site, which includes the memorial, One World Trade Center, a transportation hub and three planned office towers.

Those costs reportedly total nearly $2 billion, of which $150 million was assigned to the memorial as its fair share of overall site work.

Officials for the September 11th memorial declined to comment Friday.