(Reuters) -- Legal wrangling over the proposed $8.5 billion settlement of some of Bank of America Corp's mortgage-backed securities liability could drag through the courts for years, a top appeals court judge said during arguments in the case.

The 2nd U.S. Court of Appeals in New York is weighing whether last June's Bank of America accord, which has been closely watched by other banks and bondholders, is a matter for federal court review or belongs in state court where it was first filed.

The settlement was intended to help Bank of America address much of its remaining legal liability from its ill-fated 2008 purchase of mortgage lender Countrywide Financial Corp. But some investors have challenged the deal, saying the payout is too low and want it to get more scrutiny from a federal judge.

Regardless of what the appeals court decides, the case "could come back to us," Chief Judge Dennis Jacobs said on Wednesday.

Jacobs and two other appeals judges did not indicate how they would rule.

Robert Madden, a lawyer for about two dozen institutional investors with tens billions of dollars at stake in the settlement, said at the hearing the matter could go to the U.S. Supreme Court.

The investors he represents, including BlackRock Inc and Allianz SE's Pimco, believe the settlement should be returned to state court, as does Bank of America and trustee Bank of New York Mellon Corp .

Madden told the three-judge panel that more litigation surrounding the settlement could lead to the federal judge's work turning out "to be a waste of time."

The Bank of America pact was intended to address claims by investors who said the seemingly safe securities they bought proved toxic because they were backed by risky home loans based on faulty underwriting practices.

The agreement, which applied to 530 mortgage securitization trusts with $174 billion of unpaid principal, was seen as a template for other banks facing mortgage-backed securities breach-of-contract claims.

The same law firm that negotiated the Bank of America pact for institutional investors, Gibbs & Bruns, has also sent demands for an investigation to trustees overseeing mortgage securities sponsored by JPMorgan Chase & Co , Morgan Stanley and Wells Fargo & Co . If the settlement remains tied up in court, it could potentially also delay resolution of similar claims against other banks.

Initially, the Bank of America settlement was sent to a New York State Supreme Court judge in Manhattan to review. It was in state court that the parties used a New York trust law known as Article 77 that is normally reserved for resolving family trust issues.

But the agreement drew criticism from investors, including a group known as Walnut Place LLC, who were not part of the talks, but would be bound by the settlement terms. They complained the $8.5 billion payout was too low and wanted the case moved to federal court for more review. Walnut Place is the hedge fund Baupost Group, according to court documents.

In October, U.S. District Judge William Pauley ruled that the proposed settlement belonged in his court, citing "core federal interests" in the integrity of banks and securities markets.

Jacobs indicated on Wednesday that, even if the appeals panel affirmed Pauley's decision and he eventually signed off on a settlement in federal court, the litigation would not end there.

Appeals court Judges Peter Hall and Raymond Lohier were also on Wednesday's panel. Hall focused on arguments by Walnut Place that the accord was a "mass action" involving hundreds of trusts.

"I don't even see it as a mass action because it didn't start out as such," Hall said during the one-hour long hearing, referring to the case's introduction in state court as a settlement rather than a lawsuit on behalf of a class of plaintiffs.

The case is Bank of New York Mellon v. Walnut Place LLC et al, 2nd U.S. Circuit Court of Appeals, No. 11-4571.

(Reporting By Grant McCool and Alison Frankel; Editing by Martha Graybow and Andre Grenon)

Morgan Stanley Wells fargo Court Jpmorgan chase