The U.S. Justice Department urged a New York court on Friday to reject Google's controversial deal with authors and publishers that would allow the search engine giant to create a massive online digital library.

The Justice Department said in a filing that the court should reject the proposed settlement in its current form and encourage the parties to continue negotiations to modify it so as to comply with ... copyright and antitrust laws.

The filing was to a New York court considering settlement of a 2005 class action lawsuit. In that suit, authors and publishers had accused Google of copyright infringement for scanning libraries full of books.

The proposed settlement, which was reached last October, would establish a registry to pay authors for works in Google's book search.

The department said it had not finished its investigation but said that there was a significant potential that the division would eventually decide the settlement broke antitrust law.

In a statement, Google, the Authors Guild and the Association of American Publishers said the Department of Justice filing recognizes the value the settlement can provide by unlocking access to millions of books in the United States.

We are considering the points raised by the Department and look forward to addressing them as the court proceedings continue, they added in the statement.

The Justice Department noted that the settlement appears to give book publishers the power to restrict price competition and would give Google de facto exclusivity in distribution of orphan works, books which are in copyright but the rights holder cannot be located.

The department noted that the settlement allows the Book Registry to license these works and to then distribute the money.

The structure of the proposed settlement, therefore, pits the interests of one part of the class (known rightsholders) against the interests of another part of the class (orphan works rightsholders, the department said in its filing.

The Justice Department indicated that ongoing talks with Google -- which a department official characterized as very constructive -- could lead to changes that would make the settlement acceptable.

We'll see if they come to fruition, the official said.

John Simpson of Consumer Watchdog said he was pleased with the filing.

As the Justice brief makes clear, the proposed class-action settlement is monumentally overbroad and invites the court to overstep its legal jurisdiction, to the detriment of consumers and the public, Simpson said in an email.

Earlier on Friday, European groups opposed to the settlement urged European Union regulators to investigate.

Under the terms of the settlement, Google will pay $125 million to create a Book Rights Registry, where authors and publishers register works and are paid for books and other publications that the search giant would put online.

A fairness hearing on the deal has been set for October 7 in the federal court in Manhattan.

The case is Authors Guild et al v Google Inc 05-08136 in U.S. District Court for the Southern District of New York (Manhattan)

(Editing by Paul Simao)

Antitrust law Google