One of the biggest oil market manipulation cases undertaken by regulators is entering a new phase in which a judge will mediate settlement talks, three years after the case was launched.

The Commodity Futures Trading Commission case against the U.S. unit of Optiver Holding BV, a Netherlands-based trading firm, has been referred to U.S. Magistrate Judge Theodore Katz for settlement, according to a filing in the U.S. district court in Manhattan.

The June 15 order marks a departure from a long series of legal submissions since the CFTC in July 2008 charged Optiver with using a rapid-fire trading program to manipulate crude, gasoline and heating oil prices on the New York Mercantile Exchange.

According to the futures regulator, Optiver reaped a $1 million profit in 2007 by banging the close. This is an illegal strategy in which a firm accumulates a large position just before the market closes, and offsets that position at the close itself, manipulating prices through sheer volume of trades.

The case revealed details about computer software called the hammer that rapidly entered a series of orders to allegedly manipulate markets.

It also included emails and phone recordings showing efforts by traders at Optiver's Chicago branch to move, whack and bully oil prices -- providing rare insight into the dark side of high-speed electronic trading, which has grown rapidly in the last decade.

The latest court order was signed by Chief Judge Loretta Preska, who had handled the case from the beginning. No date had been set for mediation, and the scope of Katz's involvement is not immediately clear.

A CFTC spokesman and Stacie Hartman, a lawyer for Optiver, declined to comment, citing ongoing litigation. A lawyer for one of the three Optiver employees named in the lawsuit did not immediately return requests for comment.

NEW ENFORCEMENT POWERS, PRESSURES

Mediation can signal outstanding disagreements that the court or the opposing parties believe a judge, acting as a facilitator, can help resolve. The process is not legally binding, and talks could fall apart, resulting in further litigation.

A lot of times, the CFTC wants to have some agreed-to facts and that's a negotiated instrument, said a lawyer who deals sometimes with the regulator and requested anonymity.

Often, this lawyer said, the CFTC will seek a trading ban, which could be a sticking point. Obviously that's a pretty severe sanction.

CFTC Chairman Gary Gensler has been zeroing in on market manipulation with tough talk and new enforcement powers granted in last year's Dodd-Frank financial overhaul.

The law requires the agency to show only that a trader acted in a manner that could disrupt the market, making it easier to prove a case -- though it would not apply retroactively to Optiver.

Reflecting a more aggressive stance, the CFTC in May sued oil trader Arcadia Petroleum for manipulation in its biggest such case.

Commodity markets have been a focus of greater regulatory attention since crude oil prices soared close to $150 a barrel in 2008. This stoked criticism in Washington over speculation and worries over unfair and damaging trading practices.

While prices crashed during the financial crisis, they have traded back above $100 this year, renewing political pressure on the CFTC. Crude for August delivery traded Wednesday around $93.13 a barrel.

Optiver is a household name in Chicago's and Amsterdam's electronic trading communities, where it is known for market making and arbitrage strategies in options and other derivatives.

The proprietary firm trades its own money, employs some 600 people, and is a member of a newly formed European lobby group to weigh in on rule changes for high-frequency traders.

(Reporting by Jonathan Spicer, additional reporting by David Sheppard, editing by Gerald E. McCormick and Robert MacMillan)