The U.S. futures regulator asked traders on Thursday how best to define trading practices now banned in a new Wall Street reform law, but got little clarity from the experts, who fear an overly restrictive crackdown.

The Commodity Futures Trading Commission is grappling with how to give markets more direction on how they intend to carry out a ban in the Dodd-Frank law on three disruptive practices -- and whether regulators should go further and rein in high-frequency traders.

Beginning in July, the CFTC is required by the Dodd-Frank act to ban certain trading practices, including banging the close, acquiring a big position and then offsetting it before trading ends, and spoofing, when a trader makes bids or offers but cancels them before execution.

There is no doubt about it, we have our work cut out to provide clarity on those strategies, Scott O'Malia, a CFTC commissioner who leads the agency's technology advisory committee, told Reuters.

People want certainty from rules, but they don't want to be overly prescriptive to limit innovation, to limit function of some sort that they have in mind. That's the balancing act, he said.

O'Malia said he supported putting in place a series of requirements that traders must follow to access the market. Those measures, he said, are needed to ensure algorithms and high-frequency trading programs will not manipulate the market and give traders confidence they are getting the best price.

At a day-long public meeting, panelists said the rules must remove grey areas so firms know how to run their trading practices without fear they will be violating the law.

New regulations must be crystal clear, and allow firms to put in the procedures and processes to ensure that they're not engaging in activity that would have that character, Adam Nunes of Hudson River Trading Group told the panel.

But traders and academics had few suggestions for what those rules should be, despite repeated questions from some of the agency's top enforcement staffers.

It is dangerous to be pigeon-holed with too many rules, said Rajiv Fernando, chief executive of Chopper Trading.

FLASH CRASH CREATES MOMENTUM

The CFTC's new powers have taken on greater public profile in the wake of the May 6 flash crash where algorithmic trading was seen as contributing to volatility.

Traders warned that imposing new types of controls could threaten liquidity, making markets thinner and more volatile.

We are worried you will drive out the good with the bad and we will lose liquidity, said John Hyland, chief investment officer at the United States Commodity Funds.

Traders said the CFTC needs to look at patterns of trade over time, and closely examine intent -- something that historically has been difficult for the CFTC to prove, and may be getting even more difficult with new technology, said Joel Hasbrouck, a business professor at New York University.

Greg Mocek, a former enforcement chief at the CFTC and now a partner at McDermott Will & Emery in Washington, said unless the definition of these trading practices is clear, it would create problems for traders and the agency.

The vagueness is going to chill legitimate trading, there's no doubt about it, said Mocek, who spoke for the Commodity Markets Council.

The vagueness is also going to impede the ability of the enforcement division to bring cases, he said.

SPEED BECOMING ARMS RACE

Although a government review of May 6 did not blame high-frequency traders, the Securities and Exchange Commission and CFTC are under pressure to rein in the rapid traders, who use computer-driven algorithms.

High-frequency trading may account for 60 percent of U.S. futures trade by the end of 2010, according to one estimate -- a dramatic shift from traditional pit-dominated trade.

DRW Trading Chief Executive Don Wilson said the CFTC should require high-frequency traders to have good risk-management practices to ensure algorithms don't run amok, but stopped short of suggesting mandatory pre-trade testing for algos.

Algorithmic traders that have not put in place reasonable procedures to ensure that something like this doesn't happen absolutely should be held accountable, Wilson said.

While high-frequency traders argued their orders helped make markets efficient, New York University's Hasbrouck said the CFTC should look for ways to slow the market down.

This is ultimately an arms race, he said.

The race to be first is ultimately more unstable than people simply trying to be fast, Hasbrouck said.

Tyson Slocum, a director at Public Citizen's Energy Program who works on energy issues, said regulators should put some brakes on high-frequency trades.

We need to question whether or not these advances in technologies employed with high-frequency trading and these complex algorithms are far beyond the ability of (regulators) to be able to protect consumers from abusive practices.

(Editing by Alden Bentley)