The cost of complying with new securities rules is making it harder for small and mid-sized U.S. brokerages to stay profitable -- and may force more mergers in the sector.

The market structure changes -- from short-sale restrictions to rules that stop trading in a stock when shares drop too steeply -- are keeping compliance and technology professionals buzzing at financial institutions engaged in equity trading, according to interviews with executives.

Small brokerages face other pressures, including declining trading volumes, but the need to stay on the right side of evolving laws in the high-speed marketplace has its burdens.

Risk and compliance may be two of the bigger drivers of cost in all of the industry, said Bill Yancey, chief executive officer of Dallas-based Penson Worldwide's U.S. clearing operation, which has some 300 broker-dealer clients.

Yancey listed hardware, software, maintenance and surveillance as areas that evolve with the new regulations.

These costs of the layering on of regulation are adding up from an industry perspective, and the implementation side of it in most cases is pretty complex, he said.

Last year's flash crash brought about new rules that had long been debated at the U.S. Securities and Exchange Commission.

Among them is a ban on naked access, in which brokers rent out their access to stock markets to high-frequency traders that seek an edge through being able to buy and sell microseconds faster than others.

The naked access ban has prompted some high-frequency traders to convert into formal broker-dealers, while some existing brokers formed new BD units to secure additional licenses. The ban is set to take effect July 14 but will probably be delayed, according to a person familiar with the SEC's thinking.

Aiming to stabilize markets or to eliminate advantages of some traders, regulators in the last 18 months also tightened short selling rules and adopted trading pauses known as circuit breakers, a scheme to be replaced by price limits.

The priciest change on the horizon is the SEC's proposed consolidated audit trail. This record of all trading activity was estimated to cost the industry $4 billion to implement, plus $2.1 billion in annual maintenance.

Where we have a regulatory change that requires us to do something different with our business model, like add another broker dealer, we've had to do that, and that has increased costs modestly, said Alfred Berkeley, chairman of electronic institutional broker Pipeline Financial Services.

OTHER PROBLEMS

To be sure, most long-time brokers that execute trades for clients cite market making as their primary problem. A series of rule changes over the last 15 years, from decimalization to a proliferation of electronic trading venues, made it very tough for them to earn the bid-ask spread in large-cap stocks.

Combine that with sluggish trading volumes in the last year, due in part to an uncertain economic recovery and shaky markets, and costs of regulation can seem relatively small.

They're all there, but that would be down my list of impacts to managing that business, said Piper Jaffray Cos CEO Andrew Duff.

Mark Lehmann, president of JMP Group Inc's investment banking unit, agreed that the biggest impact is not coming from regulatory changes. The biggest changes really are due to less volume, people going off the exchanges, and that the electronic trading component has increased, he said.

Yet some say recent mergers involving stock market-focused companies is evidence of the squeeze, or opportunity, from the regulations.

Wedbush Inc, which like Penson is a top clearer of trades for clients, last week bought Lime Brokerage LLC, which is known for its rapid-fire trading technology services. And in December, Nasdaq OMX Group bought risk-surveillance provider FTEN Inc.

You're definitely seeing some of these mid-tier guys looking at other opportunities, or being gobbled up, said William Karsh, an independent consultant on market structure, and former chief operating officer at the Direct Edge stock exchange.

With all the regulatory overhang right now people are spending so much time just dealing with refreshes of technology, getting ready for the regulatory onslaught.

Private banking and wealth management executives, too, expect more broker mergers as they struggle to contain costs that are in part driven up by new regulations.

The SEC, scrambling to write dozens of financial reforms included in the Dodd-Frank bill before a summer deadline, is expected to once again turn its attention to market structure later this year -- with more tweaks and adjustments to come.

With most regulations, you get to the straw that broke the camel's back, said Jeffrey Solomon, chief operating officer at brokerage and investment bank Cowen Group Inc.

I can't tell you which one it is, he said. All I can tell you is the macro trend, that it's more expensive for Wall Street firms to trade today.

(Reporting by Jonathan Spicer; Editing by Lisa Von Ahn and Maureen Bavdek)