The London Stock Exchange's C$3.6 billion ($3.7 billion) plan to buy its Toronto counterpart collapsed on Wednesday in the face of a competing bid led by Canadian banks, leaving the UK exchange itself vulnerable to takeover.

The failure, which follows Singapore Exchange Ltd.'s scuttled bid for Australia's ASX Ltd, is the latest sign of nationalist pride frustrating cross-border deals for highly symbolic capital markets.

The failed bid from LSE opens the door to a hostile C$3.8 billion offer for TMX Group, operator of the Toronto Stock Exchange, from the Maple Group consortium. The consortium bid is a made-in-Canada alternative to a takeover that would have put a big domestic asset in foreign hands.

This is a group of Canadians, businesses that came together and have asserted themselves, said Dwight Duncan, the province of Ontario's finance minister and an early opponent of the LSE deal.

In having to retreat from Canada, the LSE also draws attention to itself as a takeover target as exchanges consolidate to try to grow and broaden geographic reach, and to fight off rivals and new market entrants.

Nasdaq OMX Group, smarting from its own failure in the United States to buy New York Stock Exchange parent NYSE Euronext, could be a contender for an alternative transatlantic combination with the LSE.

While the failed deal probably puts an end to TMX's M&A ambitions, other exchange operators will likely continue to look for partners. This reinforces my belief that we should expect more mergers, not less, said Ed Ditmire, New York-based analyst for Macquarie Securities.

Nasdaq and LSE shares rose in late trade on Wednesday, indicating that speculation about a tie-up could be brewing.

It would not surprise me to see Nasdaq and LSE talking, said Jamie Selway, New York-based market structure expert and managing director of strategy at Investment Technology Group.

The TMX deal's collapse is a black eye for LSE Chief Executive Xavier Rolet, who banked his reputation on sealing the deal.

Rolet was to have led a LSE-TMX exchange group, which would have been a heavyweight global player and No. 1 in listing energy and mining companies.

But the support he got from TMX's management and board wasn't enough to overcome opposition from within Canada's tight-knit banking sector.

Four of the country's biggest banks were the lead players in the bid from Maple, a consortium that also included pension funds and financial services firms. Canada's other two big banks were advisers to the LSE proposal.

NOT ENOUGH VOTES

In brief statements issued one day before a shareholder vote, the two exchanges said they realized from an early tally of proxy votes that TMX shareholders would not give them the two-thirds majority needed to approve their friendly deal.

TMX Group said it would now review opportunities, including the Maple offer.

LSE and TMX were both in positions where they weren't quite big enough or diverse and fast-growing enough to control their own destiny, said Justin Schack, managing director of market structure analysis at New York-based agency brokerage Rosenblatt Securities.

They did the best deal that they probably could. Now that that's not going to happen TMX has Maple to deal with, while LSE is out on its own again, and there aren't many partners out there where they could be the acquirer rather than the target.

Maple has offered C$3.8 billion for TMX, mostly in cash.

LSE's mostly stock offer was worth about C$49 a share.

It would have needed a green light from a government that last year vetoed a big international takeover as not being in Canada's best interests.

TMX shares touched a high of C$44.80 after the deal was scrapped before easing back to close at C$44.20. That's well below the Maple offer price of C$50 a share.

Maple wants to wrap Alpha, Canada's biggest alternative trading venue, and the CDS stock-trading clearing system into a post-takeover TMX. That would give it a market share of more than 80 percent and leave it facing antitrust concerns.

Now we need to see what the Competition Bureau thinks of Maple. We also need to see if shareholders support Maple. I think they will, I don't see how they won't, said Alison Crosthwait, director of global trading strategy at Instinet.

We're going back to more of a closely held, interested parties controlling the exchange.

Canada's independent Competition Bureau has bared its teeth lately on several fronts, getting a C$10 million payment from BCE Inc's Bell Canada unit for misleading advertising, and seeking to block a joint venture between Air Canada and United Continental.

(Additional reporting by Claire Sibonney, Andrea Hopkins, Euan Rocha, Solarina Ho, Jonathan Spicer, Allison Martell and Trish Nixon; editing by Janet Guttsman and Peter Galloway)