Stocks fell on Wednesday as investors worried the Federal Reserve is closer to pulling back on extraordinary measures to inject funding to shore up the economy.

The Fed's policy-setters kept interest rates unchanged, as expected, but they also said the U.S. central bank would slow purchases of mortgage debt to extend that program's life until the end of March. That was seen as a step toward a measured withdrawal of its extraordinary support for the economy during the downturn.

They're talking about removing some of the various packages they have in place for purchasing mortgages and other instruments in debt markets that was kind of keeping everything flowing, said Kurt Brunner, portfolio manager at the Swarthmore Group in Philadelphia.

There's a broad concern about what happens when the Fed gets out of the way.

Among the casualties were banks, housing stocks and energy shares. The Dow Jones home construction index <.DJUSHB> slid 3.4 percent, while the S&P energy index <.GSPE> declined 2 percent, a decline that coincided with a sharp slide in crude oil prices.

The Dow Jones industrial average <.DJI> shed 81.32 points, or 0.83 percent, to 9,748.55. The Standard & Poor's 500 Index <.SPX> declined 10.79 points, or 1.01 percent, to 1,060.87. The Nasdaq Composite Index <.IXIC> lost 14.88 points, or 0.69 percent, to 2,131.42.

Initially stocks had risen sharply following the Fed's comment that economic activity was picking up, but in the last hour of trading the market reversed course as investors fretted about the timing of the removal of some of the Fed's stimulus.

The other worry was the Fed's vow that interest rates will stay low for an extended time.

Keeping interest rates low -- it's positive for the consumer, but it's tougher for banks to make money with interest rates so low. The financial and banking sector is a big part of the economy, said Dan Faretta, senior market strategist at Lind-Waldock, a retail brokerage firm, in Chicago.

The market's runup of nearly 60 percent over six months might have also caused some investors to use the Fed's statement as reason to book profits, he added.

Slowing mortgage purchases is something the Fed could consider in another six months, Faretta said. There's still a lot of problems with mortgages, the housing market in general, as well as the banking sector.

JPMorgan was the Dow's top drag, falling 3.03 percent to $45.06.

(Additional reporting by Leah Schnurr; Editing by Kenneth Barry)