The watchdog for the $700 billion bank bailout program said on Thursday it will remain vigilant in protecting taxpayers especially now congressional supervisors have stopped oversight.

In its latest quarterly report, the special inspector general for the Troubled Asset Relief Program (TARP) emphasized that the bailout program was not over.

Until early this year, the inspector general's office was one of two government groups charged with regularly scrutinizing TARP. The Congressional Oversight Panel was created by Congress to review the program but it shut it's doors in early April as mandated by the law.

Recent events, such as the last oversight report from congressional overseers and the continued repayment of bailout funds by larger banks, have underpinned public perception that TARP is ending.

This is simply not the case. TARP may have entered a new phase, but it is far from over, said the report released on Thursday.

As of the end of March, about $146.8 billion in bailout funds were still outstanding. The special inspector general's report said it would remain on watch until the last taxpayer dollar was repaid.

The report continued to criticize the Obama administration's foreclosure prevention program, which offers incentives for lenders to modify loans.

Treasury has said it will start grading the largest mortgage servicers involved with the program and plans to withhold financial incentives for those that receive an unsatisfactory grade.

These may be encouraging first steps, said the watchdog report. However, it is too early to tell whether these steps will have a meaningful impact.

So far, about 560,000 struggling borrowers have received new loan terms under the administration's Home Affordable Modification Program. Republican lawmakers are trying to kill the program and say it is ineffective because it has only helped a small number of borrowers, among other things.

(Reporting by Rachelle Younglai; Editing by Andrew Hay)