If you watch Law and Order or many other TV shows, you already know something about Miranda warnings and taking the Fifth. The police have to Mirandize you, and even in court, you can say, “I refuse to answer on the grounds that I may incriminate myself.” There are limits of course, but most people think these rights are fundamental, across every alleged crime. But does it work with taxes and the IRS? Not usually.

Many people find that a shock. Remember the IRS official Lois Lerner, who ran the IRS unit accused of targeting conservatives some years back? She didn’t want to answer questions about alleged IRS targeting. As a result, she took the Fifth. Congress held her in contempt, but the government declined to prosecute her. It was all a controversial episode. Private taxpayers aren’t usually so lucky when it comes to their own tax returns and investigations.

In fact, merely invoking the Fifth in a tax case can invite penalties or get the IRS looking more harshly at you. Let’s start with tax returns themselves. You have to file them and you have to report your income. Way back in 1927, the Supreme Court considered a man who refused to file a tax return, claiming that to do so would incriminate him. In U.S. v. Sullivan, the Supreme Court said that it was too bad if disclosing illegal income opened him up to prosecution.

Even a criminal must file tax returns and pay taxes. After all, that is how they got Al Capone. You have to file a tax return, and you have to do it accurately. What if the IRS asks you questions you are afraid to answer? Answering IRS questions in an audit or investigation can be nerve-wracking. Do not speak up without your lawyer present, and ask your lawyer what is fair to discuss. But claiming Fifth Amendment protection in tax cases can be a mistake.

One of the biggest issues involves books and records. You have to keep them in order to fulfill your tax filing obligations. You even have to keep bank account records for accounts outside the U.S. Undisclosed offshore bank accounts can qualify as money laundering. So, if the IRS asks you if you have any foreign bank accounts, can you take the Fifth?

You can, but it probably won’t help. Even if you claim the Fifth, the IRS can hand you an “information document request” to produce your records. You can refuse, but the IRS will issue a summons. If you refuse to answer that, the IRS will take you to court, which will probably order you to comply. But, doesn’t your constitutional right to take the Fifth trump the IRS?

Not always. Ironically, you can refuse to talk, but you cannot refuse to produce the documents. Your own private papers are personal records, and if they might incriminate you, they are protected by the Fifth Amendment. But the Required Records doctrine says you must hand over documents no matter how incriminating. The government requires you to keep certain records, and the government has a right to inspect them.

The IRS and prosecutors have exploited this rule. It can mean that pleading the Fifth in response to a subpoena for foreign account records can cause even more trouble than claiming it on your tax returns. Required records are those where the reporting has a regulatory purpose, where a person must customarily keep the records the record-keeping scheme requires him to keep, and the records have public aspects.

In the case of foreign bank records particularly, the courts uniformly deny Fifth Amendment protection. Numerous courts have all given the IRS a free pass, ruling that no Fifth Amendment protection applies. Despite repeated requests, the U.S. Supreme Court has been unwilling to hear this issue.

So, is it likely that the Fifth Amendment will be much help on your taxes? Not really. In most cases, a tax audit is civil and there is little risk that it will become criminal. However, just think about this: a majority of criminal tax cases come directly out of civil tax cases. The IRS civil auditors ‘refer’ a case to the IRS Criminal Investigation Division. The IRS civil auditor will not tell you this is occurring, so the first time you hear about it, your case may have gone from bad to worse. That means having a lawyer and being careful can be wise.

Robert W. Wood is a tax lawyer and managing partner at Wood LLP. He can be reached at Wood@WoodLLP.com.