Are you stuck with IRS penalties if you get a notice from the IRS? The best way to avoid penalties is to file on time, pay on time, and don’t make mistakes. But that’s easier said than done, and sometimes, you are stuck. Deciding how much effort and money to spend on trying to undo the penalty part of the equation—and even the extra taxes and interest the IRS wants to charge you—can be tough. It depends on many issues, starting with how much is at stake.

If the extra taxes are small and the penalty is small, my advice is to pay them, even if the IRS is wrong. It’s not worth the cost and effort of fighting about it when there’s little money at stake. Besides, fighting about one tax issue can sometimes end up drawing you into tax problems on other issues when the IRS starts looking at your return more closely. Of course, size is relative, and small to one person isn’t to someone else.

Then there’s the nature of the penalty. Take penalties for filing your tax return late, or for paying late. A penalty for filing your tax return late or paying late is probably not fixable, even if your accountant had your return, you told him to file it, and he forgot. The IRS says everyone is responsible for timely filing and paying their taxes, and those duties cannot be delegated. Even if you rely on accountants, bookkeepers, or attorneys, you cannot delegate responsibility to timely filing and paying. Some errors aren’t reasonable.

But whatever the size and nature of the penalties, how about claiming that you had “reasonable cause?” That sounds simple, but the IRS may not agree. How the IRS evaluates reasonable cause depends on which penalty is at stake and how you behaved. The IRS applies reasonableness on a case-by-case basis, and the stakes can be big, from 20% to a whopping 75% of your tax bill.

The tax code is chock-full of penalty provisions, and you always want to behave reasonably and in good faith. Were you careful in doing your best? A mistake or two can often be explained, even if it is clear in the end you were wrong. But if you have a dozen errors, it’s less likely the IRS will let you off the hook.

The IRS considers your experience, knowledge, education, and whether you relied on advice from your tax adviser. If you relied on someone, cherry-picking what you tell your tax adviser to get the answer you want is not reasonable. Who your adviser is matters, too. The IRS says your adviser must have knowledge and expertise related to the tax matter to be able to advise you.

Is relying on tax advice from the IRS always reasonable? Not necessarily. First, it has to be in writing. If you point to something the IRS told you in writing, the IRS evaluates it and determines if the advice was in response to a specific request and related to the facts contained in that request. The IRS also wants to know if you actually relied on its advice.

Trying hard to report the proper tax liability is the most important factor in determining reasonable cause. The IRS considers all facts and circumstances, your compliance history, length of time, and circumstances beyond your control. You might assume that this is just about the single tax year involved. However, the IRS may look at the three previous tax years for your payment patterns and compliance history.

If you are hit with the same penalty each year, you may not be reasonable. If this is your first problem, the IRS considers it, along with your other reasons and circumstances. The rules and many specific penalties can be tricky, so if your penalty dollars are significant, getting some professional advice can be wise.

Robert W. Wood is a tax lawyer and managing partner at Wood LLP. He can be reached at Wood@WoodLLP.com