This is a very common question that I hear all the time and I can see why. Many people, as they approach retirement, have paid off all their debt except for their mortgage. They have saved diligently in their retirement accounts (ie. IRA, 401(k), TSP) and they have a good sized nest egg. It seems to be a match made in heaven. You have this debt and you have enough money to pay it off, so shouldn’t you?

The short answer: Probably not.

The rest of this article is the long answer.

One of the first questions is are you at least 59 and six months old? If you aren’t, any withdrawals may be subject to a 10% penalty as well as taxes. Sometimes this penalty is waived if you have already retired but I would do thorough research beforehand to make sure you are in the clear.

The big one: taxesThis is one of the biggest things to keep in mind. For most people, most of their retirement savings are on the traditional side and not the Roth. This means that when you put the money in, you didn’t have to pay taxes on that money but you do when you take it out. If you do have money in a Roth IRA, Roth TSP, or Roth 401(k), any qualified distribution will come out completely tax free.

One misconception that I see is that some assume that traditional retirement account distributions will be taxed at capital gains tax rates but this is not the case. These distributions are simply added to your taxable income in whatever year you take the money out.

It is also important to note that the IRS generally requires the institutions that hold retirement accounts (except IRA’s) to withhold 20% of any traditional distributions to cover taxes. Consequently, you may have to take out more than initially thought. This doesn’t mean that you’ll owe exactly 20% of your distribution in taxes. It could be more or less than that.

Let’s do a quick example to see how this would work.

Let’s say a couple is over 59 and ½ and their joint income comes to about 80K. They owe $250,000 on their house and they have $500,000 of tax deferred money in their 401(k). They decide that they want to be completely debt free and pay off their mortgage with these funds.

They request a withdrawal of 250k and the institution sends them $200,000. The remaining 50k gets sent directly to the IRS. Now that they know about the 20% withholding, they request an additional $62,500 so that they get 50k after the withholding.

They ended up having to take out a total of $312,500 from their 401(k) which is a big bill just to get rid of a mortgage payment of $1,500-$1,800.

But this is where it gets sticky. That $312,500 would then be added to their income of 80K for a total of $392,500. To keep the tax calculations really simple, let’s assume that the only deduction they have is the standard deduction for a couple which is $24,800 (in 2020). Their taxable income would be: $392,500 — $24,800 = $367,700

Before with an income of only 80k, they were in the 12% tax bracket. Now they are slammed up into the 32% marginal tax bracket with an effective tax rate of around 20%. That means they would have to pay almost $75,000 in income taxes this year!

Now this cost would be slightly lessened by the fact that you would save money in interest over the rest of their would-be mortgage term. But even if you saved $100,000 in interest over the course of the 15-20 years that you would have had a payment, it costs you $75,000 of cold hard cash right now.

Opportunity cost: The second kickerNow if the numbers already didn’t look bad enough, the opportunity cost of the money taken out of the 401(k) is sure to be the final straw. If this couple would have kept the $312,500 in their 401(k) and invested it for 15 years, they’d only have to have an annual return of 1.9% to make $100,000 over that time. The difference becomes unignorable once we assume they earned anything more than that.

Wrap upNow I am not saying that it never makes sense to pay off your mortgage early with your retirement accounts. Sometimes it does. And in those cases, it can make a lot of sense to pay it off over a period of time and not all at once. That way you are able to stay in lower tax brackets and pay less taxes over that period.

You’ll have to find a good balance between paying interest and paying taxes to know what is best for you.

Just make sure you understand the long-term ramifications. Everyone’s situation is a little bit different but it is your responsibility to know what this type of decision means for you.

Dallen Haws is a personal finance and business enthusiast, ASU grad (Fear the Fork!), co-founder and financial planner at Haws Financial Planning.