I once asked an IRS agent at a tax seminar this same question. He said yes you could if the loans had originated from the business; even if the business was no longer operating. I had the same "lone item" audit concern and also asked about that. He said that it probably would stand out, but was not an automatic audit, and was a legitimate business deduction. So, if you're lucky, have no other risky items, and have the time for a possible audit...
At the time (about 12 years ago) a client had sold her rental apartments, still owed a large loan on them, and we made her a "lone ranger" interest expense entry on an "E" for about three years. She was never audited.
Regards, BB