Charlie Munger said Berkshire would be worth double if he and Buffett used leverage

The late investment icon Charlie Munger said Berkshire Hathaway, the conglomerate he and Warren Buffett built over the last five decades, could have doubled its value if they applied leverage, or borrowed money, when buying businesses and common stocks.

Munger, Berkshire Hathaway's vice chairman who died Tuesday just a month shy of his 100th birthday, stressed that he and Buffett almost never used this common Wall Street practice, because they always put their shareholders first.

"Berkshire could easily be worth twice what it is now. And the extra risk you would've taken would've been practically nothing. All we had to do is just use a little more leverage that was easily available," Munger said in CNBC's special "Charlie Munger: A Life of Wit and Wisdom," which aired Thursday.

"The reason we didn't is the idea of disappointing a lot of people who had trusted us when we were young ... If we lost three quarters of our money, we were still very rich. That wasn't true of every shareholder," he told CNBC's Becky Quick in the previously unaired interview. "Losing three quarters of the money would've been a big letdown."

The use of leverage is prevalent on Wall Street as it provides a way to boost buying power and enhance the potential return in any given investment. But it also significantly increases the risk as losses can multiply quickly if the investment doesn't pan out as expected.

Beware an 'unsettled mind'

Buffett, often called the "Oracle of Omaha," previously explained the perils of using debt and leverage to buy stocks, saying it can make an investor short-sighted and panicky when times turn volatile.

"There is simply no telling how far stocks can fall in a short period," he wrote in his 2017 annual letter to shareholders. "Even if your borrowings are small and your positions aren't immediately threatened by the plunging market, your mind may well become rattled by scary headlines and breathless commentary. And an unsettled mind will not make good decisions."

Munger said he and Buffett had been "very cautious" in handling their shareholders' money over the years. Berkshire shareholders tend to be long-term investors like all the conglomerate's top executives, often treating their stock like a savings account.

"If Warren and I had owned Berkshire without any shareholders that we knew, we would've made more. We would've used more leverage," Munger said in the CNBC special.

Still, Munger acknowledged that Berkshire did use leverage in the form of its insurance float. Insurers receive premiums upfront and pay claims later, so they can invest the large sums collected — cost free — for their own benefit.

"Insurance float gave us some leverage. That's why we went into it," he said.