“The vast majority of our net worth would have been tied up in Clayton Homes, so there’s no way we would have been able to contribute philanthropically at these kinds of levels,” Clayton said.

Then there were business owners who received the added benefit of stock, such as the Childs, the Helzberg jewelry store family or the Alfond and Lunder families of Dexter Shoe Co. While Buffett has long preferred to pay cash for companies, he has said that Berkshire would consider issuing shares when it receives as much in intrinsic value as it gives. For those who made such deals and have hung on to their stakes, it’s paid off.

The 25,203 shares of Berkshire stock Buffett used to buy Dexter in 1993, for example, would now be worth about $8 billion.

Barnett Helzberg Jr., who struck a deal with Buffett after running into him on a New York street and telling him about the business, declined to say how much Berkshire stock he owns. But he said Buffett’s inclination to stick with businesses long-term is an attractive draw for family-run companies.

‘Worst Deal’

Financing acquisitions with stock, while attractive to the sellers, hasn’t always panned out for Buffett. The 88-year-old investor said Dexter was the “worst deal” he’s made. After first failing to properly assess the business’s competitive advantage, he compounded that mistake by using Berkshire stock, Buffett wrote to shareholders.

Irv Blumkin, who’s the grandson of Nebraska Furniture Mart founder Rose Blumkin and whose family advised Child to hold out for Berkshire shares, declined to comment. Attempts to reach Peter Lunder, as well as relatives of the late Daniel Monen, Van Tuyl and a foundation associated with Al Ueltschi were unsuccessful. Berkshire didn’t respond to a message seeking comment left with Buffett’s assistant.

Part of the Berkshire aura has been the loyal shareholder base—called “co-venturers” by Buffett—that has benefited as the firm grew. Some, like Monen, got in on the ground floor when Buffett was running his partnership and stuck around as the investor began assembling Berkshire’s many parts. Others such as Stewart Horejsi jumped in later, but ended up accumulating large Berkshire stakes.

The historical returns Buffett’s delivered over the years is the big reason why so many will make the pilgrimage to Omaha for the firm’s annual meeting Saturday. Still, the kind of performance they’ve grown to expect is increasingly hard to come by.

Berkshire has become so large that it’s squeezing Buffett’s ability to generate the returns he once did. The universe of potential investments big enough to have a significant impact on Berkshire’s results is “very, very limited,” he said in a 2015 interview with CNBC.