Aside from maybe the odd cheeseburger stain on his tie, nothing much sticks to Warren Buffett.

Whether his underlings are convicted of helping insurance companies inflate results or a major company he helps oversee is sanctioned for accounting shenanigans, his admirers don't seem to care. Or at least, they haven't historically.

But with a key Buffett lieutenant resigning under a cloud recently, some sophisticated investors are no longer willing to overlook the obvious. For all the shareholders who still consider Buffett the epitome of American capitalism, there are others who wonder whether the time may be near for Buffett to take a graceful bow and exit the stage.

Some will clamor for that this weekend, when 40,000 of his shareholders prepare to descend on Nebraska for the annual meeting of Berkshire Hathaway, the ice-cream-to-insurance conglomerate he runs with absolute authority.

I want to hear more about Sokol, I want to hear more about how they're going to outperform the markets. I want to hear about what (Buffett's recent) trip to India leads us to believe about how the money is going to be invested in the future, said Michael Yoshikami, chief executive of wealth management firm YCMNET Advisors and a widely quoted Berkshire shareholder.

Investor disappointment reflects not just the revelation that David Sokol, once Buffett's presumed successor as chief executive, bought stock in a company he then pushed Buffett to acquire. It is also because of Berkshire's lackluster performance recently, and questions about the firm's ability to thrive after its octogenarian chairman and chief executive moves on.

Berkshire Hathaway has grown exponentially over decades, but many investors question how it can possibly do as well in the future. With the dozens of companies that Berkshire Hathaway owns having had relatively little oversight for years (by Buffett's own proud admission), some wonder how much earnings power Berkshire actually has and whether future earnings can be as strong as past.

Obviously Berkshire has intrinsic value but now I have to question that intrinsic value, said Janet Tavakoli, an expert on derivatives and author of Dear Mr. Buffett, a 2009 book laden with fulsome praise for the legendary investor. Tavakoli, like many others, has revised her thinking sharply in the intervening years.

Yet she, like so many others, added an important caveat about Buffett: (His) brand is so powerful you are reluctant to question.

SOKOL AFFAIR

By now the details of Sokol affair have been told many times. Citigroup bankers pitched a long list of companies to Buffett's presumed successor, and he told them he thought Lubrizol Corp, which makes lubricants and other chemicals, might make a good acquisition target. He started buying up shares for his own account, and after building up a $10 million position he pushed Buffett to buy the company.

As Buffett put it, Sokol made only a passing mention that he owned some Lubrizol shares.

Sokol made about $3 million on the trade, perhaps at Buffett's expense. Buffett has been called to task for how he handled the matter. In a letter to investors, he announced Sokol's resignation, explained the stock issue and offered a grant of absolution: Neither Dave nor I feel his Lubrizol purchases were in any way unlawful, he wrote.

Less than three weeks later, the first shareholder suit was filed, accusing Berkshire's board of breaching its fiduciary responsibility. More are expected, particularly from bigger firms with a track record of winning large settlements for shareholders.

Governance experts say Buffett blamed the sin but not the sinner.

The response wasn't as strident as ... I would have hoped for in suggesting that personal stock transactions that are related to corporate stock transactions are problematic and not the sort of thing that the company thinks is a good idea, said Charles Elson, director of the Weinberg Center for Corporate Governance at the University of Delaware.

And I would hope in these situations that you would be pretty tough on that in your response.

Some of Buffett's biggest investors also say he should have chastised Sokol or told him to sell his stock. What is murkier, however, is the question of whether Buffett actually did anything wrong from a legal standpoint.

There's a lot of very problematic behavior here that doesn't easily find an explanation, so the question remains, what in fact was going on here? said Harvey Pitt, chief executive of Kalorama Partners and the former chairman of the U.S. Securities and Exchange Commission.

Why would somebody be allowed and be deemed to have acted properly in profiting to the tune of $3 million based on his privileged position at the company?, Pitt added.

It wasn't the first time that Buffett has been close to people behaving questionably. But few of his investors have cared, and the damage to his reputation seemed slight if at all.

In 2008, for example, the government won convictions of four executives from his reinsurance business for helping other insurers inflate their results. The nearly uniform reaction from legions of Buffett fans around the world: yawn.

And in 2005, the SEC sanctioned the Coca-Cola Co, whose audit committee Buffett sat on, for inflating earnings. His admirers barely batted an eyelash.

'THAT'S MY GUY'

Buffett, of course, benefits mightily from his folksy image. After all, it's tough to imagine how someone who drives himself to work and stops at McDonald's for a bite on the way home can also be guilty of high crimes of finance.

Warren Buffett works very hard reflecting an image that 300 million Americans, six billion people around the world say, 'That's my guy. That's the way I'd like to be like.' And he works very hard at that -- not every week or every month, but every day. And I think by working hard at it every day, he drives that image hard into people's minds, said Robert Dilenschneider, a public relations executive who heads the Dilenschneider Group in New York.

The audience at the annual meeting is one of the tools he uses to burnish his reputation. There is no better financial television than footage of Buffett having an ice cream at (Berkshire-owned) Dairy Queen, with hordes of investors thronging him and hoping he might drop a stock tip on the floor with the crumbs of his vanilla cone.

It is hard to interrupt that storyline.

With the cash that he was able to squeeze out of that dying textile business (Berkshire Hathaway) and astutely reallocating it year after year after year, the business grew from $18 a share to $120,000 a share, said Roger Lowenstein, author of a well-regarded 1995 Buffett biography and a number of other finance books.

I have no doubt that he'll be regarded as the investor and probably the financier of the era. This incident sort of tells people that he's human.

PERFORMANCE UNDER FIRE

While many would agree with Lowenstein on Buffett's place in financial history, his returns of late have not necessarily matched his reputation.

Berkshire shares have only barely matched the S&P 500 since September 2008, the depths of the crisis and the time Buffett made some of his most lucrative bets, like buying Goldman preferred shares that threw off more than $15 a second in dividends.

Just this year, Berkshire has underperformed the S&P 500 by about 4 percent. Buffett has said returns will slow, so it does not necessarily come as a surprise.

There is expected to be less reluctance to question him this year in Omaha. Author Tavakoli said shareholder dissatisfaction was already palpable at the last meeting she attended in 2009, as Buffett went on about his bet on Wells Fargo and investors grumbled that he was not talking about the crony capitalism they saw behind the crisis-era bailouts.

It seems as if Warren Buffett has sort of lost touch with the tone of the people who invest in Berkshire Hathaway and their sentiment, she said.

The Q&A session will again be moderated by financial journalists this year, so even if investors don't ask tough questions, reporters may. Words like contentious and even raucous are being thrown around. And yet, some investors still do not expect much.

I don't expect any great revelations but what I want is not necessarily what I'm going to get, said YCMNET's Yoshikami.

Yet he still expects the annual meeting to be a lovefest, given the overwhelming number of shareholders who flock to Omaha annually for nothing more than pearls of Buffett's wisdom (and perhaps some discount pearls from his jewelry business, one of a number of Buffett companies to offer steep shareholder discounts over the course of the weekend).

Yoshikami said that if Buffett gets away from the Sokol episode unscathed, it will be because he has banked sufficient goodwill with investors in the past.

When you have an inventory of transparency that you can fall back on I think you get the benefit of the doubt, he said. I think when you self-disclose enough and you have a reputation for self disclosing it buys you some reputation credits.

But no matter what Buffett says or does in Omaha, there is a growing realization that the old days have slipped by. Buffett and his partner, Charlie Munger, are aging, the questions about the future of the conglomerate are getting louder and people are recognizing, as they do, that all good things have to come to an end.

The passage of time is hitting home. This year is the end of Berkshire as it used to be, said Alice Schroeder, a former stock analyst who wrote what many view as the definitive biography of Buffett.

It will never be the same. Even if people think Buffett's not going to address all these issues and the questions won't be as tough as they should be, Berkshire as it used to be is over, Schroeder said.

(Additional reporting by Brett Gering of Reuters Insider; Editing by Jim Impoco)