How can advisers keep their millionaire clients happy? By returning their phone calls.

The failure to return clients' telephone calls quickly is the No. 1 reason most millionaires change advisers, according to a new survey released on Tuesday by consulting firm Spectrem Group.

Almost three-quarters of the nearly 1,000 millionaires surveyed expect their financial advisers to return phone calls within 12 hours, while 40 percent expect a call back within two hours. Respondents had investable assets of between $1 million and $5 million.

Nearly three-quarters of millionaires surveyed said a failure to return calls quickly was their top reason for changing advisers, while 57 percent cited not providing good ideas and advice.

Advisers have a little more leeway with emails. More than half of the respondents said they expected their adviser to reply to an email within 12 hours, while almost a quarter said they wanted a response within two hours.

Advisers who don't respond promptly to their millionaire clients' calls are playing with fire, said Spectrem Group President George H. Walper Jr. It's a very simple (thing) to address.

(Reporting by Helen Kearney, editing by Maureen Bavdek and Lisa Von Ahn)