Do they promise to guarantee #1 ranking? If they do, you have a serious reason to doubt their competencies. As the Google SEO selection tips say, no one can guarantee a #1 ranking in Google. This is true even for not so competitive words.
Get recommendation from friends, business partners, etc. Word of mouth is very important for the credibility of a company.
Google the company name. If the company is a known fraudster, chances are that you will find a lot of information about it on the Web. However, lack of negative publicity does not mean automatically that the company is great, nor do some subjective negative opinions mean that the company is a scammer.
Ask for examples of sites they have optimized. Happy customers are the best form of promotion, so feel free to ask your potential SEO company about sites they have optimized and references from clients. If you get a rejection because of confidentiality reasons, this must ring a bell about the credibility of the SEO company – former customers are not supposed to be a secret.
Check the PR of their own site. If they can’t optimize their site well enough to get a good PR (over 4-5), they are not worth hiring.
Ask them what keywords their site ranks for. Similarly to the page rank factor, if they don’t rank well for the keywords of their choice, they are hardly as professional as they are pretending to be.
Do they use automated submissions? If they do, stay away from them. Automated submissions can get you banned from search engines.
Do they use any black hat SEO tricks? You need to know in advance what black hat SEO is in order to judge them, so getting familiar with the most important black hat SEO tricks is worth before you go and start cross-examining them.
Where do they collect backlinks from? Backlinks are very, very important for SEO success but if they come from link farms and other similar sites, this can cause a lot of trouble. So, make sure the SEO firm collects links from reputable sites only.
Get some personal impressions, if possible. Gut instinct and impressions from meetings are also a way to judge a company, though sometimes it is not difficult to get mislead, so use this approach with caution.
High price does not guarantee high quality. If you are eager to pay more, this does not mean that you will get more. Just because a firm costs more DOES NOT make them better SEO’s. There are many reasons for high prices and high quality is only one of them. For instance, the company might work inefficiently and this is the reason for their ridiculously high costs, not the quality of their work.
Anyway if you work on SEO you can see results of your work but if you choose a SEO company you can lose your money and won’t get any results. Good luck!