Especially for the performance of your website, choosing the right host for your website alone can already make quite a difference. But also ease of maintenance, support and even SEO are factors to consider. Most important is to choose the right provider. A good one will even have proper support in their cheaper packages, and upgrading should not be an issue when the need arises.
Basically I would advise you to stay away from these. I know some webdesigners have good experiences, but basically there is no such thing as a free lunch: Sooner or later you will find that banners will be displayed next to your site, support is none-existent, upgrading to a higher package is expensive or moving away to another host is made difficult.
When making your choice, check the following, apart from their rates of course:
Finally, check user reviews for the host you have in mind, especially if you expect specific wishes.
Now that you have chosen a hosting company, you should also look at which package you select. Especially if you want to aim high with your website, you may want to stay away from shared packages. When you are on a shared server, you share the same IP (and resources). Especially if you share the same IP as an insecure or malicious site, this could potentially be bad for your online reputation, and in some cases, even for your ranking.
This also depends on how close your host monitors their sites. For 80-90% of websites, being on a shared server should not really be an issue.
Happily there are quite a few companies that offer excellent hosting and comply to the requirements mentioned here. Personally, I only have experience with Siteground.com and they are pretty good, but there are many more that are probably as good.
Joomlaseo.com is hosted on Siteground.com, and I have good experiences with them. They focus on performance and security and know Joomla pretty well (as well as WordPress, Drupal, etc.), so I don't mind recommending them as a choice. I remember that when I moved my first site to them from my previous host, performance immediately went up without additional changes (faster speeds, higher grades in Pagespeed). Especially for performance they have some nice great options, like advanced caching that you can activate on top of the Joomla options, and free CDN.
Joomlaseo.com is fully built and written by Simon Kloostra, SEO Specialist and Webdesigner from the Netherlands. I have also published the Joomla 3 SEO & Performance SEO book. Next to that I also sometimes blog for companies like OStraining, TemplateMonster, SEMrush and others. On the monthly Joomla Community Magazine I have also published a few articles.