How to web scrape Google search results without a scraper or any other software (for free, of course!)

Every now and then, every one of us gets into a situation where we need to scrape the URLs from the Google search results, put them into a spreadsheet and process them in some way.

May it be for keyword statistics, to prepare lists of competitors for further analysis in Ahrefs, Semrush or other professional tools, or for virtually any other purpose.

If you are a happy owner of a top SEO agency and you can afford it, there are many professional-grade scraper softwares you can pay to do the job for you. However, if you’re just a curious guy doing a random survey, you don’t have to spend your wife’s income on an instrument that you will use once or twice in a life.

To dispel any doubts and to clarify what we will need to achieve this web scraping task and what we will not need, here is a list:

To scrape Google search results we won’t need:

• web scraper software;
• proxies;
• programming experience.

To get it we will only need:

• Google (obviously);
• Two freeware plugins for Google Chrome browser.

This method brings limited results (maximum around 300 URLs per scrape), however, that is more tha enough, in most cases.

Let’s start with the Chrome plugins

Go to Google Chrome extensions repository and download these two awsome plugins:

• Google Infinity;
• Link Grabber;

Then pin them on the top bar so that you can launch them with just one click. They are the two highlighted icons.

Google Infinity allows you to list all search results on one page and the Link Grabber does what it’s name foreshadows – it grabs all the links from a page.

That means that we have a plugin combo capable of listing all URLs on one page and grabbing that URLs at once. Perfect, isn’t it?

Let’s go do some advanced Google search settings

In order to do everything as efficiently as possible, we need to change a few default Google Search settings.

We will change the number of displayed results per page to the allowed maximum, which is 100 items/page.

And I strongly suggest that you also change the preferences and turn off the usage of “private results”.

Why? Google by default shows you personalised results which are biased and subjective.

Google algorhitms present you these results based on your historical activity.

Your search history, social life networks, interactions with your friends and foes and what not, it all modulates your search results and if you want to objectivize your results as much as possible, you should just turn them off.

The search results of the other people will be modulated by their own historical activities, but that is a fact that we just can’t influence, so we will abstract from it.

Scraping the search results as a PRO

Launch Google and search whatever you want (I searched for “google loves me” :-)).

Scroll down and after 100 results you will experience a fancy thing, second page with other 100 results will be loaded, thanks to our beloved Ginfinity plugin.

Now we will grab the links but before we use the Link Grabber plugin for the first time I suggest that you change the settings/options this way.

These settings will enable us to remove by default all URLs that we are not interested in so that we will not have to filter them out later.

And we are done…

Grab the links and do with them whatever you want.

Stay open-minded, gentlemen. And tuned in too :-).

Tomorrow I will show you another trick. Something with keywords.