| gl | searchTerms | rank | title | snippet | displayLink | link | queryTime | totalResults | cacheId | count | cseName | cx | fileFormat | formattedSearchTime | formattedTotalResults | formattedUrl | htmlFormattedUrl | htmlSnippet | htmlTitle | inputEncoding | kind | mime | outputEncoding | pagemap | safe | searchTime | startIndex | cse_thumbnail | metatags | cse_image | thumbnail | listitem | table | organization | postaladdress | hcard | imageobject | person | interactioncounter | webpage | blogposting | hatomfeed | product | aggregaterating | hproduct | videoobject | collection | creativework | socialmediaposting | softwaresourcecode | comment | article | hreviewaggregate | sitenavigationelement | newsarticle | wpheader | term-def.xml | speakablespecification | maincontentofpage | question | answer | WebPage | corporation | contactpoint | ngo | BreadcrumbList | rating | offer | brand | Document | document | xfn | propertyvalue | review | individualproduct | breadcrumb | webapplication | scraped | og:image | viewport | twitter:card | msapplication-square70x70logo | sailthru.tags | og:site_name | emailt1 | msapplication-wide310x150logo | msapplication-tileimage | og:description | twitter:image | twitter:site | msapplication-square310x310logo | sailthru.title | emailcontenttype | msapplication-tilecolor | sailthru.date | og:type | twitter:title | emailvertical | og:title | msapplication-square150x150logo | sailthru.image.thumb | fb:app_id | twitter:description | og:url | sailthru.author | p:domain_verify | metatag_thumbnail | author | twitter:creator | news_keywords | og:locale | twitter:url | fb:admins | twitter:app:id:googleplay | twitter:app:id:ipad | twitter:app:id:iphone | referrer | format-detection | og:image:width | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | us | what is bitcoin | 1 | Bitcoin - Open source P2P money | Bitcoin uses peer-to-peer technology to operat... | bitcoin.org | https://bitcoin.org/ | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | eftmar-x2ocJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://bitcoin.org/ | https://<b>bitcoin</b>.org/ | <b>Bitcoin</b> uses peer-to-peer technology to... | <b>Bitcoin</b> - Open source P2P money | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn1.gstatic.com/i... | [{'og:image': 'https://bitcoin.org/img/icons/o... | [{'src': 'https://bitcoin.org/img/icons/opengr... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://bitcoin.org/img/icons/opengraph.png?16... | width=device-width, initial-scale=1.0, user-sc... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | us | what is bitcoin | 2 | Bitcoin Definition | Bitcoin is a digital or virtual currency creat... | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | q3G-51w_tGYJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.investopedia.com/terms/b/bitcoin.asp | https://www.investopedia.com/terms/b/<b>bitcoi... | <b>Bitcoin</b> is a digital or virtual currenc... | <b>Bitcoin</b> Definition | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn2.gstatic.com/i... | [{'og:image': 'https://www.investopedia.com/th... | [{'src': 'https://www.investopedia.com/thmb/wj... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://www.investopedia.com/thmb/wjXQJxSVbGQ5... | width=device-width, initial-scale=1.0 | summary | /static/1.229.0/icons/favicons/mstile-70x70.png | investing,cryptocurrency,bitcoin | Investopedia | Investing | /static/1.229.0/icons/favicons/mstile-310x150.png | /static/1.229.0/icons/favicons/mstile-144x144.png | Bitcoin is a digital or virtual currency creat... | https://www.investopedia.com/thmb/wjXQJxSVbGQ5... | @Investopedia | /static/1.229.0/icons/favicons/mstile-310x310.png | Bitcoin Definition: How Does Bitcoin Work? | TERMSC | #F4F4F4 | 2022-02-20T12:34:05.191Z | article | Bitcoin Definition: How Does Bitcoin Work? | INVESTOPEDIA | Bitcoin Definition: How Does Bitcoin Work? | /static/1.229.0/icons/favicons/mstile-150x150.png | https://www.investopedia.com/thmb/TQvY4XfgB7Za... | 6.590541e+14 | Bitcoin is a digital or virtual currency creat... | https://www.investopedia.com/terms/b/bitcoin.asp | Jake Frankenfield | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | us | what is bitcoin | 3 | What is bitcoin and how does it work? | New Sc... | Bitcoin is a digital currency which operates f... | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | EembwztvXSwJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.newscientist.com/definition/bitcoin/ | https://www.newscientist.com/definition/<b>bit... | <b>Bitcoin</b> is a digital currency which ope... | <b>What is bitcoin</b> and how does it work? |... | utf8 | customsearch#result | NaN | utf8 | {'thumbnail': [{'src': 'https://images.newscie... | off | 0.309795 | 1 | NaN | [{'p:domain_verify': 'e1ad6b074bd56738db29f7d0... | [{'src': 'https://images.newscientist.com/wp-c... | [{'src': 'https://images.newscientist.com/wp-c... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://images.newscientist.com/wp-content/upl... | width=device-width, initial-scale=1.0, maximum... | summary_large_image | NaN | NaN | New Scientist | NaN | NaN | NaN | Bitcoin is a digital currency which operates f... | https://images.newscientist.com/wp-content/upl... | @newscientist | NaN | NaN | NaN | NaN | NaN | article | What is bitcoin and how does it work? | New Sc... | NaN | What is bitcoin and how does it work? | NaN | NaN | NaN | Bitcoin is a digital currency which operates f... | https://www.newscientist.com/definition/bitcoin/ | NaN | e1ad6b074bd56738db29f7d0150698c9 | https://images.newscientist.com/wp-content/upl... | #author.fullName} | @newscientist | cryptocurrency,bitcoin | en_US | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | us | what is bitcoin | 4 | What is bitcoin? - CNNMoney | Bitcoin is a cryptocurrency created in 2009. M... | money.cnn.com | https://money.cnn.com/infographic/technology/w... | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | ejebbzzk3kEJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://money.cnn.com/infographic/technology/w... | https://money.cnn.com/infographic/technology/<... | <b>Bitcoin</b> is a <b>cryptocurrency</b> crea... | <b>What is bitcoin</b>? - CNNMoney | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn1.gstatic.com/i... | [{'og:image': 'https://cdn.cnn.com/cnn/2018/im... | [{'src': 'https://cdn.cnn.com/cnn/2018/images/... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://cdn.cnn.com/cnn/2018/images/08/08/cnni... | width=device-width, initial-scale=1.0, maximum... | summary_large_image | NaN | NaN | CNNMoney | NaN | NaN | NaN | Bitcoin is a cryptocurrency created in 2009. M... | https://cdn.cnn.com/cnn/2018/images/08/08/cnni... | @CNNMoney | NaN | NaN | NaN | NaN | NaN | article | What is bitcoin? - CNNMoney | NaN | What is bitcoin? - CNNMoney | NaN | NaN | 5.218482e+14 | Bitcoin is a cryptocurrency created in 2009. M... | https://money.cnn.com/infographic/technology/w... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://money.cnn.com/infographic/technology/w... | 100000014300691 | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | us | what is bitcoin | 5 | What is Bitcoin? | Coinbase | Bitcoin is a currency native to the Internet. ... | www.coinbase.com | https://www.coinbase.com/learn/crypto-basics/w... | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | 6QRw_cV35OkJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.coinbase.com/learn/crypto-basics/w... | https://www.coinbase.com/learn/<b>crypto</b>-b... | <b>Bitcoin</b> is a currency native to the Int... | <b>What is Bitcoin</b>? | Coinbase | utf8 | customsearch#result | NaN | utf8 | {'metatags': [{'twitter:app:id:googleplay': 'c... | off | 0.309795 | 1 | NaN | [{'twitter:app:id:googleplay': 'com.coinbase.a... | NaN | NaN | [{'item': 'Learn', 'name': 'Learn', 'position'... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | width=device-width, initial-scale=1.0 | summary_large_image | NaN | NaN | NaN | NaN | NaN | NaN | Bitcoin is the world's first widely adopted cr... | NaN | @coinbase | NaN | NaN | NaN | NaN | NaN | website | What is Bitcoin? | NaN | What is Bitcoin? | NaN | NaN | NaN | Bitcoin is the world's first widely adopted cr... | NaN | NaN | NaN | NaN | NaN | @coinbase | NaN | NaN | NaN | NaN | com.coinbase.android | 886427730.0 | 886427730.0 | NaN | NaN | NaN |
SERP Heatmap
Using multiple SERP datasets, create a heatmap for each domain, showing how often it appeared on each position of the SERPs, with some summary statistics.
serp_heatmap
serp_heatmap (serp_df, num_domains=10, height=650, width=None, title='SERP Heatmap', theme='none')
Create a heatmap for visualizing domain positions on SERPs.
| Type | Default | Details | |
|---|---|---|---|
| serp_df | pandas.DataFrame | A DataFrame retreived from running advertools.serp_goog or advertools.serp_youtube. Any similar SERP DataFrame would do, with the simple requirement of containing three columns: searchTerms, displayLink, and rank. |
|
| num_domains | int | 10 | The number of domains to display in the chart, default 10. |
| height | int | 650 | The height in pixels of the chart. |
| width | NoneType | None | The width in pixels of the chart. |
| title | str | SERP Heatmap | The title of the chart. |
| theme | str | none | The theme to apply to the chart. |
| Returns | plotly.graph_objects.Figure |
Exploring the SERP dataset
This table is the result of using the advertools.serp_goog function, which connects to the official Google Custom Search API. You can use other SERP data providers if you want.
Watch a discussion about analyzing SERPs:
This SERP dataset was created by running the adv.serp_goog function as follows:
- Get the top 36 crypto currencies’ names
- Generate two variations for each coin: “what is <coin>” and “<coin> price”
- Run the
serp_googfunction for all coins, query variations and in two countries, US, and UK
36 coins x 2 variations x 2 countries = 144 requests (1,440 SERP results)
As you can see, there are numerous columns that are returned by the API, although many of them are empty. The default ones will always be full though, and they are placed in the beginning of the DataFrame.
Sample results for the “what is <coin>” query
| gl | searchTerms | rank | displayLink | link | snippet | |
|---|---|---|---|---|---|---|
| 0 | us | what is bitcoin | 1 | bitcoin.org | https://bitcoin.org/ | Bitcoin uses peer-to-peer technology to operat... |
| 1 | us | what is bitcoin | 2 | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | Bitcoin is a digital or virtual currency creat... |
| 2 | us | what is bitcoin | 3 | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | Bitcoin is a digital currency which operates f... |
| 10 | uk | what is bitcoin | 1 | bitcoin.org | https://bitcoin.org/ | Bitcoin uses peer-to-peer technology to operat... |
| 11 | uk | what is bitcoin | 2 | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | Bitcoin is a digital currency which operates f... |
| 12 | uk | what is bitcoin | 3 | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | Bitcoin is a digital or virtual currency creat... |
| 20 | us | what is litecoin | 1 | litecoin.org | https://litecoin.org/ | What is Litecoin? ... Litecoin is a peer-to-pe... |
| 21 | us | what is litecoin | 2 | www.investopedia.com | https://www.investopedia.com/articles/investin... | Litecoin (LTC) is a cryptocurrency created fro... |
| 22 | us | what is litecoin | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 30 | uk | what is litecoin | 1 | www.investopedia.com | https://www.investopedia.com/articles/investin... | Litecoin (LTC) is a cryptocurrency created fro... |
| 31 | uk | what is litecoin | 2 | litecoin.org | https://litecoin.org/ | What is Litecoin? ... Litecoin is a peer-to-pe... |
| 32 | uk | what is litecoin | 3 | www.investopedia.com | https://www.investopedia.com/articles/investin... | 18 Dec 2021 ... An application-specific integr... |
| 40 | us | what is namecoin | 1 | www.namecoin.org | https://www.namecoin.org/ | Namecoin is an experimental open-source techno... |
| 41 | us | what is namecoin | 2 | www.investopedia.com | https://www.investopedia.com/terms/n/namecoin.asp | Mar 1, 2021 ... Namecoin is a key/value pair r... |
| 42 | us | what is namecoin | 3 | en.wikipedia.org | https://en.wikipedia.org/wiki/Namecoin | or NMC) is a cryptocurrency originally forked ... |
Sample results for the “<coin> price” query
| gl | searchTerms | rank | displayLink | link | snippet | |
|---|---|---|---|---|---|---|
| 720 | us | bitcoin price | 1 | www.coindesk.com | https://www.coindesk.com/price/bitcoin/ | The Bitcoin price is $39,934.91, a change of -... |
| 721 | us | bitcoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/bitcoin | April 23, 2022 - The current price of Bitcoin ... |
| 722 | us | bitcoin price | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/bitcoin/ | The live Bitcoin price today is $39,717.61 USD... |
| 730 | uk | bitcoin price | 1 | www.coindesk.com | https://www.coindesk.com/price/bitcoin/ | The Bitcoin price is $39,934.91, a change of -... |
| 731 | uk | bitcoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/bitcoin | April 23, 2022 - The current price of Bitcoin ... |
| 732 | uk | bitcoin price | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/bitcoin/ | The live Bitcoin price today is $39,606.70 USD... |
| 740 | us | litecoin price | 1 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 741 | us | litecoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/litecoin | The price of Litecoin has fallen by 7.35% in t... |
| 742 | us | litecoin price | 3 | www.coindesk.com | https://www.coindesk.com/price/litecoin/ | Litecoin Price ; 24H Open. $114.00 ; 24H Chang... |
| 750 | uk | litecoin price | 1 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 751 | uk | litecoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/litecoin | The price of Litecoin has fallen by 7.74% in t... |
| 752 | uk | litecoin price | 3 | www.coindesk.com | https://www.coindesk.com/price/litecoin/ | Litecoin Price ; 24H Open. $114.00 ; 24H Chang... |
| 760 | us | namecoin price | 1 | www.namecoin.org | https://www.namecoin.org/ | Namecoin. Namecoin is an experimental open-sou... |
| 761 | us | namecoin price | 2 | coinmarketcap.com | https://coinmarketcap.com/currencies/namecoin/ | The live Namecoin price today is $2.12 USD wit... |
| 762 | us | namecoin price | 3 | www.coinbase.com | https://www.coinbase.com/price/namecoin | April 23, 2022 - The current price of Namecoin... |
We can easily create our hetmap with one command:
Modify some of the default options
serp_heatmap(
serp,
num_domains=15,
title='SERP Heatmap - crypto currencies<br><b>"what is <coin>"</b> and <b>"<coin> price"</b>',
theme='ggplot2')Analysis - splitting the SERP DataFrame for a more meaningful subset
what_is_df = serp[serp['searchTerms'].str.contains('what is')]
serp_heatmap(what_is_df, title='SERP Heatmap<br><b>"what is <coin>"</b> keywords')price_df = serp[serp['searchTerms'].str.contains('price')]
serp_heatmap(price_df, title='SERP Heatmap<br><b>"<coin> price"</b> keywords')It’s clear how different the dominant domains are when split by keyword variant, in this case the “what is <coin>” keywords vs. the “coin <price>” keywords
Add shapes and annotations for better guiding your users
- Add a rectangle to highight one or more domains
- Make the rectangle colored, or simply color the edges of the rectangle
- Optionally add a label to inform users about domain(s) in this rectangle
fig = serp_heatmap(
# "price" keywords in the USA:
serp[serp['gl'].eq('us') & serp['searchTerms'].str.contains('price')],
theme='plotly_white',
title='SERP Heatmap <b>"<coin> price"</b> keywords – USA',
num_domains=13)
fig.add_shape(
type='rect', x0=.5, x1=1.5, y0=-0.5, y1=10.5,
fillcolor='gray', opacity=0.15)
fig.add_shape(
type='rect', x0=4.5, x1=6.5, y0=-0.5, y1=11.5,
line={'color': 'darkgreen'},
label={
'text': 'This is a label',
'font': {'family':'Arial Black'},
'textposition': 'bottom center'
})
figSERP Heatmap chart details
- Avg. Pos.: The average position of the domain shown on this column
- Coverage: The number of appearances of the domain on the SERP, divided by the total queries. A ratio above 100% can exist, where some domains appear more than once in the same SERP.
- Total appearances: the number of times the domain appeared in the SERP (from this dataset).
- Numbers in circles: Show the number of times each domain appeared in its respective position.
Data from other sources
This chart doest not need the data from Google custom search API exclusively. There are other similar data sources like YouTube, SERP API, and SearchAPI for example. The only requirement is that you have a DataFrame with “rank”, “searchTerms”, and “displayLink”. These columns might be named differently, in which case you only need to rename those columns. You might have “searchTerms” named as “keyword” or “query” for example, or “displayLink” named “domain”. This should be straightforward.