| gl | searchTerms | rank | title | snippet | displayLink | link | queryTime | totalResults | cacheId | count | cseName | cx | fileFormat | formattedSearchTime | formattedTotalResults | formattedUrl | htmlFormattedUrl | htmlSnippet | htmlTitle | inputEncoding | kind | mime | outputEncoding | pagemap | safe | searchTime | startIndex | cse_thumbnail | metatags | cse_image | thumbnail | listitem | table | organization | postaladdress | hcard | imageobject | person | interactioncounter | webpage | blogposting | hatomfeed | product | aggregaterating | hproduct | videoobject | collection | creativework | socialmediaposting | softwaresourcecode | comment | article | hreviewaggregate | sitenavigationelement | newsarticle | wpheader | term-def.xml | speakablespecification | maincontentofpage | question | answer | WebPage | corporation | contactpoint | ngo | BreadcrumbList | rating | offer | brand | Document | document | xfn | propertyvalue | review | individualproduct | breadcrumb | webapplication | scraped | og:image | viewport | twitter:card | msapplication-square70x70logo | sailthru.tags | og:site_name | emailt1 | msapplication-wide310x150logo | msapplication-tileimage | og:description | twitter:image | twitter:site | msapplication-square310x310logo | sailthru.title | emailcontenttype | msapplication-tilecolor | sailthru.date | og:type | twitter:title | emailvertical | og:title | msapplication-square150x150logo | sailthru.image.thumb | fb:app_id | twitter:description | og:url | sailthru.author | p:domain_verify | metatag_thumbnail | author | twitter:creator | news_keywords | og:locale | twitter:url | fb:admins | twitter:app:id:googleplay | twitter:app:id:ipad | twitter:app:id:iphone | referrer | format-detection | og:image:width | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | us | what is bitcoin | 1 | Bitcoin - Open source P2P money | Bitcoin uses peer-to-peer technology to operat... | bitcoin.org | https://bitcoin.org/ | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | eftmar-x2ocJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://bitcoin.org/ | https://<b>bitcoin</b>.org/ | <b>Bitcoin</b> uses peer-to-peer technology to... | <b>Bitcoin</b> - Open source P2P money | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn1.gstatic.com/i... | [{'og:image': 'https://bitcoin.org/img/icons/o... | [{'src': 'https://bitcoin.org/img/icons/opengr... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://bitcoin.org/img/icons/opengraph.png?16... | width=device-width, initial-scale=1.0, user-sc... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | us | what is bitcoin | 2 | Bitcoin Definition | Bitcoin is a digital or virtual currency creat... | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | q3G-51w_tGYJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.investopedia.com/terms/b/bitcoin.asp | https://www.investopedia.com/terms/b/<b>bitcoi... | <b>Bitcoin</b> is a digital or virtual currenc... | <b>Bitcoin</b> Definition | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn2.gstatic.com/i... | [{'og:image': 'https://www.investopedia.com/th... | [{'src': 'https://www.investopedia.com/thmb/wj... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://www.investopedia.com/thmb/wjXQJxSVbGQ5... | width=device-width, initial-scale=1.0 | summary | /static/1.229.0/icons/favicons/mstile-70x70.png | investing,cryptocurrency,bitcoin | Investopedia | Investing | /static/1.229.0/icons/favicons/mstile-310x150.png | /static/1.229.0/icons/favicons/mstile-144x144.png | Bitcoin is a digital or virtual currency creat... | https://www.investopedia.com/thmb/wjXQJxSVbGQ5... | @Investopedia | /static/1.229.0/icons/favicons/mstile-310x310.png | Bitcoin Definition: How Does Bitcoin Work? | TERMSC | #F4F4F4 | 2022-02-20T12:34:05.191Z | article | Bitcoin Definition: How Does Bitcoin Work? | INVESTOPEDIA | Bitcoin Definition: How Does Bitcoin Work? | /static/1.229.0/icons/favicons/mstile-150x150.png | https://www.investopedia.com/thmb/TQvY4XfgB7Za... | 6.590541e+14 | Bitcoin is a digital or virtual currency creat... | https://www.investopedia.com/terms/b/bitcoin.asp | Jake Frankenfield | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | us | what is bitcoin | 3 | What is bitcoin and how does it work? | New Sc... | Bitcoin is a digital currency which operates f... | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | EembwztvXSwJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.newscientist.com/definition/bitcoin/ | https://www.newscientist.com/definition/<b>bit... | <b>Bitcoin</b> is a digital currency which ope... | <b>What is bitcoin</b> and how does it work? |... | utf8 | customsearch#result | NaN | utf8 | {'thumbnail': [{'src': 'https://images.newscie... | off | 0.309795 | 1 | NaN | [{'p:domain_verify': 'e1ad6b074bd56738db29f7d0... | [{'src': 'https://images.newscientist.com/wp-c... | [{'src': 'https://images.newscientist.com/wp-c... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://images.newscientist.com/wp-content/upl... | width=device-width, initial-scale=1.0, maximum... | summary_large_image | NaN | NaN | New Scientist | NaN | NaN | NaN | Bitcoin is a digital currency which operates f... | https://images.newscientist.com/wp-content/upl... | @newscientist | NaN | NaN | NaN | NaN | NaN | article | What is bitcoin and how does it work? | New Sc... | NaN | What is bitcoin and how does it work? | NaN | NaN | NaN | Bitcoin is a digital currency which operates f... | https://www.newscientist.com/definition/bitcoin/ | NaN | e1ad6b074bd56738db29f7d0150698c9 | https://images.newscientist.com/wp-content/upl... | #author.fullName} | @newscientist | cryptocurrency,bitcoin | en_US | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | us | what is bitcoin | 4 | What is bitcoin? - CNNMoney | Bitcoin is a cryptocurrency created in 2009. M... | money.cnn.com | https://money.cnn.com/infographic/technology/w... | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | ejebbzzk3kEJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://money.cnn.com/infographic/technology/w... | https://money.cnn.com/infographic/technology/<... | <b>Bitcoin</b> is a <b>cryptocurrency</b> crea... | <b>What is bitcoin</b>? - CNNMoney | utf8 | customsearch#result | NaN | utf8 | {'cse_thumbnail': [{'src': 'https://encrypted-... | off | 0.309795 | 1 | [{'src': 'https://encrypted-tbn1.gstatic.com/i... | [{'og:image': 'https://cdn.cnn.com/cnn/2018/im... | [{'src': 'https://cdn.cnn.com/cnn/2018/images/... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://cdn.cnn.com/cnn/2018/images/08/08/cnni... | width=device-width, initial-scale=1.0, maximum... | summary_large_image | NaN | NaN | CNNMoney | NaN | NaN | NaN | Bitcoin is a cryptocurrency created in 2009. M... | https://cdn.cnn.com/cnn/2018/images/08/08/cnni... | @CNNMoney | NaN | NaN | NaN | NaN | NaN | article | What is bitcoin? - CNNMoney | NaN | What is bitcoin? - CNNMoney | NaN | NaN | 5.218482e+14 | Bitcoin is a cryptocurrency created in 2009. M... | https://money.cnn.com/infographic/technology/w... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | https://money.cnn.com/infographic/technology/w... | 100000014300691 | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | us | what is bitcoin | 5 | What is Bitcoin? | Coinbase | Bitcoin is a currency native to the Internet. ... | www.coinbase.com | https://www.coinbase.com/learn/crypto-basics/w... | 2022-04-24 18:10:18.884769+00:00 | 8100000000 | 6QRw_cV35OkJ | 10 | PySearch | 012859022920491477448:pubdbfjmmec | NaN | 0.31 | 8,100,000,000 | https://www.coinbase.com/learn/crypto-basics/w... | https://www.coinbase.com/learn/<b>crypto</b>-b... | <b>Bitcoin</b> is a currency native to the Int... | <b>What is Bitcoin</b>? | Coinbase | utf8 | customsearch#result | NaN | utf8 | {'metatags': [{'twitter:app:id:googleplay': 'c... | off | 0.309795 | 1 | NaN | [{'twitter:app:id:googleplay': 'com.coinbase.a... | NaN | NaN | [{'item': 'Learn', 'name': 'Learn', 'position'... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | width=device-width, initial-scale=1.0 | summary_large_image | NaN | NaN | NaN | NaN | NaN | NaN | Bitcoin is the world's first widely adopted cr... | NaN | @coinbase | NaN | NaN | NaN | NaN | NaN | website | What is Bitcoin? | NaN | What is Bitcoin? | NaN | NaN | NaN | Bitcoin is the world's first widely adopted cr... | NaN | NaN | NaN | NaN | NaN | @coinbase | NaN | NaN | NaN | NaN | com.coinbase.android | 886427730.0 | 886427730.0 | NaN | NaN | NaN |
SERP Heatmap
Using multiple SERP datasets, create a heatmap for each domain, showing how often it appeared on each position of the SERPs, with some summary statistics.
serp_heatmap
Create a heatmap for visualizing domain positions on SERPs.
| Type | Default | Details | |
|---|---|---|---|
| serp_df | Tidy SERP data: one row per query, domain, rank. | ||
| queries_col | Column holding the search queries. | ||
| domains_col | Column holding the domains. | ||
| ranks_col | Column holding the rank/position. | ||
| num_domains | int | 10 | Number of domains to display. |
| height | int | 650 | Chart height in pixels (per row when faceting). |
| width | NoneType | None | Chart width in pixels. |
| title | str | SERP Heatmap | Chart title. |
| subtitle | NoneType | None | Chart subtitle. |
| template | str | none | Plotly template. |
| facet_row | NoneType | None | Column to split into one row per value. |
| facet_col | NoneType | None | Column to split into one column per value. |
| kwargs | |||
| Returns | Figure | The heatmap figure. |
Exploring the SERP dataset
This table is the result of using the advertools.serp_goog function, which connects to the official Google Custom Search API. You can use other SERP data providers if you want.
Watch a discussion about analyzing SERPs:
This SERP dataset was created by running the adv.serp_goog function as follows:
- Get the top 36 crypto currencies’ names
- Generate two variations for each coin: “what is <coin>” and “<coin> price”
- Run the
serp_googfunction for all coins, query variations and in two countries, US, and UK
36 coins x 2 variations x 2 countries = 144 requests (1,440 SERP results)
As you can see, there are numerous columns that are returned by the API, although many of them are empty. The default ones will always be full though, and they are placed in the beginning of the DataFrame.
Sample results for the “what is <coin>” query
| gl | searchTerms | rank | displayLink | link | snippet | |
|---|---|---|---|---|---|---|
| 0 | us | what is bitcoin | 1 | bitcoin.org | https://bitcoin.org/ | Bitcoin uses peer-to-peer technology to operat... |
| 1 | us | what is bitcoin | 2 | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | Bitcoin is a digital or virtual currency creat... |
| 2 | us | what is bitcoin | 3 | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | Bitcoin is a digital currency which operates f... |
| 10 | uk | what is bitcoin | 1 | bitcoin.org | https://bitcoin.org/ | Bitcoin uses peer-to-peer technology to operat... |
| 11 | uk | what is bitcoin | 2 | www.newscientist.com | https://www.newscientist.com/definition/bitcoin/ | Bitcoin is a digital currency which operates f... |
| 12 | uk | what is bitcoin | 3 | www.investopedia.com | https://www.investopedia.com/terms/b/bitcoin.asp | Bitcoin is a digital or virtual currency creat... |
| 20 | us | what is litecoin | 1 | litecoin.org | https://litecoin.org/ | What is Litecoin? ... Litecoin is a peer-to-pe... |
| 21 | us | what is litecoin | 2 | www.investopedia.com | https://www.investopedia.com/articles/investin... | Litecoin (LTC) is a cryptocurrency created fro... |
| 22 | us | what is litecoin | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 30 | uk | what is litecoin | 1 | www.investopedia.com | https://www.investopedia.com/articles/investin... | Litecoin (LTC) is a cryptocurrency created fro... |
| 31 | uk | what is litecoin | 2 | litecoin.org | https://litecoin.org/ | What is Litecoin? ... Litecoin is a peer-to-pe... |
| 32 | uk | what is litecoin | 3 | www.investopedia.com | https://www.investopedia.com/articles/investin... | 18 Dec 2021 ... An application-specific integr... |
| 40 | us | what is namecoin | 1 | www.namecoin.org | https://www.namecoin.org/ | Namecoin is an experimental open-source techno... |
| 41 | us | what is namecoin | 2 | www.investopedia.com | https://www.investopedia.com/terms/n/namecoin.asp | Mar 1, 2021 ... Namecoin is a key/value pair r... |
| 42 | us | what is namecoin | 3 | en.wikipedia.org | https://en.wikipedia.org/wiki/Namecoin | or NMC) is a cryptocurrency originally forked ... |
Sample results for the “<coin> price” query
| gl | searchTerms | rank | displayLink | link | snippet | |
|---|---|---|---|---|---|---|
| 720 | us | bitcoin price | 1 | www.coindesk.com | https://www.coindesk.com/price/bitcoin/ | The Bitcoin price is $39,934.91, a change of -... |
| 721 | us | bitcoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/bitcoin | April 23, 2022 - The current price of Bitcoin ... |
| 722 | us | bitcoin price | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/bitcoin/ | The live Bitcoin price today is $39,717.61 USD... |
| 730 | uk | bitcoin price | 1 | www.coindesk.com | https://www.coindesk.com/price/bitcoin/ | The Bitcoin price is $39,934.91, a change of -... |
| 731 | uk | bitcoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/bitcoin | April 23, 2022 - The current price of Bitcoin ... |
| 732 | uk | bitcoin price | 3 | coinmarketcap.com | https://coinmarketcap.com/currencies/bitcoin/ | The live Bitcoin price today is $39,606.70 USD... |
| 740 | us | litecoin price | 1 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 741 | us | litecoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/litecoin | The price of Litecoin has fallen by 7.35% in t... |
| 742 | us | litecoin price | 3 | www.coindesk.com | https://www.coindesk.com/price/litecoin/ | Litecoin Price ; 24H Open. $114.00 ; 24H Chang... |
| 750 | uk | litecoin price | 1 | coinmarketcap.com | https://coinmarketcap.com/currencies/litecoin/ | The live Litecoin price today is $104.96 USD w... |
| 751 | uk | litecoin price | 2 | www.coinbase.com | https://www.coinbase.com/price/litecoin | The price of Litecoin has fallen by 7.74% in t... |
| 752 | uk | litecoin price | 3 | www.coindesk.com | https://www.coindesk.com/price/litecoin/ | Litecoin Price ; 24H Open. $114.00 ; 24H Chang... |
| 760 | us | namecoin price | 1 | www.namecoin.org | https://www.namecoin.org/ | Namecoin. Namecoin is an experimental open-sou... |
| 761 | us | namecoin price | 2 | coinmarketcap.com | https://coinmarketcap.com/currencies/namecoin/ | The live Namecoin price today is $2.12 USD wit... |
| 762 | us | namecoin price | 3 | www.coinbase.com | https://www.coinbase.com/price/namecoin | April 23, 2022 - The current price of Namecoin... |
We can easily create our hetmap with one command:
Modify some of the default options
Subtitle and passing extra options to Plotly
Add a subtitle, and pass any other option straight through to the underlying Plotly figure with **kwargs (forwarded to fig.update_layout).
Faceting: split the chart by a categorical column
Instead of manually subsetting the DataFrame and calling the function again and again, pass facet_row and/or facet_col to split the chart by any categorical column. Combining both produces a 2D grid, and extra make_subplots options (like vertical_spacing) are forwarded automatically.
First, add a column that labels each query’s template:
| searchTerms | rank | displayLink | gl | query template | |
|---|---|---|---|---|---|
| 0 | what is bitcoin | 1 | bitcoin.org | us | what is <coin> |
| 1 | what is bitcoin | 2 | www.investopedia.com | us | what is <coin> |
| 2 | what is bitcoin | 3 | www.newscientist.com | us | what is <coin> |
| 3 | what is bitcoin | 4 | money.cnn.com | us | what is <coin> |
| 4 | what is bitcoin | 5 | www.coinbase.com | us | what is <coin> |
Faceting makes it easy to compare subsets side by side — notice how different the dominant domains are across query templates and countries, all without manually splitting the DataFrame.
Add shapes and annotations for better guiding your users
- Add a rectangle to highight one or more domains
- Make the rectangle colored, or simply color the edges of the rectangle
- Optionally add a label to inform users about domain(s) in this rectangle
fig = serp_heatmap(
# "price" keywords in the USA:
serp[serp["gl"].eq("us") & serp["searchTerms"].str.contains("price")],
queries_col="searchTerms",
domains_col="displayLink",
ranks_col="rank",
template="plotly_white",
title='SERP Heatmap <b>"<coin> price"</b> keywords – USA',
num_domains=13,
)
fig.add_shape(
type="rect", x0=0.5, x1=1.5, y0=-0.5, y1=10.5, fillcolor="gray", opacity=0.15
)
fig.add_shape(
type="rect",
x0=4.5,
x1=6.5,
y0=-0.5,
y1=11.5,
line={"color": "darkgreen"},
label={
"text": "This is a label",
"font": {"family": "Arial Black"},
"textposition": "bottom center",
},
)
figSERP Heatmap chart details
- Avg. Pos.: The average position of the domain shown on this column
- Coverage: The number of appearances of the domain on the SERP, divided by the total queries. A ratio above 100% can exist, where some domains appear more than once in the same SERP.
- Total appearances: the number of times the domain appeared in the SERP (from this dataset).
- Numbers in circles: Show the number of times each domain appeared in its respective position.
Data from other sources
This chart doesn’t need the data from the Google custom search API exclusively. There are other similar data sources like YouTube, SERP API, and SearchAPI for example. The only requirement is that you have a DataFrame with columns for the queries, domains, and ranks. Whatever they are named, you simply pass those names through the queries_col, domains_col, and ranks_col parameters — no renaming required. For example, your queries column might be called keyword or query, and your domains column might be called domain.
Example: data from an LLM (Claude’s serp_claude)
The chart isn’t limited to Google data. The datasets below were produced with a serp_claude function that collects the search results Claude returns for a set of queries. Their columns are named differently (query, domain, serp_rank), so we just pass those names through queries_col, domains_col, and ranks_col — no renaming required.
| query | city | num | serp_rank | domain | |
|---|---|---|---|---|---|
| 0 | 3 star hotels in Paris | Paris | 3 | 1 | www.booking.com |
| 1 | 3 star hotels in Paris | Paris | 3 | 2 | us.trip.com |
| 2 | 3 star hotels in Paris | Paris | 3 | 3 | www.parisinsidersguide.com |
| 3 | 3 star hotels in Paris | Paris | 3 | 4 | www.tripadvisor.com |
| 4 | 3 star hotels in Paris | Paris | 3 | 5 | www.top-paris-hotels.com |
Exporting the chart
Because these charts are interactive Plotly figures, you can save them as a standalone HTML file (readers can hover over the circles to see the queries), or as a static image for reports and social previews.
fig = serp_heatmap(
serp, queries_col="searchTerms", domains_col="displayLink", ranks_col="rank"
)
# Interactive HTML — hover over the circles to see the queries:
fig.write_html("data/serp_heatmap.html")
# Static image (PNG) — also used as this page's social preview image:
fig.write_image("data/images/serp_heatmap.png", width=1000, engine="kaleido", scale=2)