Web Scraping in R

August 18 2017
R

Let’s walk through some steps for web scraping with R. At the bottom of this page there is a table of references that I want to scrape. Let’s use the rvest package to get the HTML associated with that page:

library(rvest)

html <- read_html("https://www.google.com/patents/US608845")
html
## {xml_document}
## <html>
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset= ...
## [2] <body>\n<div id="gbar"><nobr><a class="gb1" href="https://www.google ...

Now let’s use the html_nodes() function to extract the table of interest. I used Chrome’s Developer Tools to get the XPath of the table.

referenced_by <- html_node(html, xpath='//*[@id="intl_patents_v"]/div[8]/table')
referenced_by
## {xml_node}
## <table class="patent-data-table">
##  [1] <thead class="patent-data-table-thead"><tr class="patent-data-table ...
##  [2] <tr>\n<td class="patent-data-table-td citation-patent">\n<a href="/ ...
##  [3] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [4] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [5] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [6] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [7] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [8] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
##  [9] <tr>\n<td class="patent-data-table-td citation-patent"><a href="/pa ...
## [10] <tr>\n<td class="patent-data-table-td citation-patent">\n<a href="/ ...
## [11] <tr>\n<td class="patent-data-table-td citation-patent">\n<a href="/ ...
## [12] <tr>\n<td class="patent-data-table-td citation-patent">\n<a href="/ ...
## [13] <tr>\n<td class="patent-data-table-td citation-patent">\n<a href="/ ...

Now let’s convert that HTML table into a data frame.

html_table(referenced_by)
##      Citing Patent  Filing date Publication date
## 1      US5647309 * Jun 11, 1996     Jul 15, 1997
## 2        US6230683  Dec 8, 1999     May 15, 2001
## 3        US6276334 Feb 23, 1999     Aug 21, 2001
## 4        US6286482 Aug 22, 1997     Sep 11, 2001
## 5        US6915776 Oct 15, 2003     Jul 12, 2005
## 6        US8683982  Aug 9, 2011      Apr 1, 2014
## 7        US8715372 Aug 10, 2009      May 6, 2014
## 8        US8807992  Apr 9, 2010     Aug 19, 2014
## 9  US20040103860 * Oct 15, 2003      Jun 3, 2004
## 10 US20100064574 * Aug 10, 2009     Mar 18, 2010
## 11 US20100266971 *  Apr 9, 2010     Oct 21, 2010
## 12 US20150343625 * Dec 31, 2013      Dec 3, 2015
##                                 Applicant
## 1                        Avery; Alfred J.
## 2            Cummins Engine Company, Inc.
## 3            Cummins Engine Company, Inc.
## 4            Cummins Engine Company, Inc.
## 5                            Cummins Inc.
## 6  Great Plains Diesel Technologies, L.C.
## 7      Petroleo Brasileiro S.A.—Petrobras
## 8                       Jeffrey R. Wagner
## 9                            Cummins Inc.
## 10  Petr\xf3leo Brasileiro S.A.-Petrobras
## 11                       Wagner Jeffrey R
## 12               Illinois Tool Works Inc.
##                                                                               Title
## 1                                          Internal combustion engine firing system
## 2       Premixed charge compression ignition engine with optimal combustion control
## 3       Premixed charge compression ignition engine with optimal combustion control
## 4       Premixed charge compression ignition engine with optimal combustion control
## 5       Premixed charge compression ignition engine with optimal combustion control
## 6                                                 Programmable diesel fuel injector
## 7  Diesel cycle fuel compositions containing dianhydrohexitols and related products
## 8                            Fire generator having a controllable venting mechanism
## 9       Premixed charge compression ignition engine with optimal combustion control
## 10 Diesel cycle fuel compositions containing dianhydrohexitols and related products
## 11                           Fire generator having a controllable venting mechanism
## 12                                            Electropneumatic gas fastening device

We’ve only scratched the surface here, but I hope this example shows off the convenience of the rvest package.

comments powered by Disqus