Recology - R

Stashing and playing with raw data locally from the web

Mon Jun 17 00:00:00 -0700 2013

It is getting easier to get data directly into R from the web. Often R packages that retrieve data from the web return useful R data structures to users like a data.frame. This is a good thing of course to make things user friendly.

However, what if you want to drill down into the data that's returned from a query to a database in R? What if you want to get that nice data.frame in R, but you think you may want to look at the raw data later? The raw data from web queries are often JSON or XML data. This type of data, especially JSON, can be easily stored in schemaless so-called NoSQL databases, and queried later.

A brief aside: What are JSON and XML? This is what JSON looks like (ps if you ever wonder if your JSON is correct, go here):

{
  "name": "joe",
  "hobby": "codemonkey",
  "lives": [
      {
          "city": "San Jose",
          "state": "CA"
      }
  ]
}

This is what XML looks like:

<?xml version="1.0" encoding="UTF-8" ?>
  <name>joe</name>
    <hobby>codemonkey</hobby>
    <lives>
        <city>San Jose</city>
        <state>CA</state>
    </lives>

But don't worry if it looks complicated - the project I talk about below, sofa, tries to make the interface to JSON and XML easy. Web APIs almost always return either JSON or XML, so this is the raw data.

So here's the use case I imagine, or workflow:

Query a database on the web, and choose to write the raw data to a local database.
Do whatever you want with the output R object - analyze, visualize, etc.
Now you want to go back and search through some of the raw data. But, that query took an hour. Since you wrote it to a local database, you can search the data.
If you hadn't written it locally, you would have to make a new web call.

Note that if you are doing calls to web APIs that get small amounts of data you don't need to worry too much as you can easily just do the call again.

I've started an R package to interact with the NoSQL database CouchDB. CouchDB is a schemaless database that speaks JSON, so you can store JSON and get back JSON (don't worry XML, we got you covered - we can just wrap the XML in JSON before putting it in CouchDB). What's especially cool is you can interact with CouchDB via a RESTful API. CouchDB doesn't have full text search built in (though you can build map-reduce Views, basically preset queries on the database), so I added functions (and docs to help) to interact with the CouchDB River plugin for Elasticsearch, which provides powerful full text search via an API interface. But nevermind the tech details - all this just means you can search on the full text of your stored data.

There are plenty of databases you can interact with from R, so why CouchDB? For one, it makes a lot of sense to write to a NoSQL database since this blog post is dealing with a use case writing JSON, which isn't a good fit for databases like MySQL, SQLite, PostgreSQL, etc. (though postgres allows you to write JSON). It didn't have to be CouchDB, but at least to me it seems relatively easy to install, you can interact with it via an HTTP API (if you're into that, which I am), and it has a nice web interface (navigate to http://localhost:5984/_utils/ after starting couchdb).

Is this for the casual R user? Probably not. But, I imagine there are R users out there that want some more flexibility when it comes to interacting with web data in R. It is nice and tidy to get back an R data.frame from a web call, but having the raw data at your fingertips could be super powerful. I'll describe using an R package to interact with a web database with sofa baked in, and discuss a bit about the functions within sofa.

First start CouchDB in your terminal

You can do this from anywhere in your directory. See here for instructions on how to install CouchDB.

bash couchdb

Then start elasticsearch in your terminal

See here for instructions on how to install Elasticsearch and the River CouchDB plugin.

bash cd /usr/local/elasticsearch bin/elasticsearch -f

Install sofa

# Uncomment these lines if you don't have these packages installed
# install.packages('devtools') library(devtools) install_github('sofa',
# 'schamberlain') install_github('alm', 'ropensci', ref='couch')
library(sofa)
library(alm)

Simultaneously write data to CouchDB along with API calls using the alm package to get altmetrics data on PLoS papers. Ping to make sure CouchDB is on

sofa_ping()

  couchdb   version 
"Welcome"   "1.2.1"

Create a new database

sofa_createdb(dbname = "alm_db")

  ok 
TRUE

Write couchdb database name to options

options(couch_db_name = "alm_db")

List the databases

sofa_listdbs()

 [1] "_replicator"                "_users"                    
 [3] "alm_couchdb"                "alm_db"                    
 [5] "dudedb"                     "example"                   
 [7] "poop"                       "rplos_db"                  
 [9] "shit"                       "shitty"                    
[11] "shitty2"                    "test_suite_db"             
[13] "test_suite_db/with_slashes" "test_suite_reports"        
[15] "testr2couch"                "twitter_db"

Search for altmetrics normally, w/o writing to a database

head(alm(doi = "10.1371/journal.pone.0029797"))

          .id pdf html shares groups comments likes citations total
1   bloglines  NA   NA     NA     NA       NA    NA         0     0
2   citeulike  NA   NA      1     NA       NA    NA        NA     1
3    connotea  NA   NA     NA     NA       NA    NA         0     0
4    crossref  NA   NA     NA     NA       NA    NA         6     6
5      nature  NA   NA     NA     NA       NA    NA         4     4
6 postgenomic  NA   NA     NA     NA       NA    NA         0     0

Search for altmetrics normally, while writing to a database

head(alm(doi = "10.1371/journal.pone.0029797", write2couch = TRUE))

          .id pdf html shares groups comments likes citations total
1   bloglines  NA   NA     NA     NA       NA    NA         0     0
2   citeulike  NA   NA      1     NA       NA    NA        NA     1
3    connotea  NA   NA     NA     NA       NA    NA         0     0
4    crossref  NA   NA     NA     NA       NA    NA         6     6
5      nature  NA   NA     NA     NA       NA    NA         4     4
6 postgenomic  NA   NA     NA     NA       NA    NA         0     0

Make lots of calls, and write them simultaneously

# install_github('rplos', 'ropensci')
library(rplos)
dois <- searchplos(terms = "evolution", fields = "id", limit = 100)
out <- alm(doi = as.character(dois[, 1]), write2couch = TRUE)
lapply(out[1:2], head)

$`01`
          .id pdf html shares groups comments likes citations total
1   bloglines  NA   NA     NA     NA       NA    NA         0     0
2   citeulike  NA   NA      0     NA       NA    NA        NA     0
3    connotea  NA   NA     NA     NA       NA    NA         0     0
4    crossref  NA   NA     NA     NA       NA    NA         0     0
5      nature  NA   NA     NA     NA       NA    NA         0     0
6 postgenomic  NA   NA     NA     NA       NA    NA         0     0

$`02`
          .id pdf html shares groups comments likes citations total
1   bloglines  NA   NA     NA     NA       NA    NA         0     0
2   citeulike  NA   NA      1     NA       NA    NA        NA     1
3    connotea  NA   NA     NA     NA       NA    NA         0     0
4    crossref  NA   NA     NA     NA       NA    NA         2     2
5      nature  NA   NA     NA     NA       NA    NA         0     0
6 postgenomic  NA   NA     NA     NA       NA    NA         0     0

Writing data to CouchDB does take a bit longer

system.time(alm(doi = as.character(dois[, 1])[1:60], write2couch = FALSE))

   user  system elapsed 
  1.739   0.016   4.554

system.time(alm(doi = as.character(dois[, 1])[1:60], write2couch = TRUE))

   user  system elapsed 
  3.579   0.062   6.460

Search using elasticsearch

tell elasticsearch to start indexing your database

elastic_start(dbname = "alm_db")

$ok
[1] TRUE

Search your database

out <- elastic_search(dbname = "alm_db", q = "twitter", parse_ = TRUE)
out$hits$total

[1] 679

Using views

Write a view - here letting key be the default of null

sofa_view_put(dbname = "alm_db", design_name = "myview", value = "doc.baseurl")

$ok
[1] TRUE

$id
[1] "_design/myview"

$rev
[1] "1-e7c17cff1b96e4595c3781da53e16ad8"

Get info on your new view

sofa_view_get(dbname = "alm_db", design_name = "myview")

$`_id`
[1] "_design/myview"

$`_rev`
[1] "1-e7c17cff1b96e4595c3781da53e16ad8"

$views
$views$foo
                                    map 
"function(doc){emit(null,doc.baseurl)}"

Get data using a view

out <- sofa_view_search(dbname = "alm_db", design_name = "myview")
length(out$rows)  # 160 results

[1] 161

sapply(out$rows, function(x) x$value)[1:5]  # the values, just the API call URLs

[1] "http://alm.plos.org/api/v3/articles"
[2] "http://alm.plos.org/api/v3/articles"
[3] "http://alm.plos.org/api/v3/articles"
[4] "http://alm.plos.org/api/v3/articles"
[5] "http://alm.plos.org/api/v3/articles"

Delete the view

sofa_view_del(dbname = "alm_db", design_name = "myview")

[1] ""

What now?

Well, if no one uses this, then probably nothing. Though, if people think this could be useful:

It would be cool to make easy hooks into any package making web calls to allow users to write data to CouchDB if they choose to, sort of like the example above with rplos.
Perhaps automate some of the setup for CouchDB for users, making system calls so they don't have to.
Performance: As shown above, simultaneously writing data to CouchDB takes longer than not doing so - removing this time difference will make writing to couch more palatable.

What do you think?

What is your reaction to this post? Do you have a need for this sort of tool? Do you have similar use cases that could be addressed with sofa?

Fylopic, an R wrapper to Phylopic

Sat Jun 01 00:00:00 -0700 2013

What is PhyloPic?

PhyloPic is an awesome new service - I'll let the creator, Mike Keesey, explain what it is (paraphrasing here):

PhyloPic stores silhouette images of organisms, and each image is associated with taxonomic names, and stores the taxonomy of all taxa, allowing searching by taxonomic names. Anyone can submit silhouettes to PhyloPic.

What is a silhouette? It's like this:

by Gareth Monger

What makes PhyloPic not just awesome, but super awesome? All or most images are licensed under Creative Commons licenses. This means you can use the silhouettes without having to ask or pay - just attribute.

What is fylopic?

The idea behind Fylopic is to create modular bits and pieces (i.e., functions) to allow you to add silhouettes to not only ggplot2 plots, but base plots as well. That is, you can simply load fylopic in your R session, and add some silhouettes to your phylogeny, or your barchart, etc. - that is, fylopic is meant to be a helper in your workflow to add in silhouettes to visualizations.

Some people prefer base plots while others prefer ggplot2 plots (me!), so it would be nice to have both options. Phylogenies at the moment render faster in base plots. I don't yet have implementations for base plots, but will come soon, or you can send a pull request to add it.

One interesting use case could be to be able to get a set of silhouettes, then get a phylogeny for taxa associatd with the silhouettess using the NCBI taxonomy, but it's not easily available yet (though I may be able to use Ben Morris' phylocommons soon. This isn't doable yet, so in the example below the function make_phylo creates a phylogeny using ape::rcoal.

You could also do the reverse -> you have a phylogeny and then you could search Phylopic for silhouettes.

Info

Check out the Phylopic website here, and Phylopic API developer documentation here.

Also check out Ben Morris' Python wrapper to Phylopic here.

What can you do with fylopic?

Install fylopic

install.packages("devtools")
library(devtools)
install_github("fylopic", "schamberlain")

library(fylopic, quietly = TRUE)

Plot a phylogeny with silhouettes at the tips

Here, I search for names based on keyword Homo sapiens - which returns many matche codes. With those results we search for any silhouettes associated with those codes. Then we download images. Finally, make a phylogeny with the silhouettes at the tips. Note that in this eample the phylogeny is just a random coalescent tree made using ape::rcoal - obviously, in the real world you'd want to do something more useful.

## search on Homo sapiens
searchres <- search_text(text = "Homo sapiens", options = "names")

### which returns UUIDs
searchres[1:3]

[1] "74aea16b-666b-497a-b2cb-72201ad75a8e"
[2] "1ee65cf3-53db-4a52-9960-a9f7093d845d"
[3] "cc9ad8ee-3a82-4add-8d50-bc78f4ff6956"

## search for images based on the UUIds
output <- search_images(uuid = searchres, options = c("pngFiles", "credit", 
    "canonicalName"))

### we got eight matches
output

$`15444b9c-f17f-4d6e-89b5-5990096bcfb0`
$`15444b9c-f17f-4d6e-89b5-5990096bcfb0`$supertaxa
[1] "e547cd01-7dd1-495b-8239-52cf9971a609"
[2] "bd88f674-6976-4cb2-a46e-e6a12a8ba463"


$`fedf0e5f-f20a-442c-accf-eb84a3af8c6b`
$`fedf0e5f-f20a-442c-accf-eb84a3af8c6b`$supertaxa
[1] "e547cd01-7dd1-495b-8239-52cf9971a609"
[2] "bd88f674-6976-4cb2-a46e-e6a12a8ba463"


$`a88d3a4c-44d3-409e-87b6-516bd188c709`
$`a88d3a4c-44d3-409e-87b6-516bd188c709`$supertaxa
[1] "e547cd01-7dd1-495b-8239-52cf9971a609"
[2] "bd88f674-6976-4cb2-a46e-e6a12a8ba463"


$`d88164ec-3152-444b-b41c-4757a344a764`
$`d88164ec-3152-444b-b41c-4757a344a764`$supertaxa
[1] "9c6af553-390c-4bdd-baeb-6992cbc540b1"


$`da5eaeb7-1ed2-4b2e-ad4a-49993881d706`
$`da5eaeb7-1ed2-4b2e-ad4a-49993881d706`$supertaxa
[1] "9c6af553-390c-4bdd-baeb-6992cbc540b1"

## download images
myobjs <- get_image(uuids = output, size = "128")

## make the phylogeny
make_phylo(pngobj = myobjs)

Plot a silhouette behind a plot

Notice in the below example that you can use normal ggplot2 syntax, and simply add another layer (add_phylopic from fylopic) to the plot.

library(ggplot2)
img <- get_image("27356f15-3cf8-47e8-ab41-71c6260b2724", size = "512")[[1]]
qplot(x = Sepal.Length, y = Sepal.Width, data = iris, geom = "point") + add_phylopic(img)

What's next?

This is a side project, so if anyone has interest in helping please do contribute code, report bugs, request features, etc.

BISON USGS species occurrence data

Mon May 27 00:00:00 -0700 2013

The USGS recently released a way to search for and get species occurrence records for the USA. The service is called BISON (Biodiversity Information Serving Our Nation). The service has a web interface for human interaction in a browser, and two APIs (application programming interface) to allow machines to interact with their database. One of the APIs allows you to search and retrieve data, and the other gives back maps as either a heatmap or a species occurrence map. The latter is more appropriate for working in a browser, so I'll leave that to the web app folks.

The Core Science Analytics and Synthesis (CSAS) program of the US Geological Survey are responsible for BISON, and are the US node of the Global Biodiversity Information Facility (GBIF). BISON data is nested within that of GBIF, but has (or wil have?) additional data not in GBIF, as described on their About page:

BISON has been initiated with the 110 million records GBIF makes available from the U.S. and is integrating millions more records from other sources each year

Have a look at their Data providers and Statistics tabs on the BISON website, which list where data comes from and how many searches and downloads have been done on each data provider.

We (rOpenSci) started an R package to interact with the BISON search API >> rbison. You may be thinking, but if the data in BISON is also in GBIF, why both making another R package for BISON? Good question. As I just said, BISON will have some data GBIF won't have. Also, the services (search API and map service) are different than those of GBIF.

Check out the package on GitHub here https://github.com/ropensci/rbison.

Here is a quick run through of some things you can do with rbison.

Install ribson

# Install rbison from GitHub using devtools, uncomment to install
# install.packages('devtools') library(devtools) install_github('rbison',
# 'ropensci')
library(rbison)

Search the BISON database for, of course, bison

# Do the search
out <- bison(species = "Bison bison", type = "scientific_name", start = 0, count = 10)

# Check that the returned object is the right class ('bison')
class(out)

[1] "bison"

Get a summary of the data

bison_data(out)

  total observation fossil specimen unknown
1   761          30      4      709      18

Summary by counties (just the first 6 rows)

head(bison_data(input = out, datatype = "counties"))

  record_id total county_name      state
1     48295     7    Lipscomb      Texas
2     41025    15      Harney     Oregon
3     49017     8    Garfield       Utah
4     35031     2    McKinley New Mexico
5     56013     1     Fremont    Wyoming
6     40045     2       Ellis   Oklahoma

Summary of states

bison_data(input = out, datatype = "states")

      record_id total county_fips
1    Washington     1          53
2         Texas     8          48
3    New Mexico     8          35
4          Iowa     1          19
5       Montana     9          30
6       Wyoming   155          56
7        Oregon    15          41
8      Oklahoma    14          40
9        Kansas    10          20
10      Arizona     1          04
11       Alaska    29          02
12         Utah    16          49
13     Colorado    17          08
14     Nebraska     1          31
15 South Dakota    61          46

Map the results

# Search for Ursus americanus (american black bear)
out <- bison(species = "Ursus americanus", type = "scientific_name", start = 0, 
    count = 200)

# Sweet, got some data
bison_data(out)

  total observation fossil specimen literature unknown centroid
1  3792          59    125     3522         47      39       78

Make some maps! Note that right now the county and state maps just plot the conterminous lower 48. The map of individual occurrences shows the lower 48 + Alaska

# By county
bisonmap(out, tomap = "county")

# By state
bisonmap(out, tomap = "state")

# Individual locations
bisonmap(out)

## Rendering map...plotting 199 points

When plotting occurrences, you can pass additional arguments into the `bisonmap` function.

For example, you can jitter the points

bisonmap(input = out, geom = geom_jitter)

## Rendering map...plotting 199 points

And you can specify by how much you want the points to jitter (here an extreme example to make it obvious)

library(ggplot2)
bisonmap(input = out, geom = geom_jitter, jitter = position_jitter(width = 5))

## Rendering map...plotting 199 points

Let us know if you have any feature requests or find bugs at our GitHub Issues page.

Scholarly metadata in R

Sat Mar 16 00:00:00 -0700 2013

Scholarly metadata - the meta-information surrounding articles - can be super useful. Although metadata does not contain the full content of articles, it contains a lot of useful information, including title, authors, abstract, URL to the article, etc.

One of the largest sources of metadata is provided via the Open Archives Initiative Protocol for Metadata Harvesting or OAI-PMH. Many publishers, provide their metadata through their own endpoint, and implement the standard OAI-PMH methods: GetRecord, Identify, ListIdentifiers, ListMetadataFormats, ListRecords, and ListSets. Many providers use OAI-PMH, including DataCite, Dryad, and PubMed.

Some data-/article-providers provide their metadata via their own APIs. For example, Nature Publishing Group provides their own metadata API here in non OAI-PMH format; you can get PLoS metadata through their search API, and the BHL (see below) provides their own custom metadata service.

In addition, CrossRef provides a number of metadata search services: metadata search and openurl.

What about the other publishers? (please tell me if I'm wrong about these three)

Springer has a metadata API, but it is terrible, soooo...
Elsevier, are you kidding? Well, they do have some sort of API service, but its a pain in the ass.
Wiley, no better than Elsevier.

Note that metadata can live in other places:

Another package being developed by David Springate, rpubmed can get PubMed metadata.
Our wrapper to the Mendeley API, RMendeley, gets article metadata via Mendeley's database.
Our wrapper to the Biodiversity Heritage Library API here gets their metadata.

No, you can't get metadata via Google Scholar - the don't allow scraping, and don't have expose their data via an API.

I have discussed this package in a previous blog post, but have since worked on the code a bit, and thought it deserved a new post.

You can see a tutorial for this package here, and contribute to the code here.

Install rmetadata

# install_github('rmetadata', 'ropensci') # uncomment to install
library(rmetadata)

Count OAI-PMH identifiers for a data provider.

# For DataCite.
count_identifiers("datacite")

  provider   count
1 datacite 1216193

Lookup article info via CrossRef with DOI and get a citation.

As Bibtex

print(crossref_citation("10.3998/3336451.0009.101"), style = "Bibtex")

@Article{,
  title = {In Google We Trust?},
  author = {Geoffrey Bilder},
  journal = {The Journal of Electronic Publishing},
  year = {2006},
  month = {01},
  volume = {9},
  doi = {10.3998/3336451.0009.101},
}

As regular text

print(crossref_citation("10.3998/3336451.0009.101"), style = "text")

Bilder G (2006). "In Google We Trust?" _The Journal of Electronic
Publishing_, *9*. <URL:
http://dx.doi.org/10.3998/3336451.0009.101>.

Search the CrossRef Metatdata for DOIs using free form references.

Search with title, author, year, and journal

crossref_search_free(query = "Piwowar Sharing Detailed Research Data Is Associated with Increased Citation Rate PLOS one 2007")

                                                                                             text
1 Piwowar Sharing Detailed Research Data Is Associated with Increased Citation Rate PLOS one 2007
  match                   doi score
1  TRUE 10.1038/npre.2007.361 4.905

Get a DOI and get the citation using \code{crossref_search}

# Get a DOI for a paper
doi <- crossref_search_free(query = "Piwowar sharing data PLOS one")$doi

# Get the metadata
crossref_search(doi = doi)[, 1:3]

                           doi score normalizedScore
1 10.1371/journal.pone.0000308 18.19             100

Get a random set of DOI's through CrossRef.

# Default search gets 20 random DOIs
crossref_r()

 [1] "10.4028/www.scientific.net/MSF.126-128.467"
 [2] "10.2139/ssrn.548523"                       
 [3] "10.1016/S0012-821X(02)00562-9"             
 [4] "10.1093/rsq/13.2-3.167"                    
 [5] "10.5772/55055"                             
 [6] "10.1515/BC.1999.050"                       
 [7] "10.1016/S0020-7292(98)90160-6"             
 [8] "10.1111/j.1439-0418.1985.tb02788.x"        
 [9] "10.1089/aid.2012.0115"                     
[10] "10.1016/0002-9378(95)90155-8"              
[11] "10.1001/jama.1949.02900490055028"          
[12] "10.1051/jphyscol:1989172"                  
[13] "10.1016/s0301-2115(03)00298-7"             
[14] "10.1007/BF02735292"                        
[15] "10.1016/0003-4916(65)90026-6"              
[16] "10.4156/jdcta.vol5.issue5.12"              
[17] "10.1007/s10904-009-9316-2"                 
[18] "10.1023/A:1021690001832"                   
[19] "10.1007/s12262-012-0724-0"                 
[20] "10.1007/bf02192860"

# Specify you want journal articles only
crossref_r(type = "journal_article")

 [1] "10.1016/j.jacc.2011.09.055"                                 
 [2] "10.1002/dev.420170603"                                      
 [3] "10.4315/0362-028X.JFP-10-403"                               
 [4] "10.1016/S0925-4927(98)00016-X"                              
 [5] "10.1111/j.1933-1592.2002.tb00141.x"                         
 [6] "10.1541/ieejfms.127.629"                                    
 [7] "10.5539/enrr.v3n1p62"                                       
 [8] "10.1016/S0960-9776(96)90038-7"                              
 [9] "10.1016/0925-9635(94)05240-9"                               
[10] "10.1016/s0929-693x(97)86846-7"                              
[11] "10.1002/(SICI)1096-9071(199601)48:1<53::AID-JMV9>3.0.CO;2-K"
[12] "10.1016/s0267-7261(01)00016-1"                              
[13] "10.1111/j.1748-0361.2003.tb00575.x"                         
[14] "10.1097/00005721-197701000-00011"                           
[15] "10.1007/s00894-009-0593-z"                                  
[16] "10.1071/AR9830063"                                          
[17] "10.1186/gb-2009-10-4-r39"                                   
[18] "10.2165/00128415-201113540-00038"                           
[19] "10.1007/BF00522986"                                         
[20] "10.1080/19407963.2011.539385"

Search the CrossRef Metatdata API.

# Search for two different query terms
crossref_search(query = c("renear", "palmer"), rows = 4)[, 1:3]

                            doi score normalizedScore
1       10.1126/science.1157784 3.253             100
2  10.1002/meet.2009.1450460141 2.169              66
3 10.4242/BalisageVol3.Renear01 2.102              64
4 10.4242/BalisageVol5.Renear01 2.102              64

# Get results for a certain year
crossref_search(query = c("renear", "palmer"), year = 2010)[, 1:3]

                                  doi  score normalizedScore
1            10.1002/meet.14504701218 1.0509             100
2            10.1002/meet.14504701240 1.0509             100
3           10.5270/OceanObs09.cwp.68 1.0442              99
4               10.1353/mpq.2010.0003 0.6890              65
5                  10.1353/mpq.0.0041 0.6890              65
6                  10.1353/mpq.0.0044 0.6890              65
7                  10.1353/mpq.0.0057 0.6890              65
8                    10.1386/fm.1.1.2 0.6890              65
9                    10.1386/fm.1.2.2 0.6890              65
10                   10.1386/fm.1.3.2 0.6890              65
11       10.1097/ALN.0b013e3181f09404 0.6090              57
12      10.1016/j.urology.2010.02.033 0.6090              57
13              10.1353/ect.2010.0025 0.6090              57
14               10.1117/2.4201001.04 0.6090              57
15 10.1111/j.1835-9310.1977.tb01159.x 0.6090              57
16    10.4067/S0717-69962010000100001 0.6090              57
17    10.4067/S0717-69962010000200001 0.6090              57
18           10.2105/AJPH.2009.191098 0.6029              57
19              10.1353/mpq.2010.0004 0.5167              49
20                 10.1353/mpq.0.0048 0.5167              49

Get a short DOI from shortdoi.org.

# Geta a short DOI, just the short DOI returned
short_doi(doi = "10.1371/journal.pone.0042793")

[1] "10/f2bfz9"

# Geta a short DOI, all data returned
short_doi(doi = "10.1371/journal.pone.0042793", justshort = FALSE)

$DOI
[1] "10.1371/journal.pone.0042793"

$ShortDOI
[1] "10/f2bfz9"

$IsNew
[1] FALSE

Get a record from a OAI-PMH data provider.

# Single provider, one identifier
md_getrecord(provider = "pensoft", identifier = "10.3897/zookeys.1.10")

                                                                                                title
1 A new candidate for a Gondwanaland distribution in the Zodariidae (Araneae): Australutica in Africa
      creator date             type
1 Jocqué,Rudy 2008 Research Article

# Single provider, multiple identifiers
md_getrecord(provider = "pensoft", identifier = c("10.3897/zookeys.1.10", "10.3897/zookeys.4.57"))

                                                                                                   title
1    A new candidate for a Gondwanaland distribution in the Zodariidae (Araneae): Australutica in Africa
2 Studies of Tiger Beetles. CLXXVIII. A new Lophyra (Lophyra) from Somaliland (Coleoptera, Cicindelidae)
        creator date             type
1   Jocqué,Rudy 2008 Research Article
2 Cassola,Fabio 2008 Research Article

List available metadata formats from various providers.

# List metadata formats for a provider
md_listmetadataformats(provider = "dryad")

  metadataPrefix
1         oai_dc
2            rdf
3            ore
4           mets
                                                       schema
1              http://www.openarchives.org/OAI/2.0/oai_dc.xsd
2                 http://www.openarchives.org/OAI/2.0/rdf.xsd
3 http://tweety.lanl.gov/public/schemas/2008-06/atom-tron.sch
4                  http://www.loc.gov/standards/mets/mets.xsd
                            metadataNamespace
1 http://www.openarchives.org/OAI/2.0/oai_dc/
2    http://www.openarchives.org/OAI/2.0/rdf/
3                 http://www.w3.org/2005/Atom
4                    http://www.loc.gov/METS/

# List metadata formats for a specific identifier for a provider
md_listmetadataformats(provider = "pensoft", identifier = "10.3897/zookeys.1.10")

            identifier metadataPrefix
1 10.3897/zookeys.1.10         oai_dc
2 10.3897/zookeys.1.10           mods
                                             schema
1    http://www.openarchives.org/OAI/2.0/oai_dc.xsd
2 http://www.loc.gov/standards/mods/v3/mods-3-1.xsd
                            metadataNamespace
1 http://www.openarchives.org/OAI/2.0/oai_dc/
2                  http://www.loc.gov/mods/v3

Some plotting - mean number of authors per paper

Okay, so this isn't a super useful visualization, but you can surely think of something better.

library(ggplot2)
library(ggthemes)
library(reshape)


temp <- md_listrecords(provider = "pensoft", from = "2011-10-01", until = "2012-01-01")
temp2 <- ldply(temp)[, -1]
auths <- sapply(temp2$creator, function(x) length(strsplit(as.character(x), 
    ";")[[1]]))
toplot <- data.frame(authors = auths, articletype = temp2$type)
toplot_ <- ddply(toplot, .(articletype), summarise, authors = mean(authors))
toplot_$articletype <- reorder(toplot_$articletype, toplot_$authors)

ggplot(toplot_, aes(articletype, authors)) + theme_tufte(base_size = 16) + geom_bar(stat = "identity") + 
    coord_flip()

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr, and knitcitations.

Visualizing rOpenSci collaboration

Fri Mar 08 00:00:00 -0800 2013

We (rOpenSci) have been writing code for R packages for a couple years, so it is time to take a look back at the data. What data you ask? The commits data from GitHub ~ data that records who did what and when.

Using the Github commits API we can gather data on who commited code to a Github repository, and when they did it. Then we can visualize this hitorical record.

Install some functions for interacting with the Github API via R

install_github('sandbox', 'ropensci') 

library(sandbox)
library(httr)
library(ggplot2)
library(scales)
library(reshape2)
library(bipartite)
library(doMC)
library(plyr)
library(ggthemes)
library(picante)

# And authenticate - pops open a page in your default browser, then tells 
# you authentication was successful
github_auth()

Get all repos for an organization, here ropensci of course

ropensci_repos <- github_allrepos(userorg = "ropensci")

Get commits broken down in to additions and deletions, though below we just collapse them to all commits

registerDoMC(cores = 4)
github_commits_safe <- plyr::failwith(NULL, github_commits)
out <- llply(ropensci_repos, function(x) github_commits_safe("ropensci", x, 
    since = "2009-01-01T", limit = 500), .parallel = TRUE)
names(out) <- ropensci_repos
out2 <- compact(out)
outdf <- ldply(out2)

Plot commits by date and repo

outdf_subset <- outdf[!outdf$.id %in% c("citeulike", "challenge", "docs", "ropensci-book", 
    "usecases", "textmine", "usgs", "ropenscitoolkit", "neotoma", "rEWDB", "rgauges", 
    "rodash", "ropensci.github.com", "ROAuth"), ]
outdf_subset$.id <- tolower(outdf_subset$.id)
outdf_subset <- ddply(outdf_subset, .(.id, date), summarise, value = sum(value))

mindates <- llply(unique(outdf_subset$.id), function(x) min(outdf_subset[outdf_subset$.id == 
    x, "date"]))
names(mindates) <- unique(outdf_subset$.id)
mindates <- sort(do.call(c, mindates))
outdf_subset$.id <- factor(outdf_subset$.id, levels = names(mindates))

ggplot(outdf_subset, aes(date, value, fill = .id)) + 
    geom_bar(stat = "identity", width = 0.5) + 
    geom_rangeframe(sides = "b", colour = "grey") + 
    theme_bw(base_size = 9) + 
    scale_x_date(labels = date_format("%Y"), breaks = date_breaks("year")) + 
    scale_y_log10() + 
    facet_grid(.id ~ .) + 
    labs(x = "", y = "") + 
    theme(axis.text.y = element_blank(), 
        axis.text.x = element_text(colour = "black"), 
        axis.ticks.y = element_blank(), 
        strip.text.y = element_text(angle = 0, size = 8, ), 
        strip.background = element_rect(size = 0), 
        panel.grid.major = element_blank(), 
        panel.grid.minor = element_blank(), 
        legend.text = element_text(size = 8), 
        legend.position = "none", 
        panel.border = element_blank())

The plot above plots the sum of additions+deletions, and is sorted by the first commit date of reach repo, with the first being treebase, which wraps the Treebase API, and the most recent being rwbclimate, which wraps the World Blank climate data API.

You can see that some repos have recieved commits more or less consistently over their life time, while others have seen a little development here and there.

In addition, there are quite a few people that have committed code now to rOpenSci repos, calling for a network vizualization of course.

outdf_network <- droplevels(outdf[!outdf$.id %in% c("citeulike", "challenge", 
    "docs", "ropensci-book", "usecases", "textmine", "usgs", "ropenscitoolkit", 
    "retriever", "rodash", "ropensci.github.com", "ROAuth", "rgauges", "sandbox", 
    "rfna", "rmetadata", "rhindawi", "rpmc", "rpensoft", "ritis"), ])
casted <- dcast(outdf_network, .id + date + name ~ variable, fun.aggregate = length, 
    value.var = "value")
names(casted)[1] <- "repo"
casted2 <- ddply(casted, .(repo, name), summarise, commits = sum(additions))
casted2 <- data.frame(repo = casted2$repo, weight = casted2$commits, name = casted2$name)
mat <- sample2matrix(casted2)
plotweb(sortweb(mat, sort.order = "dec"), method = "normal", text.rot = 90, 
    adj.high = c(-0.3, 0), adj.low = c(1, -0.3), y.width.low = 0.05, y.width.high = 0.05, 
    ybig = 0.09, labsize = 0.7)

The plot above shows repos on one side and contributors on the other. Some folks (the core rOpenSci team: cboettig, karthikram, emhart, and schamberlain) have committed quite a lot to many packages. We also have amny awesome contributors to our packages (some contributors and repos have been removed for clarity).

rOpenSci is truly a collaborative effort to develop tools for open science, so thanks to all our contributors - keep on forking, pull requesting, and commiting.

Getting a simple tree via NCBI

Thu Feb 14 00:00:00 -0800 2013

I was just at the Phylotastic hackathon in Tucson, AZ at the iPlant facilities at the UofA.

A problem that needs to be solved is getting the incrasingly vast phylogenetic information to folks not comfortable building their own phylogenies. Phylomatic has made this super easy for people that want plant phylogenies (at least 250 or so papers have used and cited Phylomatic in their papers) - however, there are few options for those that want phylogenies for other taxonomic groups.

One cool tool that was brought up was the Common Tree service provided by NCBI. Here's some help on the service. Unlike Phylomatic, Common Tree is purely based off of taxonomic relationships (A and B are both in the C family, so are sisters), not an actual phylogeny as Phylomatic is based on.

But how do you use Common Tree?

Get a species list

Grab the taxon list from my github account here

Go to the site

Go to the Common Tree site here

Choose file

Hit the "choose file" button, then select the species.txt file you downloaded in the first step.

Add the species list to make the tree

Then hit "add from file", and you got a "tree"

Download

You can download the tree in a variety of formats, including a .phy file

Plot the tree on your machine

Make a tree, in R for me

# install.packages('ape') # install if you don't have ape
library(ape)

# Read the tree in. YOu get the tree back with alot of newlines (\n) -
# can easily take these out with a good text editor.
tree <- read.tree(text = "(Lampetra:4,((((((Umbra:4,((Lota:4,Microgadus:4)Gadiformes:4,((Culaea:4,Apeltes:4,Pungitius:4,Gasterosteus:4)Gasterosteidae:4,(Morone:4,(Ambloplites:4,Micropterus:4,Lepomis:4)Centrarchidae:4,(Sander:4,Perca:4)Percidae:4)Percoidei:4,Cottus:4)Percomorpha:4)Holacanthopterygii:4)Neognathi:4,(((Prosopium:4,Coregonus:4)Coregoninae:4,(Salvelinus:4,Salmo:4,Oncorhynchus:4)Salmoninae:4)Salmonidae:4,Osmerus:4)Protacanthopterygii:4)Euteleostei:4,(Alosa:4,(Ameiurus:4,(Catostomus:4,(Semotilus:4,Rhinichthys:4,Margariscus:4,Couesius:4,Pimephales:4,Luxilus:4,Notemigonus:4,Notropis:4,Carassius:4)Cyprinidae:4)Cypriniformes:4)Otophysi:4)Otocephala:4)Clupeocephala:4,Anguilla:4)Elopocephala:4,Acipenser:4)Actinopteri:4,Scyliorhinus:4)Gnathostomata:4)Vertebrata:4;")

# stretch the branches so tips line up
tree2 <- compute.brlen(tree, method = "Grafen")

# Plot the tree
plot(tree2, no.margin = TRUE, cex = 0.7)

w00p, there it is...

And the answer is NO to the question: Is there an API for Common Tree?

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr.

Waiting for an API request to complete

Sat Jan 26 00:00:00 -0800 2013

Dealing with API tokens in R

In my previous post I showed an example of calling the Phylotastic taxonomic name resolution API Taxosaurus here. When you query their API they give you a token which you use later to retrieve the result (see examples on their page above). However, you don't know when the query will be done, so how do we know when to send the query to rerieve the data?

As the time this takes depends on how big the query is and other things, we don't know when we can get the result. I struggled with this for a bit, but then settled on using a while loop.

So what does this look like? Basically we just keep sending the request for data until we get it.

iter <- 0  # make an iterator so each time we call
output <- list()  # make an empty list to put data into
timeout <- "wait"
while (timeout == "wait") {
    iter <- iter + 1  # increase the iterator each time
    temp <- fromJSON(getURL(retrieve))  # send the request and parse the JSON
    if (grepl("is still being processed", temp["message"]) == TRUE) {
        timeout <- "wait"
    } else {
        output[[iter]] <- temp  # put result from query in the list
        timeout <- "done"  # we got the result so timeout is now done, making the while loop stop
    }
}

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr.

Resolving species names when you have a lot of them

Fri Jan 25 00:00:00 -0800 2013

taxize use case: Resolving species names when you have a lot of them

Species names can be a pain in the ass, especially if you are an ecologist. We ecologists aren't trained in taxonomy, yet we often end up with huge species lists. Of course we want to correct any spelling errors in the names, and get the newest names for our species, resolve any synonyms, etc.

We are building tools into our R package taxize, that will let you check your species names to make sure they are correct.

An important use case is when you have a lot of species. Someone wrote to us recently, saying that they had thousands of species, and they wanted to know how to check their species names efficiently in R.

Below is an example of how to do this.

Install taxize

# install_github('taxize_', 'ropensci') # install the GitHub version, not
# the CRAN version, uncomment if you don't have it installed
library(taxize)

Get some species, in this case all species in the Scrophulariaceae family from theplantlist.org

tpl_get(dir_ = "~/foo2", family = "Scrophulariaceae")

## Reading and writing csv files to ~/foo2...

dat <- read.csv("~/foo2/Scrophulariaceae.csv")

Lets grab the species and concatenate to genus_species

species <- as.character(ddply(dat[, c("Genus", "Species")], .(), transform, 
    gen_sp = as.factor(paste(Genus, Species, sep = " ")))[, 4])

It's better to do many smaller calls to a web API instead of few big ones to be nice to the database maintainers.

## Define function to split up your species list into useable chuncks
slice <- function(input, by = 2) {
    starts <- seq(1, length(input), by)
    tt <- lapply(starts, function(y) input[y:(y + (by - 1))])
    llply(tt, function(x) x[!is.na(x)])
}
species_split <- slice(species, by = 100)

Query for your large species list with pauses between calls, with 3 seconds in between calls to not hit the web service too hard. Using POST method here instead of GET - required when you have a lot of species.

tnrs_safe <- failwith(NULL, tnrs)  # in case some calls fail, will continue
out <- llply(species_split, function(x) tnrs_safe(x, getpost = "POST", sleep = 3))

Calling http://taxosaurus.org/retrieve/90fcd9ae425ad7c6103b06dd9fd78ae2
Calling http://taxosaurus.org/retrieve/223f73b83fcddcb8b6187966963660a8
Calling http://taxosaurus.org/retrieve/72bacdbb8938316e321d4c709c8cdd09
Calling http://taxosaurus.org/retrieve/979ce9cc4dec376710f61de162e1294e
Calling http://taxosaurus.org/retrieve/03a39a124561fec2fdfc0f483d9fb607
Calling http://taxosaurus.org/retrieve/d4bf4e5a1403f45a1be1ca3dd87785d7
Calling http://taxosaurus.org/retrieve/a9a9bdde6fda7e325d80120e27ccb480
Calling http://taxosaurus.org/retrieve/215ccdcf2b00362278bf19d1942e1395
Calling http://taxosaurus.org/retrieve/9d43c0b99b4dfb5ea1b435adab17b980
Calling http://taxosaurus.org/retrieve/42e166f8e43f1fb349e36459cd5938b3
Calling http://taxosaurus.org/retrieve/2c42e4b5227c5464f9bfeeafcdf0651d

# Looks like we got some data back for each element of our species list
lapply(out, head)[1:2]  # just look at the first two

[[1]]
                 submittedName                 acceptedName    sourceId
1        Aptosimum welwitschii                              iPlant_TNRS
2        Anticharis ebracteata        Anticharis ebracteata iPlant_TNRS
3            Aptosimum lineare            Aptosimum lineare iPlant_TNRS
4     Antherothamnus pearsonii     Antherothamnus pearsonii iPlant_TNRS
5 Barthlottia madagascariensis Barthlottia madagascariensis iPlant_TNRS
6         Agathelpis mucronata                              iPlant_TNRS
  score                  matchedName     annotations
1     1        Aptosimum welwitschii                
2     1        Anticharis ebracteata          Schinz
3     1            Aptosimum lineare Marloth & Engl.
4     1     Antherothamnus pearsonii        N.E. Br.
5     1 Barthlottia madagascariensis      Eb. Fisch.
6     1         Agathelpis mucronata                
                                    uri
1                                      
2 http://www.tropicos.org/Name/29202501
3 http://www.tropicos.org/Name/29202525
4 http://www.tropicos.org/Name/29202728
5 http://www.tropicos.org/Name/50089700
6                                      

[[2]]
                     submittedName           acceptedName    sourceId
1 Buddleja pichinchensis x bullata Buddleja pichinchensis iPlant_TNRS
2                 Buddleja soratae       Buddleja soratae iPlant_TNRS
3              Buddleja euryphylla    Buddleja euryphylla iPlant_TNRS
4                  Buddleja incana        Buddleja incana iPlant_TNRS
5                  Buddleja incana                 Incana        NCBI
6                    Buddleja nana Buddleja brachystachya iPlant_TNRS
  score            matchedName        annotations
1   0.9 Buddleja pichinchensis              Kunth
2   1.0       Buddleja soratae           Kraenzl.
3   1.0    Buddleja euryphylla Standl. & Steyerm.
4   1.0        Buddleja incana        Ruiz & Pav.
5   1.0        Buddleja incana               none
6   1.0          Buddleja nana              Diels
                                          uri
1       http://www.tropicos.org/Name/19000333
2       http://www.tropicos.org/Name/19001018
3       http://www.tropicos.org/Name/19000790
4       http://www.tropicos.org/Name/19000596
5 http://www.ncbi.nlm.nih.gov/taxonomy/405077
6       http://www.tropicos.org/Name/19001133

# Now we can put them back together as so into one data.frame if you like
outdf <- ldply(out)
head(outdf)

                 submittedName                 acceptedName    sourceId
1        Aptosimum welwitschii                              iPlant_TNRS
2        Anticharis ebracteata        Anticharis ebracteata iPlant_TNRS
3            Aptosimum lineare            Aptosimum lineare iPlant_TNRS
4     Antherothamnus pearsonii     Antherothamnus pearsonii iPlant_TNRS
5 Barthlottia madagascariensis Barthlottia madagascariensis iPlant_TNRS
6         Agathelpis mucronata                              iPlant_TNRS
  score                  matchedName     annotations
1     1        Aptosimum welwitschii                
2     1        Anticharis ebracteata          Schinz
3     1            Aptosimum lineare Marloth & Engl.
4     1     Antherothamnus pearsonii        N.E. Br.
5     1 Barthlottia madagascariensis      Eb. Fisch.
6     1         Agathelpis mucronata                
                                    uri
1                                      
2 http://www.tropicos.org/Name/29202501
3 http://www.tropicos.org/Name/29202525
4 http://www.tropicos.org/Name/29202728
5 http://www.tropicos.org/Name/50089700
6

Note that there are multiple names for some species because data sources have different names for the same species (resulting in more than one row in the data.frame 'outdf' for a species). We are leaving this up to the user to decide which to use. For example, for the species Buddleja montana there are two names for in the output

data <- ddply(outdf, .(submittedName), summarize, length(submittedName))
outdf[outdf$submittedName %in% as.character(data[data$..1 > 1, ][6, "submittedName"]), 
    ]

       submittedName     acceptedName    sourceId score      matchedName
123 Buddleja montana Buddleja montana iPlant_TNRS     1 Buddleja montana
124 Buddleja montana          Montana        NCBI     1 Buddleja montana
         annotations                                         uri
123 Britton ex Rusby       http://www.tropicos.org/Name/19000601
124             none http://www.ncbi.nlm.nih.gov/taxonomy/441235

The source iPlant matched the name, but NCBI actually gave back a genus of cricket (follow the link under the column uri for Montana). If you look at the page for Buddleja on NCBI here there is no Buddleja montana at all.

Another thing we could do is look at the score that is returned. Let's look at those that are less than 1 (i.e., )

outdf[outdf$score < 1, ]

                        submittedName           acceptedName    sourceId
94   Buddleja pichinchensis x bullata Buddleja pichinchensis iPlant_TNRS
340                Diascia ellaphieae                        iPlant_TNRS
495              Eremophila decipiens                        iPlant_TNRS
500            Eremophila grandiflora             Eremophila iPlant_TNRS
808           Jamesbrittneia hilliard                        iPlant_TNRS
1051                 Verbascum patris              Verbascum iPlant_TNRS
1081             Verbascum barnadesii              Verbascum iPlant_TNRS
1097              Verbascum calycosum              Verbascum iPlant_TNRS
     score            matchedName annotations
94    0.90 Buddleja pichinchensis       Kunth
340   0.98      Diascia ellaphiae            
495   0.98  Eremophila decipiense            
500   0.50             Eremophila      R. Br.
808   0.50         Jamesbrittenia            
1051  0.50              Verbascum          L.
1081  0.50              Verbascum          L.
1097  0.50              Verbascum          L.
                                       uri
94   http://www.tropicos.org/Name/19000333
340                                       
495                                       
500  http://www.tropicos.org/Name/40004761
808                                       
1051 http://www.tropicos.org/Name/40023766
1081 http://www.tropicos.org/Name/40023766
1097 http://www.tropicos.org/Name/40023766

As we got this speies list from theplantlist.org, there aren't that many mistakes, but if it was my species list you know there would be many :)

That's it. Try it out and let us know if you have any questions at info@ropensci.org, or ask questions/report problems at GitHub.

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr.

Open Science Challenge

Tue Jan 08 00:00:00 -0800 2013

Open Science

Science is becoming more open in many areas: publishing, data sharing, lab notebooks, and software. There are many benefits to open science. For example, sharing research data alongside your publications leads to increased citation rate (Piwowar et. al. 2007). In addition, data is becoming easier to share and reuse thanks to efforts like FigShare and Dryad.

If you don't understand the problem we are currently facing due to lack of open science, watch this video:

I just want Data

Another way to look at this challenge is to think about how you can get data more easily. Right now you probably go to a website that has an interface to a database. You do a search, and then download a .csv file perhaps. Then you open it in Excel, and do some pivot tables to get the data in the right format. Only then will you bring the data in to R.

The advantage of using our packages is that they allow you to do that data collection part in a few lines of code. Therefore, you can easily do all those steps in the above paragraph using a few lines of code in one R file. Why does this matter? You can more easily reproduce your own work months later after that summer vacation. In addition, others can reproduce your research more easily.

The challenge

We (ropensci) have just kicked off the rOpenSci Open Science Challenge. If you aren't familiar with rOpenSci, it is a software collective connecting scientists to open science data on the web. Since R is the most popular programming language for life scientists, it made sense to do this in R (instead of Python e.g.).

What is the challenge about?

At rOpenSci, we create R software to make getting open source text from publications and open source data easy. An important result of this is that we are facilitating open science. Why? Because R is an open source programming language, and all of our software is open source. . This challenge asks you to propose a project using one or more of our packages - or perhaps you want to propose a new dataset to connect to R. The rOpenSci core developer team will help you with any problems using our packages, and attempt to modify packages according to feedback from participants. Do you use one or more of our R packages? If you do, great. If not, check out our packages here.

How to apply

Just send an email to info@ropensci.org.

The deadline

January 31, 2013

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr, and knitcitations from Carl Boettiger.

References

Piwowar HA, Day RS, Fridsma DB and Ioannidis J (2007). “Sharing Detailed Research Data is Associated With Increased Citation Rate.” Plos One, 2. http://dx.doi.org/10.1371/journal.pone.0000308.

Is invasive?

Thu Dec 13 00:00:00 -0800 2012

The Global Invasive Species Database (GISD) (see their website for more info here) has data on the invasiveness status of many species. From taxize you can now query the GISD database.

Introducing the function gisd_isinvasive. This function was contributed to taxize by Ignasi Bartomeus, a postdoc at the Swedish University Agricultural Sciences.

There are two possible outputs from using gisd_isinvasive: "Invasive" or "Not in GISD". If you use simplify=TRUE in the function you get "Invasive" or "Not in GISD", but if you use simplify=FALSE you get verbose description of the invasive species instead of just "Invasive" (and you still just get "Not in GISD").

Install taxize from GitHub

# install_github('taxize_', 'ropensci') # install if you don't already
# have the GitHub version
library(taxize)

Make a vector of species

sp <- c("Carpobrotus edulis", "Rosmarinus officinalis", "Nasua nasua", "Martes melampus", 
    "Centaurea solstitialis")

Using the function `gisd_isinvasive` you can query the GISD database for the invasiveness status of your species, at least according to GISD. Calling `gisd_isinvasive` with the second parameter set to default `simplify=FALSE`, you get verbose output, with details on the species.

gisd_isinvasive(sp)

Checking species 1
Checking species 2
Checking species 3
Checking species 4
Checking species 5
Done

                 species
1     Carpobrotus edulis
2 Rosmarinus officinalis
3            Nasua nasua
4        Martes melampus
5 Centaurea solstitialis
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               status
1                                                                                                                                                                                                                                                  You searched for invasive species named Carpobrotus edulis:     1.  Carpobrotus edulis (succulent)        Carpobrotus edulis is a mat-forming succulent native to South Africa which is invasive primarily in coastal habitats in many parts of the world. It was often introduced as an ornamental plant or used for planting along roadsides, from which it has spread to become invasive. Its main impacts are smothering, reduced regeneration of native flora and changes to soil pH and nutrient regimes.\r\nCommon Names: balsamo, Cape fig, figue marine, freeway iceplant, ghaukum, ghoenavy, highway ice plant, higo del Cabo, higo marino, Hottentosvy, hottentot fig, Hottentottenfeige, iceplant, ikhambi-lamabulawo, Kaapsevy, patata frita, perdevy, pigface, rankvy, sea fig, sour fig, suurvy, umgongozi, vyerank\r\nSynonyms: Mesembryanthemum edule L., Mesembryanthemum edulis
2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         Not in GISD
3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            You searched for invasive species named Nasua nasua:1.  Nasua nasua (mammal)             Interim profile, incomplete informationCommon Names: Achuni, Coatí, South American Coati, Tejón
4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       You searched for invasive species named Martes melampus:1.  Martes melampus (mammal)             Interim profile, incomplete informationCommon Names: Japanese Marten, Tsushima Island Marten
5 You searched for invasive species named Centaurea solstitialis:     1.  Centaurea solstitialis (herb)        Centaurea solstitialis is a winter annual that can form dense impenetrable stands that displace desirable vegetation in natural areas, rangelands, and other places. It is best adapted to open grasslands with deep, well-drained soils and an annual precipitation range of 25 to 150cm per year. It is intolerant of shade. Although populations can occur at elevations as high as 2,400 m, most large infestations are found below 1,500 m. Human activities are the primary mechanisms for the long distance movement of C. solstitialis seed. The short, stiff, pappus bristles are covered with barbs that readily adhere to clothing, hair, and fur.  The movement of contaminated hay and uncertified seed are also important long distance transportation mechanisms. Wind disperses seeds over short distances.\r\nCommon Names: geeldissel, golden star thistle, sonnwend-Flockenblume, St. Barnaby's thistle, yellow centaury, yellow cockspur, yellow star thistle\r\nSynonyms: Leucantha solstitialis (L.) A.& D. Löve

Simpler output, just the invasive status.

gisd_isinvasive(sp, simplify = TRUE)

Checking species 1
Checking species 2
Checking species 3
Checking species 4
Checking species 5
Done

                 species      status
1     Carpobrotus edulis    Invasive
2 Rosmarinus officinalis Not in GISD
3            Nasua nasua    Invasive
4        Martes melampus    Invasive
5 Centaurea solstitialis    Invasive

Recology - R

Stashing and playing with raw data locally from the web

First start CouchDB in your terminal

Then start elasticsearch in your terminal

Install sofa

Simultaneously write data to CouchDB along with API calls using the alm package to get altmetrics data on PLoS papers. Ping to make sure CouchDB is on

Create a new database

Write couchdb database name to options

List the databases

Search for altmetrics normally, w/o writing to a database

Search for altmetrics normally, while writing to a database

Make lots of calls, and write them simultaneously

Writing data to CouchDB does take a bit longer

Search using elasticsearch

tell elasticsearch to start indexing your database

Search your database

Using views

Write a view - here letting key be the default of null

Get info on your new view

Get data using a view

Delete the view

What now?

What do you think?

Fylopic, an R wrapper to Phylopic

What is PhyloPic?

What is fylopic?

Info

What can you do with fylopic?

Install fylopic

Plot a phylogeny with silhouettes at the tips

Plot a silhouette behind a plot

What's next?

BISON USGS species occurrence data

Install ribson

Search the BISON database for, of course, bison

Get a summary of the data

Summary by counties (just the first 6 rows)

Summary of states

Map the results

Make some maps! Note that right now the county and state maps just plot the conterminous lower 48. The map of individual occurrences shows the lower 48 + Alaska

When plotting occurrences, you can pass additional arguments into the bisonmap function.

For example, you can jitter the points

And you can specify by how much you want the points to jitter (here an extreme example to make it obvious)

Let us know if you have any feature requests or find bugs at our GitHub Issues page.

Scholarly metadata in R

Install rmetadata

Count OAI-PMH identifiers for a data provider.

Lookup article info via CrossRef with DOI and get a citation.

As Bibtex

As regular text

Search the CrossRef Metatdata for DOIs using free form references.

Search with title, author, year, and journal

Get a DOI and get the citation using \code{crossref_search}

Get a random set of DOI's through CrossRef.

Search the CrossRef Metatdata API.

Get a short DOI from shortdoi.org.

Get a record from a OAI-PMH data provider.

List available metadata formats from various providers.

Some plotting - mean number of authors per paper

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr, and knitcitations.

Visualizing rOpenSci collaboration

Install some functions for interacting with the Github API via R

Get all repos for an organization, here ropensci of course

Get commits broken down in to additions and deletions, though below we just collapse them to all commits

Plot commits by date and repo

In addition, there are quite a few people that have committed code now to rOpenSci repos, calling for a network vizualization of course.

Getting a simple tree via NCBI

Get a species list

Go to the site

Choose file

Add the species list to make the tree

Download

Plot the tree on your machine

w00p, there it is...

And the answer is NO to the question: Is there an API for Common Tree?

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr.

Waiting for an API request to complete

Dealing with API tokens in R

When plotting occurrences, you can pass additional arguments into the `bisonmap` function.

Using the function `gisd_isinvasive` you can query the GISD database for the invasiveness status of your species, at least according to GISD. Calling `gisd_isinvasive` with the second parameter set to default `simplify=FALSE`, you get verbose output, with details on the species.