Analysis of ArXiv.org database of scientific papers

MSAN 694 - Distributed Computing
Team: D. Wen, A. Romriell, J. Pastor, J. Pollard

Data Description:

Source: ArXiv Electronic Archive of Scientific Papers
We analyzed the entire database of arXiv.org (1.6GB):

• 1.26 million papers
  
• 600,000 authors
  
• 86,262,827 words
  

Goal:

Exploratory Data Analysis and Community Detection, implemented with three different technologies (postgreSQL, Hive and Spark) for performance comparison.

Experimental environment:

Local:

MacBook Pro 2.7 GHz Intel Core i5 16 GB 1867 MHz DDR3

Distributed:

4-node cluster of r3.xlarge (160GB) emr-4.6.0
Hadoop distribution: Amazon 2.7.2
Applications: Hive 1.0.0, Pig 0.14.0, Spark 1.6.1

Summary results:

The following table summarizes the running time (in seconds) of the tasks in each of the different platforms (postgreSQL, Hive and SparkSQL): As the queries grew in complexity (4 and 5), Hive and SparkSQL perform drastically better than PostgreSQL. In particular, we observed a reduction in running times between 80% and 97% using SparkSQL.

We concluded that - in the context of this problem - SparkSQL was the optimal tool given its speed, ease of use and flexibility.