This is a simple experiment by Jason Long and Bryce Thornton to test how many real-life, publicly available datasets satisfy Benford's Law.
If you find this to be an interesting idea, we'd encourage you to help add more datasets to the site. We've intentionally kept the site as simple and lightweight as possible. There is no real backend - the data has been crunched in advance and the results are simply entered into JSON files.
To contribute a new dataset, you'll need to do two things:
The format of js/datasets/index.json is simply a key/value pair:
{
"twitter-users-by-followers-count": "Twitter users by followers count",
"distance-of-stars-from-earth-in-light-years": "Distance of stars from Earth in light years",
"loan-amounts-on-kiva-org": "Loan amounts on kiva.org",
"total-number-of-print-materials-in-us-libraries": "Total number of print materials in US libraries",
"population-of-spanish-cities": "Population of Spanish cities"
}
Add your new file in the /js/datasets/ directory with a name that matches the key used in step 1. The format looks like this:
{
"values": {
"1": 32.62,
"2": 16.66,
"3": 11.80,
"4": 9.26,
"5": 7.63,
"6": 6.55,
"7": 5.76,
"8": 5.14,
"9": 4.56
},
"num_records": "38,670,514",
"min_value": "1",
"max_value": "4,706,631",
"source": "http://www.infochimps.com/datasets/twitter-census-twitter-users-by-friends-count"
}
It's important to include the source of the data used so that others can verify and reproduce the results.
Generating Benford stats is a fairly straightforward process. We've made a simple ruby class for you to use if you'd like.
First, grab a copy of the class from here: https://gist.github.com/1044174
Second, include the class in your script like so:
require 'benford_counter'
require 'rubygems'
require 'csv'
counter = BenfordCounter.new
CSV.foreach("spain.txt") do |row|
counter.count(row[9])
end
counter.resultsfweez contributed the Linux filesize dataset and created a Python script for tallying filesizes in a directory.
We're using CoffeeScript for the Javascript and Sass/Compass for the CSS.
Once CoffeeScript is installed (see the CoffeeScript docs), run this command from the project root to observe and compile changes:
coffee --watch -o js/ --compile js/coffee/*.coffee
Note that the only file that should be edited is /js/coffee/app.coffee. The /js/coffee/app.js file is generated by CoffeeScript.
To make changes to the CSS, you need to install Sass and Compass (see the Compass docs. Then edit /css/sass/screen.scss. You observe and compile changes by running this command from the project root:
compass watch
To compile a production-ready compressed version:
compass compile --output-style compressed --force