Saturday, February 23, 2013

Simulating Population Growth in Cities Using R

R is great for anyone who wants to get started on learning Simulation. (Both Discrete Event or Agent-based, with stochastic elements in the process.)

This post is inspired by Matt Asher's "quick-and-dirty" R simulation work on Population Growth. Matt uses it to create aRt. I felt that his core idea provided a very good framework to introduce various concepts of simulation.

The Basic Idea behind this Simulation

We take an area (it could be a country, state, city) that is made up of units (cells) that can be settled (occupied).
We allow the initial 'pioneers' to settle first. (Seeding the cells)
We then establish some rules for future residents to settle. (Settling Rules)
Plot the area, and collect some stats to study the growth of the population under different starting conditions and different settling rules.

Pseudo-Code for the Simulation

Define parameters: Area Width, Height
Number of Pioneers P and future settlers S

for each pioneer p{ 
  settle p according to seeding rules in a valid cell in the area
}
for each settler S{
  Find an unoccupied Cell.
  Settle S in that cell, if the settling criteria permits it
}
Print Iterations Stats
Print out the Plot of the Area

A Few Changes from Matt's R code

  1. I switched from his matrix to a dataframe. I also switched from his image to ggplot.
  2. Delineated the initial parameters, seeding and settling clearly so that anyone who downloads the code can experiment
  3. Created a few different Seeding Functions for the pioneers (beyond random seeding)
  4. Matt allowed settling if the new cell was adjacent to an occupied cell. I played around with a few different population settling functions.  (rectilinear, diagonal etc).
  5. Added a function to collect a few statistics regarding population growth
  6. Ran the simulation a few 100 times (without plotting) to compare different growth and settling schemes

Color Coding the Population

  • Black cells : Unoccupied
  • Blue cells: Pioneers
  • Orange cells: Settlers who found a cell in 1-2 attempts
  • Yellow cells: Settlers who found a cell after more than 2 scouting attempts
The full R code I used for these R can be found here.

Here are some runs with a 30x30 area.

Here's what one run looked like. (Blue cells are the initial pioneers. The other colors represent the number of steps taken for them to find a home cell.)

Trying Out Different Seeding Schemes (for the pioneers)


As opposed to starting out with randomly strewn cells, what if we started out with a city downtown (a densely populated central rectangle) and let the settlers follow them?



I tried a rectangluar ring (annular cells for pioneers) starting point to see how the population grew inside and outside. So I used this seeding function:



One final seeding scheme I tried was to have two thick bands (columns of pioneer cells) and to let the population grow from there. Seeding function:

 

 

 

 

 

 

Different Population Growth Schemes

The default settling rule that Matt Asher uses is a very good one, one that I like. If a given cell is adjacent to any occupied cell (one of its 8 neighbors) then it becomes a valid cell to settle down in.

Just for experimentation, I tried a few other schemes.

1. Rectilinear Growth

Rule: You can only settle down in the 4 cells that are due N, S, East or West of an occupied cell.

2. Diagonal Growth

Rule: A settler can occupy any cell that is diagonally adjacent to another occupied cell.

The code can be found here.

 

 

 

 

 

 

Collecting Stats

Now that we have all the functionality that we wanted, it is time to start collecting some statistics for each run ('replication'). There are many possibilities for what we could collect in a population simulation.
Here are a couple:
  • How many settlers found a 'home cell?'
  • How many scouting attempts did each settler make before they found a home?

Making Multiple Replications

 


>st
  Iter FoundHome NumSettlers  Percent
1    1       243         350 69.42857
2    2       214         350 61.14286
3    3       213         350 60.85714
4    4       258         350 73.71429
5    5       235         350 67.14286

The full R code I used for these runs can be found here. Feel free to give it a spin and try out your own settling/growth schemes.
Reference: Matt Asher's post on his StatisticsBlog

Wednesday, December 7, 2011

A pre-requisite to be a Data Scientist

So what should be in the toolkit of people who call themselves a data scientist?

A fundamental skill is the ability to manipulate data. A data scientist should be familiar and comfortable with a number of platforms and scripting tools to get the job done. What is difficult in Excel might be trivial in R. And when R struggles, you should switch to Unix (or use a programming language such as Python) get that portion of the data munging done. Along the way, you pick up a lot of tips and tricks. For example: how to read a big datafile in R?

The goal is to get the job done. Familiarity with a wide variety of tools, and expertise in some is the hallmark of any good would-be data scientist.

Friday, December 2, 2011

O'Reilly's Data Science Kit - Books

It is not as if I don't have enough books (and material on the web) to read. But this list compiled by the O'Reilly team should make any data analyst salivate.

http://shop.oreilly.com/category/deals/data-science-kit.do

The Books and Video included in the set are:

  1. Data Analysis with Open Source Tools
  2. Designing Data Visualizations
  3. An Introduction to Machine Learning with Web Data (Video)
  4. Beautiful Data
  5. Think Stats
  6. R Cookbook
  7. R in a Nutshell
  8. Programming Collective Intelligence

Wednesday, November 30, 2011

Tips for getting started on Kaggle (datamining)

Ever since I heard about Kaggle.com at this year's Bay Area Data Mining Camp, I've wanted to participate. But I was feeling somewhat intimidated.
Jeremy Howard's "Intro to Kaggle" talk at yesterday's MeetUp (DataMining for a Cause) was exactly what I needed.
He had a number of tips for beginners. His was exactly the talk that I was looking for, though I didn't know it. I am sharing some of his tips here, in case it helps others as well.

Jeremy Howard's Tips for Getting Started on Data Mining competitions at Kaggle

* Visit the Kaggle site and spend at least 30 minutes every day hanging around. Read the forum, the competition pages, and read the Kaggle blog
* It is much better to start participating in competitions which are just starting up, rather than in ones where there are 100s of entries and teams already well on their way
* Aim to make at least one submission each and every day
* Jeremy himself participates in competitions to see where he stands, and to learn and get better
* He'd start out making trivial submissions (all zero's, or alternate zero's, all entries as averages) until his algorithm got better
* A lot of people who compete use R (and SAS, Excel or Python)
* Nearly 50% of the winning entries use Random Forest techniques.
* If you place in the top 3, that is great. But personal improvement and learning should be the goal.
* As you get better, you might get invited to "private competitions."
* Every day, strive to do a little better and improve your submission's performance, scoring and ranking

Related Links:

Tuesday, October 18, 2011

Fusion Tables by Google

Google's Fusion Tables look impressive, for those who want to try geo-visualizations of their data. You don't need much programming experience to be able to use it.

For those who want to try it out, here's a nice intro that Kathyrn Hurley presented at the recent SVCC (Silicon Valley Code Camp). When combined with ShpEscape (note spelling) it becomes very powerful.

The Guardian (UK), Texas Tribune, and WNYC seem to be organizations that are taking advantage of it.
I'll post a couple of their examples soon. If you have a Google account, it's easy to test out Fusion Tables.

Related Link: Journalist’s guide to mapping data by county, district using ShpEscape

Monday, October 17, 2011

Get the Basics right - Suggestion for R Beginners

I am always looking for suggestions on how to get better at R, esp. for beginners. So when I see someone who's gotten adept at it, I ask them how they got there.

This weekend, at the Bay Area ACM Data Mining Camp, one person gave me what seemed like a good suggestion. Just wanted to share it here, for anyone else who's just getting started.

He told me that there are tons and tons of libraries, and if you start going down that path, you might know how to use a library or two, but you may not learn the basics of data manipulation, which is one of R's main strengths.

His suggestion:
Get some data and learn to manipulate it - understand the differences between vectors, data-frames, arrays and matrices. Once you have this down, only then should you start exploring the different libraries. Don't rush in to try every new library that someone praises.

Sunday, October 16, 2011

Geo-doodlers - Paul Butler and FlowingData

I found this great R-Visualization example via an R-Blogger post that xingmowang made. (One more good reason for why it is important to read lots of field-related blogs!)

Here's the image:

If this was merely eye-candy, I would have enjoyed it, but not included it here. But to think that this was done in R -- that means the rest of us can learn from it!

When Paul Butler writes about how he created it, he shares with us how he had to tweak it, and how the results surprised him. That is true data-doodling. You guide things along, but then the data surprises (or delights) you.

I also like this small bit of musing that he includes:
What really struck me, though, was knowing that the lines didn't represent coasts or rivers or political borders, but real human relationships. Each line might represent a friendship made while travelling, a family member abroad, or an old college friend pulled away by the various forces of life.
For those of us who are new to R, this example has a few things to try. Take any dataset with Lat/Long values in it, and plot it over a world map. Once you can do that successfully, try this.
(Also pointed out courtesy of Xingmowang.)

We may not all create infographics that are great, but these examples will point us in the right direction.