{"id":117,"date":"2014-04-12T04:02:43","date_gmt":"2014-04-12T04:02:43","guid":{"rendered":"https:\/\/krushton.com\/blog\/?p=117"},"modified":"2023-05-11T23:44:43","modified_gmt":"2023-05-11T23:44:43","slug":"a-statistical-analysis-of-rprogresspics","status":"publish","type":"post","link":"https:\/\/krushton.com\/blog\/a-statistical-analysis-of-rprogresspics\/","title":{"rendered":"Visualizing Reddit Data in IPython Notebook"},"content":{"rendered":"
\n

In my last\u00a0semester\u00a0at the I School\u00a0I took an introductory course in\u00a0data analysis using Python. I was pretty\u00a0unfamiliar with statistics prior to the course and am still very much an amateur data scientist, but the course gave me just enough skills to be\u00a0brave (perhaps foolishly so) in the face of an unruly data set.<\/p>\n

Since the end of the semester I\u2019ve found myself going back to these tools frequently, because they enable me to actually follow up on the random questions\/curiosities\/whims I get from time to time using public datasets and a few lines of code.<\/p>\n

What is r\/ProgressPics?<\/strong><\/p>\n

For this post I pulled data from a subreddit called Progress Pics<\/a>. Progress pics is a place where people who are working on some form of body transformation go to post before and after pictures of themselves. I\u2019ve been trying to improve my fitness lately so I’ve been frequenting the sub on and off for about six months.<\/p>\n

It caught my eye as a potential source of interesting\u00a0analysis because, unlike most subreddits, the community at progress pics has a title post format that encourages the use of\u00a0structured data.<\/strong>\u00a0Posters who are providing pictures are instructed to include their gender, age, height, start weight, and end weight in the post title, in a format that looks something like:\u00a0F\/23\/5\u20195\u2033 [189lbs > 169lbs = 20lbs].\u00a0<\/strong><\/p>\n

The more structure you add to a blob of text, the easier it is to understand programmatically,\u00a0so this data seemed like a good opportunity for analysis.<\/p>\n

Method<\/strong><\/p>\n

This analysis was done in\u00a0IPython notebook, using\u00a0data analysis packages Pandas, numpy, and matplotlib. I used the Reddit\u00a0API<\/a> to pull in as many posts from the subreddit as I could before the API complained, and ended up with a dataset of about 1600 posts. The Reddit API provided post metadata including title, number of comments, number of votes, date\/time, and more. Then, after some initial data cleanup, I used a series of regular expressions to extract the poster\u2019s gender, age, height, start weight, and end weight from the post title.<\/p>\n

After being filtered through these regexes, about half \u00a0(863) of the posts had good data for all of these metrics, so I dropped the remaining ones as this is a sufficiently large sample.<\/p>\n

About the Data Set<\/strong><\/p>\n

So, who are the posters of r\/ProgressPics? Some quick facts:<\/p>\n