Danielle Yoseloff

Qualitative Data

2017-04-27T01:44:03+00:00

The data set we will be working with is an online resource from Professor James Sidanius at UCLA. It contains information from 400 students’ faculty evaluations. The data set includes 10 variables corresponding to the scores for each evaluation question, Sex of instructor (1 for male 2 for female), Salary in thousands of dollars, Years of teaching experieence, and Number of students in the class being evaluated. Also make a note that the data is clean and has no missing entries.

By the end of our analysis we’d like to be able to answer two questions:

Do faculty with more experience receive better reviews?
Do faculty with smaller class sizes receive better reviews?

Let’s take a look at the evaluation questions and try to summarize the information. Before we get into any calculations, it’s nice to notice every question is scored the same way aka a 1 is always the worse score and a 5 is always the best. This will make computaions less complicated.

Our goal in summarizing is to capture the attitude students have towards their professor. The first step is to perform a principal component analysis so we can determine how many components are needed to fully summarize the 10 questions.

To perform principal component analysis in R, we will be using two packages: ‘psych’ and ‘GPArotation’. Be sure to install these packages if this is your first time using them.

attach(data)
library("psych")
library("GPArotation")
summary(princomp(~item13+item14+item15+item16+item17+item18+item19+item20+item21+item22, cor=TRUE))

By convention, a component is said to still contain an important amount of information when the eigenvalue is greater than 1. Note: R gives ‘standard deviation’ which is the square root of the eigenvalue but gives the proportion and cumulative proportion on the eigen scale.

Squaring the standard deviations gives us the eigenvalues for each component: Comp.1= 5.27, Comp.2= 1.21, Comp.3= 0.67. Because component three has an eigenvalue less than 1, we will move forward using only 2 components to explain the data. The cumulative proportion values explain how much of the data is captured. Components one and two capture 64.8% of the information in the ten variables. Components one, two, and three capture 71.6% of the information in the ten variables.

The second step is to perform a varimax(orthogonal) rotation. Rotation redefines the factors in a way that gives more interpretable results. Orthogonal rotation is the most common type.

factorvariables <- data.frame(item13,item14,item15,item16,item17,item18,item19,item20,item21,item22)
prin <- principal(factorvariables,nfactors=2,rotate="varimax"scores=TRUE)
prin

The important part of the output is the RC1 and RC2 columns. These numbers indicated which questions should be grouped together in the same component. When these numbers are greater than 0.5, an item is significant in the said group.

So, our first component will be composed of items 13,14,15,16,and 17. The second component will be composed of items 18,19,20,21,and 22. Let’s take another look at what questions these items refer to and identify a theme for the components.

It looks like Component 1 is describing quality of the instructor and Component 2 is describing technique of the instructor.

Now we’re ready to answer our goal questions: how do class size and experience effect performance ratings? We can do this by running correlation tests between the components’ scores and the variables we are interested in (yrsteach and nstudents).

data2 <- cbind(data,prin$scores)
cor(data2$RC1,data2$yrsteach)
cor(data2$RC2,data2$yrsteach)
cor(data2$RC1,data2$nstudents)
cor(data2$RC2,data2$nstudents)

We see that an instructors’ years of experiance has no significant correlation with either component group. So, years of experience does not influence how students perceive their teaching. The correlation between number of students in a course and component group 2 is significant but weak. It indicated that as class sizes increase, instructors have lower ratings on questions relating to their technique. This is in line with expectations, in larger classes students tend to feel they are not able to ask question and that the professor is less accessible outside of class. As for component two and class size there is no significant correlation.

Analyzing Utah Housing Market

2017-04-18T01:44:03+00:00

For this time series data I explore all paths to select the model that best predicts the market. This includes several methods of model selection, model comparison, and forecasting.

Read document here

Forecasting Time Series Data

2017-03-10T01:44:03+00:00

Let’s look at the time series dataset in R called co2. This dataset measures monthly Mauna Loa atmospheric CO2 concentration from 1959 to 1997 totaling 468 observations.

We will be forecasting the data using three methods then comparing the methods of forecast. How do you compare methods or their accuracy? We will slip the data up into two parts. The first 1 thorough 444 observations will be called the training data. This is the portion of the data we will base our forecasting on. The last 5% of the data (i.e. the last 24 observations) is called the test data. The training data will be used to predict the last 24 observations, then predicted values and test data can be compared to measure accuracy of the model. For what we want to do we need additional packages. If you have never before worked with them, remember to install the packages before you can load them into your library.

library(TSA)
library(fArma)
library(forecast)
train <- co2[1:444]
test <- co2[445:468]

Let’s take a look at the data.

plot.ts(co2)
par(mfrow=c(1,2))
acf(co2)
pacf(co2)

The time series data has a positive linear trend. There are no issues with heteroscedasticity. The constant up and down pattern indicates a seasonal trend of period 12.

This is a unique ACF PACF combination that indicates nonstationarity. The ACF decays to zero at a very slow rate while the PACF is only really significant at lag 1. We expected to see signs of nonstationarity because of the non-constant mean we previously noted.

Forecast 1

The first method of forecasting we will try is called subset selection method. This is a computer method where we input the maximum AR and MA orders we believe may explain the model, here we will choose 15 for both. The computer program then runs through all possible ARMA model combinations of those coefficients and returns the most probable models based on their BIC values.

sub1 <- armasubsets(diff(diff(train,12)),nar=15,nma=15)
plot(sub1)

Based on our output, we believe the best model is an ARIMA(9,1,12). The 1 indicates that we applied a differencing to the data. The AR component is 9 because that is the highest order coefficient however there is also a nonzero coefficient value for phi 1. Similarly, the MA component is 12 because that is the highest order coefficient however there is also a nonzero coefficient value for theta 2. The Arima function below gives us the values for the coefficients of the ARIMA model.

arma1 <- Arima(train,order=c(9,1,12), seasonal=c(0,1,0),fixed=c(NA,rep(0,7),NA,0,NA,rep(0,9),NA))
arma1

We can see below that the forecast isn’t doing a very good job at following the pattern of the data in the prediction. Additionally the confidence intervals are very large. While all of the test data points are in the respective confidence intervals, this model does not look ideal.

Forecast 2

The next method of forecasting will be to identify potential SARIMA models from the ACF and PACF. Then fit the candidate models and compare their AICc values to choose a final SARIMA model to run the forecast on.

We need to look at the ACF/PACF of the differenced data.

ddtrain <- diff(diff(train),12)
par(mfrow=c(1,2))
acf(ddtrain,lag.max=40,main="Differenced co2")
pacf(ddtrain,lag.max=40,main="Differenced co2")

Both the ACF and PACF decay to zero for the first few lags which indicates that possible models could be AR(1) or MA(1) or ARMA(1,1). As for the seasonal component, the ACF cuts off after lag 1 (aka lag 12 but this is the first seasonal lag) and the PACF decays to zero at the seasonal lags (every twelfth lag). These suggest an MA(1) model is appropriate for the seasonal component.

Arima(ddtrain, order=c(0,1,1), seasonal=c(0,1,1))
Arima(ddtrain, order=c(1,1,0), seasonal=c(0,1,1))
Arima(ddtrain, order=c(1,1,1), seasonal=c(0,1,1))

Of the three models we tested, SARIMA(1,1,1)x(0,1,1)[12] performed best with the lowest AICc value of 374.47. Below we see the model is not very good at predicting the testing data. The model does not keep the same seasonal pattern and is not even behaving in a positive linear manner. The confidence intervals are also very large.

Forecast 3

The third method of forecasting is called a Holt-Winters seasonal forecast. We specify an additive seasonal trend instead of multiplicative because there is no heteroscedasticity. The “gamma = T” condition specifies that there is a seasonal component to address. The Holt-Winters is a form of double exponential smoothing that is good to use when there is a linear trend and/or a known seasonal component.

hw <- HoltWinters(ts(train,frequency=12),gamma = T, seasonal = "additive")
forecast(hw, h=24)
plot(forecast(x, h=24))

The Holt-Winters is a good predictor. The trend and seasonality patters are consistent with the previous data and the confidence intervals are narrow. Additionally, all 24 testing data points lie within the 95% prediction interval.

Comparing Forecastings

To compare the forecast, we will use two common criteria that quantify how much error the model produced. The lower the RMSE and MAPE the better the model was at predicting the testing data.

HWfcast1 <- forecast(arma1,h=24)
HWerr1 <- test-HWfcast1$mean
HWrmse1 <- sqrt(mean(HWerr1^2))
HWmape1 <- mean(abs((HWerr1*100)/test))
HWrmse1
HWmape1

ff2 <- Arima(train, order=c(1,1,1), seasonal=c(0,1,1))
HWfcast2 <- forecast(ff2,h=24)
HWerr2 <- test-HWfcast2$mean
HWrmse2 <- sqrt(mean(HWerr2^2))
HWmape2 <- mean(abs((HWerr2*100)/test))
HWrmse2
HWmape2

ff3 <- HoltWinters(ts(train,frequency=12),gamma = TRUE,seasonal="additive")
HWfcast3 <- forecast(ff3,h=24)
HWerr3 <- test-HWfcast3$mean
HWrmse3 <- sqrt(mean(HWerr3^2))
HWmape3 <- mean(abs((HWerr3*100)/test))
HWrmse3
HWmape3

As we expected the Holt- Winters forecasting was the best. Next, the ARIMA(9,1,12) model performed slightly better than the SARIMA(1,1,1)x(0,1,1)[12] model. A second look at the forecasting graphs for these two models confirms their performances. The SARIMA model had no trend or seasonal pattern while the ARIMA model did have a slight seasonal pattern in its prediction.

Best Model for the Data

2017-02-13T01:44:03+00:00

It can be complicated to assess the best way to model data. How can you know if your model is good? How can you know what steps to take to make it better? Did you really make it better? In our exploration of these questions we will come across concepts such at interaction variables, residual plots, outliers, and collinearity.

I’m working with a csv file which I read into my R document and named ‘data2’. It contains 100 observations with the variables y, age, sexf, x1, x2, and x3. Our goal is to come up with a model that is very good at predicting y.

First, let’s look at how good the go-to regression is. That is, y regressed on all the independent variables.

two <- lm(y~ age + sexf + x1 + x2 +x3, data2)
summary(two)

The Rsquared is .081 which is quite low indicating the model is not a good predictor of y. Let’s also note, we observe age and x1 to be the significant variables.

Step 1: Linearity

One of the fundamental assumptions of a linear model is the assumption that all variables are linear. Let’s check by looking at plots of all independent variables graphed against the residuals.

residuals <- resid(two)
plot(residuals~age,data2)
plot(residuals~sexf,data2)
plot(residuals~x1,data2)
plot(residuals~x2,data2)
plot(residuals~x3,data2)

For the regression plots to be linear we want to see a random scatter of the observations. The plots for age, x1, and x3 look random, so we can assume they are linear. The residual plot for sexf looks a bit different but is also random when we think about it. The variable for sex is either 0 or 1, so given that the observations can only be at those two marks on the axis we are really looking for a random spread on the y axis. And yes, there doesn’t appear to be a cluster of the data at any point of the y axis. Now finally we have the residual plot for x2. It appears to be following the trend of a negative parabola. This is a red flag that the x2 variable is nonlinear.

One way to compensate for this linearity problem is by adding another variable to the model that is the square of the nonlinear variable. To do this, I created a new column in our data set that is x2squared, then added that term in our regression. Additionally, I run a summary of our model so we can assess it.

data2$x22 <- data2$x2^2
three <- lm(y~ age + sexf + x1 + x2 +x3 + x22, data2)
summary(three)

Our Rsquared value jumps to 0.787. This model is a much better fit than our previous model without the x2squared variable. Now we see that x2 and x2squared are the significant variables.

Step 2: Outliers

We can improve a model by checking for outliers and removing any we find significant. Let’s look at two criteria for determining outliers. The first is studentized residuals which follow a t table with n-k-1 degrees of freedom (n=sample size, k= number of regressor in model). In our case n=100 and k=6. our output will give us studentized residual values for each observation but these can be hard to read so, let’s also graph it.

studentr <- rstudent(three)
student r 
plot(studentr)
points(7,studentr[7],col="red")

It’s easy to see one point which is significant. I found the outlier point to be observation 7 because I could see it was around 10, then played around with coloring different observations on the graph until the outlier light up red. This was an easy way to get around searching through the list of numbers in the R output

The second way to identify outliers is by calculating the Cook’s distance for each observation. Influential point for Cook’s distance will have a value greater than 4/n. In our case, we’re looking for values greater than 4/100 or .04.

cooks.distance(three)
cook <- cooks.distance(three)
plot(cook)
points(7,cook[7],col="red")

Similar to the studentized residuals we see observation 7 is an outlier.

After confirming observation 7 is an outlier the next step is to remove it. To do this we will be creating a new dataframe called ‘newdata’ which is a subset of our data. It will have all 8 of our variables but only 99 observations.

newdata <- data2[-c(7),]

Let’s see if removing the outlier improved our model

m <- lm(y~ age + sexf + x1 + x2 +x3 + x22, newdata)
summary(m)

Yes, removing the outlier increase the Rsquared of the model to .9871 which is very high. We still see that only variables x2 and x2 squared are significant.

Step 3: Collinearity

Variance inflation factor(VIF) is a value assigned to each independent variable in a model to describe collinearity. A high VIF value means that variable is highly correlated with another variable in the model. We should expect that x2 and x2squared will have high VIF values because they are highly correlated. But, let’s check if any other variables in our model are correlated.

vif(m)

It appears age and x1 are also highly correlated. When two variables that are highly correlated are in the model it can create problems. Because both are present neither would seem like an important contribution. However, when one is not present the other may be highly significant. So, let’s remove x1 from our model and the updated model.

m5 <- lm(y~ age + sexf + x2 +x3 + x22, newdata)
summary(m5)

Our Rsquared value has stayed just as strong as before at .9871 so our model is good at predicting the data. The difference is now, x2 x2squared, and age are significant predictors. At this point we can be very satisfied with the model we have chosen to predict our data.

Choosing a Time Series Model

2017-01-19T01:44:03+00:00

In this post I will look at common graphs to analyze a time series dataset covering topics such as stationarity, transformation, ACF plots, and PACF plots.

Let’s look at the dataset built in R called sunspot.year. It is a time series dataset that records the yearly number of sunspots from 1700 to 1988.

First, we should take a look at the data.

plot.ts(sunspot.year,ylab="Number of Sunspots")
title(main = "sunspots.year")

It looks like as the years go by, the average number of sunspots is increasing. In other words, there is a positive linear trend, which can also be preferred to as a non-constant mean. The variance also appears to be increasing over time. The non-constant mean and non-constant variance are both indicators that the data is non stationary.

Time series data needs to be made stationary before we can judge what model to fit the data with. A way to correct for a non-constant mean is to apply a square root transformation to the data. So, let’s do this and reassess the plotted data

newsunspot=sqrt(sunspot.year)
plot.ts(newsunspot, ylab= "Square Root Number of Sunspots")
title(main = "Transformed sunspots.year")

The transformed data now appears to have a constant variance around 6. Additionally, there are no obvious issues with the variance. The transformation was a success because the data now looks stationary.

Now that the data has been corrected to behave stationary, we can identify a model looking at the autocorrelation(ACF) and partial autocorrelation functions(PACF). Keep in mind only the ACF and PACF graphs obtained from the transformed data are useful because they have been corrected. Note, the first line of code displays the graphs side by side for convenience.

par(mfrow=c(1,2))
acf(newsunspot, main="Transformed sunspots.year")
pacf(newsunspot, main="Transformed sunspots.year")

The ACF corrosponds to the MA (moving average) component of the model, while the PACF corrosponds to the AR (autoregressive) component. Models be composed of both components or just one.

When the ACF graph has all lags significant while decaying to zero, this indicates an MA(0) component. Likewise, if the PACF was significant at all lags and decaying to zero, this would indicate AR(0). If both decay to zero inthis manner it would suggest a lower order ARMA model. But here we believe an MA(0)compnent. The PACF of our data is highly significant at lag one and two but becomes insignificant at lag three onward. Because the PACF “cuts off” after lag 2 we believe an AR(2) component is the best fit.

So, in this case the graphs indicate that the data follows an ARMA(2,0) also expressed as AR(2).

When Autocorrect Fails

2016-12-13T01:44:03+00:00

During a time when autocorrect on our phones and computers carries a heavy load, sometimes poor grammar slips through the cracks. So, where in the U.S. has the worst spelling on Twitter? To explore this question, I looked at five major cities which represent different regions of the country. To have the “worst spelling” is a bit broad so I choose it to mean the percentage of time a city misspelled one of the top four most commonly misspelled words with its most common misspelling.

I will use data analytics processes in conjunction with the following tools: Rstudio, twittR, wordcloud, ggplot2.

A special note, by ‘data analytics processes’ I mean methods of how I extracted the data, choose what data to use and discard, and the best ways to visually display the data so it is easiest to interpret.

The first step is to set up the user API

consumer_key <- "XXXXXXXXXXXXXXXXX"
consumer_secret <- "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
access_token <- "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
access_secret <- "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
setup_twitter_oauth(consumer_key, consumer_secret, access_token, access_secret)

Next, I setup the correct and incorrect word spellings

correct <- c("their", "a lot", "received", "separate")
misspelled <- c("thier", "alot", "recieved", "seperate")

And the cities I want to study

cities <- data.frame(matrix(ncol = 2, nrow = 0))
colnames(cities) <- c("name", "geocode")
cities <- rbind(cities, data.frame(name="Boston", geocode="42.3601,-71.0589,30mi"))
cities <- rbind(cities, data.frame(name="Los Angeles", geocode="34.0522,-118.2437,30mi"))
cities <- rbind(cities, data.frame(name="Chicago", geocode="41.8781,-87.6298,30mi"))
cities <- rbind(cities, data.frame(name="Houston", geocode="29.7604,-95.3698,30mi"))
cities <- rbind(cities, data.frame(name="Atlanta", geocode="33.7490,-84.3880,30mi"))

Now let’s obtain a collection of tweets then calculate the percent of tweets misspelled per city and per word so we get a total of 20 outputs like “Boston misspells their with thier 2 percent of the time”

count <- 100
data_txt <- list()
results <- list()
apply(cities, 1, function(city) {
  for(wordIndex in 1:length(correct)) {
    searchTerm <- paste(paste(correct[wordIndex], "OR", misspelled[wordIndex], sep=' '), '-filter:retweets', sep=' ')
    tweets <- searchTwitter(searchTerm, n=count, geocode=as.character(city['geocode']), resultType="recent")
    sapply(tweets, function(x) {
      index <- length(data_txt) + 1
      data_txt[[index]] <<- x$text
    })
    incorrectPercentage <- 100 - length(Filter(function(x) grepl(correct[wordIndex], tolower(iconv(x$text, to="utf-8-mac")), fixed=TRUE), tweets))/count*100
    results[[length(results) + 1]] <<- c(city = city['name'], percent = incorrectPercentage)
    print(paste(city['name'], "misspells", correct[wordIndex], "with", misspelled[wordIndex], incorrectPercentage, "percent of the time"))
  }
})
data_txt <- unlist(data_txt)

Now that we have some data to work with let’s display it with a word cloud. We don’t expect there to be a theme in all of the tweets we collected but it’s worth a look. Just clean up the tweets a bit so the material extrapolated for the word cloud looks uniform and is interesting. (removing numbers and whitespace, taking out capitals and punctuation, removing stopwords like ‘the’ ‘a’ ‘to’ etc.)

data_txt <- iconv(data_txt,to="utf-8-mac")
data_corpus <- Corpus(VectorSource(data_txt))
data_clean <- tm_map(data_corpus, removePunctuation, lazy=TRUE)
data_clean <- tm_map(data_clean, content_transformer(tolower), lazy=TRUE)
data_clean <- tm_map(data_clean, removeWords, stopwords("english"), lazy=TRUE)
data_clean <- tm_map(data_clean, removeNumbers, lazy=TRUE)
data_clean <- tm_map(data_clean, stripWhitespace, lazy=TRUE)
wordcloud(data_clean, scale = c(4,1), max.words = 200, random.order = F, colors=brewer.pal(10,"Paired"), vfont=c("script", "plain"))

Here is one of the word clouds produced by the code. Keep in mind the word cloud is randomly generated each time the code is run.

Lastly, let’s look at a more complex way to visually display our findings. I will make a map of the U.S. where each city is marked with a dot. Each city’s dot size corresponds to its calculated percentage of the Twitter population which misspelled common words.

boston <- 0
houston <- 0
los_angeles <- 0
chicago <- 0
atlanta <- 0
sapply(results, function(element) {
  if(as.character(element['city.name']) == "Boston") {
    boston <<- boston + as.numeric(element['percent'])
  }
  if(as.character(element['city.name']) == "Houston") {
    houston <<- houston + as.numeric(element['percent'])
  }
  if(as.character(element['city.name']) == "Los Angeles") {
    los_angeles <<- los_angeles + as.numeric(element['percent'])
  }
  if(as.character(element['city.name']) == "Chicago") {
    chicago <<- chicago + as.numeric(element['percent'])
  }
  if(as.character(element['city.name']) == "Atlanta") {
    atlanta <<- atlanta + as.numeric(element['percent'])
  }
})
results
percents <- c(boston, los_angeles, chicago, houston, atlanta)
max_percents <- max(percents)
percents <- sapply(percents, function(x) x/max_percents*6)
map <- cbind(geocode(as.character(cities$name)), cities)
ggmap(get_map(location = 'usa', zoom = 3)) + geom_point(data=map, aes(x=lon, y=lat, size=percents),show.legend=F, color="orange")

Here is our result

For more details check it out on github

Cleaning and Visualizing Data

2016-11-24T01:44:03+00:00

There is no lack of data on the internet but to make something useful of it is 90% of the battle. Here, I’m going to step through common practice of cleaning a large data set in R. The process I will use to clean the data is based off conventions introduced in an article by R expert Hadley Wickham. Ultimately, we’d like to have a data set we can easily work with to pull interesting results.

I chose a large messy data set to explore job patterns for minorities and women in private industries during 2009. Government data sets are good to use when practicing data cleaning skills because the files are typically large and messy. To start the cleaning process, I read the data into R

data <- read.table("YEAR09_CBSA_NAC3.txt", header= TRUE, sep=";")

Step 1: I need to rename columns to accurately reflect their data. I did the majority of this manually. (I know this is not the most efficient way but it’s one of my first R projects)

names(data)[1] <- "Area"
names(data)[2] <- "Industry"
names(data)[3] <- "to be removed"
names(data)[4] <- "to be removed"

Now, because this data set is so big let’s narrow the area of focus to the Boston category.

data <- subset(data, Area=='Boston-Cambridge-Quincy, MA-NH')

Consequentially, we will want to make changes such as removing the city, state column and removing columns where the notation was not in the legend.

data <- data[-c(1, 3, 4, 588:598, 599, 600)]

Step 2: At this point our observations are represented as columns; however, convention is for each observation to be a row. The process of turning columns into rows is called ‘melting’. So, now I will melt the data using the reshape package written by Hadley Wickham.

library(reshape)
melted <- melt(data, id=c("Industry"))

Step 3: Another characteristic of tidy data is that each variable represents a column. Here we have information of several variables contained in one column. So, the column should be separated into several. In this example the variables: race, gender, type of data, and position are all in one column. I will use another Hadley Whickham package tidyr to create four individual columns from the one.

library(tidyr)
split <- separate(melted, variable, c("Race", "Gender", "Type", "Position"), sep = ":")

Step 4: I delete rows with NA entries. As convention it is okay to delete them because sometimes information is just missing.

split <- subset(split, !is.na(value))

Step 5: The final major characteristic for tidy data is for each observational unit to be a table. In this case we have a column for type of data in which there are two types: number total and percent. So, let’s make two tables instead of having one.

totals <- subset(split, Type=="Total")
percentage <- subset(split, Type=="PoW")

Now, we no longer need the type of data column because it is reflected in the table title so let’s get rid of that.

totals <- totals[-c(4)]
percentage <- percentage[-c(4)]

Congratulations, our data set is clean! It’s time to explore our data.

First, Let’s look at a breakdown of number of workers by gender and position. These results visualized can provide intuition about trends and important questions such as: Are there significantly less female senior managers than male?

Here we use ggplot to graph a subset version of our data. We want a subset version which excludes information that would distort our graph. What questions would you ask based on the graph we’ve extrapolated?

totalOverallByGender <- subset(totals, Gender != "Overall" & Race != "Minority")
ggplot(data = totalOverallByGender, aes(x=Position,y=Total,fill=factor(Gender))) + 
  geom_bar(stat="identity",position='dodge') +
  theme(axis.text.x=element_text(angle=70,hjust=1,vjust=1)) +
  labs(title='Total Workforce in Positions by Gender', fill='Gender', y='# of Workers')

Second, let’s use our percentage table to look at the percentage of workers broken down by race for each position in the Food Service and Drinking Places industry. Again we need to subset the data to suit our interests and use ggplot to customize the visuals of our chart.

percentageFood <- subset(percentage, Gender == "Overall" & Industry == "Food Services and Drinking Places" & Race != "Minority")
ggplot(data = percentageFood, aes(x=Position,y=Total,fill=factor(Race))) + 
  geom_bar(stat="identity") +
  theme(axis.text.x=element_text(angle=70,hjust=1,vjust=1)) +
  labs(title='Food Services and Drinking Places Industry', y='% of Workforce', fill='Race')

There are countless questions that can be asked using the information stored in large data sets. Using the conventions of tidy data makes a world of difference in being able to answer those questions. Ultimately, we have data that is easy to work with as we take the next steps to answer these questions using statistical methods.

For more details check it out on github

Interactive Visualization

2016-11-08T01:44:03+00:00

I want to make interactive distributions of a few random variables. I want to see how changing sample size and other parameters changes the distribution.

We will be using a shinyApp with multiple tabs for each distribution. This requires using the ‘shiny’ package and since we are using random sampling, I’ve set the seed to 5.

library(shiny)
set.seed(5)

Note: the full code is at the bottom if at any point you would like to interact with the application on your own device or refer to it in full.

Let’s get started. The format of a shinyApp consists of two parts: the ui and the server. Each is commonly used as the variable to represent functions. The ui consists of the front end code because it is where page layout and parameters are defined. The server is the back end. This code is run each time parameters are updated. The line of code at the bottom is a function that creates the shinyApp and pairs the ui and server.

ui <- fluidPage(...
)
server <- function(input, output) {...
}
shinyApp(ui = ui, server = server)

We want the application to have a tabbed format for each distribution, this is represented in the ui function. Each distribution is enveloped in a tabPanel. All tab panels are enclosed around a tabsetPanel. Additionally, we need to nest all of this in a mainPanel. The application will still run without the mainPanel; however, it makes the format more visually appealing.

Normal Distribution

Let’s start by creating a normal distribution. First we’ll take a look at the ui component. As we saw each distribution will be enclosed in its own tabPanel. The first input in the tabPanel is the name of the tab. Next we use the numericInput function to represent each parameter for a normal distribution. The inputId represents the input and will be referenced in the server code. Sample size is the name of the variable as it will appear on the application and value is the preset value for the sample size variable. Similarly, we have created a sample mean variable for the normal distribution. Finally, plotOutput is used to indicate the need for a plot and is referenced in the server code.

      tabPanel("Normal", numericInput(inputId = "n", "Sample size", value=30),
               numericInput(inputId = "mean", "Sample mean", value=0),
               plotOutput("normal"))

In the sever portion, the code accompanying each distribution is defined by an output association then wrapped in a render plot function. The output association is referencing the plotOutplut name. We are rendering a histogram using random sampling from a normal distribution and using the two parameters we defined in the ui code. The break number is seen as the thickness of the bars on the histogram. A large break number means a less detailed histogram so for our purposes we’d like a large break number. Finally, we define the title and labels of the graph.

  output$normal <- renderPlot({
    hist(rnorm(input$n,input$mean),breaks=25,main="Histogram of a random Normal variable", xlab="")
  })

Here’s a look at the interactive normal distribution in use. As sample size increases the histogram looks more like the probability density function.

Poisson Distribution

The Poisson distribution has parameters: sample size and lambda. For the lambda parameter I’ve set the default value to 10, the minimum value to 1 and the maximum value to 100. I have also indicated that the slider should increase/ decrease by increments of 1. Take notice, sample size is a variable in every distribution but the inputId names need to be unique.

ui:

      tabPanel("Poisson", numericInput(inputId = "xtop", "Sample size", value=30),
               sliderInput(inputId = 'lambda', label="Lambda", min=1, max=100, value=10, step=1),
               plotOutput("poisson")),

server:

  output$poisson <- renderPlot({
    hist(rpois(input$xtop,input$lambda), breaks=25, main="Historgram of a random Poission variable", xlab = "")
  })

With large sample size, the poisson follows its probability mass function at various lambdas.

Discrete Uniform Distribution

The discrete uniform distribution has parameter: sample size.

ui:

      tabPanel("Uniform", numericInput(inputId = "n2", "Sample size", value=30),
               plotOutput("discrete")),

server:

  output$discrete <- renderPlot({
    hist(runif(input$n2,min=0,max=1),breaks=25,main="Histogram of a random discrete Uniform variable", xlab="")
  })

The larger the sample size, the more the histogram mirrors its ideal probability mass function.

Chi-Squared Distribution

The chi-squared distribution has parameters: sample size and degrees of freedom. The degrees of freedom default value is 5 with a minimum of 2 and a maximum of 100. The degrees of freedom scale should increase/decrease by increments of 1.

ui:

      tabPanel("Chi-Squared", numericInput(inputId = "n3", "Sample size", value=30),
               sliderInput(inputId = 'df', label="Degrees of Freedom", min=2, max=100, value=5, step=1),
               plotOutput("chiSquared"))

server:

  output$chiSquared <- renderPlot({
    hist(rchisq(input$n3,input$df), breaks=25, main="Historgram of a random Chi-Squared variable", xlab = "")
  })

With large sample size the histogram looks like its probability density function. It may not at first but notice how frequency changes. The maximum frequency for df=1 is much greater than when degrees of freedom decreases.

Sampling with Replacement

2016-11-05T01:44:03+00:00

“In statistics, bootstrapping is any test or metric that relies on random sampling with replacement.” I’m going to use this method of sampling to find means of a random variable.

First, we set the scenario. X is a random variable following a uniform distribution with the parameters a=1 and b=2 and 200 observations. We create a vector of size 1000 called xbars. Since we are dealing with random sampling I’ve set the seed to 5 so exact means and graphs can be recreated.

set.seed(5)
x = runif(200,1,2)
len <- length(x)
Max <- 1000
xbars <- c(1:1000)

For each placeholder in xbars, the function takes a random sample of 200 with replacement from our random variable X and computes the mean. This process is resampling and the means in xbars are called bootstrap estimates.

for(i in 1:Max){
  xx1 <- sample(x, size=len, replace=TRUE)
  xbars[i] <- mean(xx1)
}

A Q-Q plot of our random variable X confirms it follows a uniform distribution. And a Q-Q plot of the sample means confirms normality.

The same process works with our random variable Y where Y=1/x and X follows our uniform distribution from above.

y = 1/x
len <- length(y)
Max <- 1000
ybars <- c(1:1000)
for(i in 1:Max){
  xx1 <- sample(y, size=len, replace=TRUE)
  ybars[i] <- mean(xx1)
}

Again we can confirm the process is working as expected because there are no irregularities in the Q-Q plots.

It is noteworthy to take a look at the means of our random variables and the means of the bootstrap estimators. We expect these numbers to be very close but not exactly the same because we are resampling many times with large n. When resampling, the most interesting information comes from the histogram of bootstrap means and analysis of the means’ distribution.

hist(xbars, xlab="Bootstrap Estimates X~")
hist(ybars, xlab="Bootstrap Estimates Y~")

Log Transformation

2016-10-24T01:44:03+00:00

In modern statistical practice, methods to analyze linear data are much more understood than nonlinear data. So, ideally the data you’re working with has a linear trend. When it does not, what can be done? Transformations. Some common types of transformations include: log, square root, cube root, and reciprocal. While there is broad criteria to determine which method of transformation should be used, it takes years of experience to master the art.

Here, we will use R to look at a simple linear regression, identify it as nonlinear, apply a log transformation, and confirm our changes have improved the data.

Let’s use the UN11 data from the alr4 package. Remember you only need to install the package the first time you use it. The last line of code will provide a detailed description of the interpretation of the data set.

install.packages("alr4")
library(alr4)
?UN11

The simple linear model we would like to study is how per capita gpd (ppgdp) predicts number of children per women (fertility). For data that is linear, the plot of these variables should show a linear trend.

plot(UN11$ppgdp, UN11$fertility)

Here we see the data exponentially decreasing. This is a problem because the exponential function is most certainly not linear.

Another way to check for linearity of variables is to look at their histograms. Linear data will conform to a normal bell curve while nonlinear data will have skewed histograms.

hist(UN11$ppgdp, main ="Histogram of Ppgdp", xlab= 'ppgdp')
hist(UN11$fertility,main ="Histogram of Fertility",xlab = "fertility")

As expected, our variables confirm to be nonlinear

At this point a seasoned statistician would suggest this data needs to be transformed with the log function. So, lets plot the regression where each variable has been transformed.

plot(log(fertility) ~ log(ppgdp), UN11,ylab="log(fertility)", xlab="log(ppgdp)")

Where we before saw an exponential trend we now see a negative linear trend. This plot looks good because log transformation has successfully made the data behave linearly.

Finally, let’s take an individual look at the histograms of our transformed variables.

hist(log(UN11$ppgdp),main ="Histogram of Log Ppgdp", xlab = 'log(ppgdp)')
hist(log(UN11$fertility), main = "Histogram of Log Fertility", xlab = "log(fertility)")

We can see a great improvement to the skewedness of the histograms. While they don’t look like completely standard bell curves, the transformation has greatly changed the behavior of the data to act more linear.