\chapter{Some Infrastructure:  
Probability and Statistics}
\label{chap:infra2}   

Many RS methods are probabilistic in nature, so we will lay out some
infrastructure.  It is assumed the reader has a background in
calculus-based probability structures, such as expected value and
density functions.  Background in statistics (as opposed to probability)
and machine learning is \textit{not} assumed.\footnote{The reader may
wish to consult my open source book on probability and statistics, N.
Matloff, \textit{From Algorithms to Z-Scores: Probability and
Statistical Modeling in Computer Science},
\url{http://heather.cs.ucdavis.edu/probstatbook}.}

Note that while we will develop some statistical methods here, notably
regression and classification models, we will not cover
\textit{inferential} statistical methods such as confidence intervals
and significance tests.  For readers familiar with such topics,
occasional footnotes will be provided. 

Other than laying some groundwork, e.g. introducing the term
\textit{dummy variables}, the primary goal of this chapter will be to
discuss the issue of \textit{overfitting}.

\section{Data as a Sample}

In statistics, the data are usually considered a sample from a
population.  For instance, during an election campaign pollsters will
take a \underline{sample}, typically of 1200 people, from the
\underline{population} of all voters.  Say they are interested in $p$,
the population proportion of voters favoring Candidate Jones. They
calculate their \underline{estimate} of $p$, denoted $\widehat{p}$, to
be the proportion of voters in the \underline{sample} who like Jones.

Sometimes the notion of sampling is merely conceptual. Say for instance
we are studying hypertension, on data involving 1000 patients.  We think
of them as a sample from the population of all sufferers of
hypertension, even though we did not go through an actual sampling
process.

In RS contexts, this means that we treat the users in our dataset as a
sample from the conceptual population of all potential users.  We may
even treat the items as a sample from a conceptual population of all
possible items.

In machine learning circles, it is not customary to think explicitly in
terms of populations, samples and estimates.  Nevertheless, it's
implied, as ML people do talk about predicting new data from the model
fitted on the original data.  For the model to be useful, the new data
needs to come from the same source as the original --- what
statisticians call a population.

We will usually think in terms of sample data here.

\section{Probability, Expected Value and Variance}

We will speak in terms of a \textit{repeatable experiment}, which again
could be physical or conceptual.

We think of probability as the long-run proportion of the time some
event occurs.  Say we toss a fair coin.  What do we mean by
$P(\textrm{heads} = 0.5)$?  Here our repeatable experiment is tossing the
coin.  If we were to perform that experiment many times ---
ideally, infinitely many times --- then in the long run, 50\% of the
repetitions would give us heads.

Now suppose our experiment, say a game, is to keep tossing a coin until
we get three consecutive heads.  Let $X$ denote the number of tosses
needed.  Then for instance $P(X = 4) = 0.5^4 = 0.0625$ (we get a tail
then three heads).  Imagine doing this experiment infinitely many times:
We toss the coin until we get three consecutive heads, and record $X$;
we toss the coin until we get three consecutive heads, and record $X$;
we toss the coin until we get three consecutive heads, and record $X$;
and so on.  This would result in infinitely many $X$ values.  Then in
the long run, 6.25\% of the $X$ values would be 4.

The \textit{expected value} $E(X)$ of a random variable $X$ is its long-run
average value over infinitely many repetitions of the experiment.  In
that 3-consecutive heads game above, it can be shown that $E(X) = 14.7$.
In other words, if we were to play the game infinitely many times,
yielding infinitely $X$ values, their long-run average would be 14.7.

If there is no danger of ambiguity, we usually omit the parentheses,
writing $EX$ instead of $E(X)$.

The \textit{variance} of a random variable is a measure of its
dispersion, i.e.\ how much it varies from one repetition to the next.
It is defined as $Var(X) = E[(X - EX)^2]$.

Say we have a population of people and our our experiment is to 
randomly draw one person from the population, denoting that person's
height by $H$.  Then intuitively, $EH$ will be the mean height of all
the people in the population, traditionally written as $\mu$.

\section{Regression Models} 

Regression models, both \textit{parametric} and \textit{nonparametric},
\textbf{form the very core of statistics and machine learning (ML)}.  Their
importance cannot be overemphasized.\footnote{For many, the term
\textit{regression analysis} connotes a linear parametric model.  But
actually the term is much more general, defined to be the conditional
mean as discussed below.  Most ML techniques are nonparametric, as
explained below, but are still regression methods.}

\subsection{Definition}

Suppose we are predicting a variable $Y$ from a vector $X$ of variables,
say predicting human weight from the vector (height,age).  The
\textit{regression function} at $t = (t_1,t_2)$ of $Y$ on $X$ is defined
to be the mean weight of all people in the subpopulation consisting of
all people of height $t_1$ and age $t_2$.

Let $W$, $H$ and $A$ denote weight, height and age.  We write the
regression function as the \textit{conditional expectation} of $W$ given
$H$ and $A$, 

\begin{equation}
\label{regdef}
E(W ~| H=t_1, A=t_2)
\end{equation}

If, say $E(W ~| H=70, A=28) = 162$, it means that the mean weight of all
people in the subpopulation consisting of 28-year-olds of height 70 is
162.

Note that in (\ref{regdef}), the expression has a different value for
each $(t_1,t_2)$ pair.  So it is a \underline{function} of $t_1$ and
$t_2$.  This is why it is called the \textit{regression function} of $W$ on $H$
and $A$.

\textit{Terminology:} It is common to refer to $W$ here at the
\textit{response variable} and $H$ and $A$ as the \textit{predictor
variables}.  The latter may also be called \textit{explanatory
variables} (in economics and other social sciences) or \textit{features}
(in ML).

\subsection{Prediction}

Say we have a person whose height and age are 70 and 28, but with
unknown weight.  It can be shown that the optimal (under a certain
criterion) predictor of her weight is the value of the regression
function at (70,28), $E(W ~|~ H=70, A=28) = 162$.  It is optimal
in the sense of minimizing expected squared prediction error.

\subsection{Estimation}

The regression function is an attribute of the population.  Yet all we
have is sample data.  How do we estimate the regression function from
our data?

\subsubsection{Nonparametric}

Intuitively, we could use a nearest-neighbor approach.  To estimate
$E(W ~| H=70, A=28)$, we could find, say, the 25 people in our
sample for whom $(H,A)$ is closest to (70,28), and average their weights
to produce our estimate of $E(W ~| H=70, A=28)$.  

This kind of approach is common in ML.  The famous \textit{random
forests} method is basically a more complex form of kNN, as we will see
in Chapter \ref{chap:covars}.  

Statisticians also use methods like kNN.  In fact, kNN and random
forests were invented by statisticians.  But more commonly, statistics
uses \textit{parametric} methods, as follows.

\subsubsection{Parametric}

The basic idea is to assume the regression function is linear in
\underline{parameters} $\beta_i$, e.g.

\begin{equation}
\label{wthtage}
\textrm{mean weight} = \beta_0 + \beta_1 ~ \textrm{height} + \beta_2 ~ \textrm{age}
\end{equation}

for some unknown values of the $\beta_i$.

Make sure to take careful note of the word ``mean''!  Clearly, the
weights of \underline{individual} people are not linear functions of
their height and age.

As noted, the $\beta_i$ are unknown, and need to be estimated
from our sample data.  The estimates will be denoted
$\widehat{\beta}_i$.  They are obtained by minimizing a certain sum of
squares, to be discussed in Section \ref{lmdetails}.

By the way, if the reader is familiar with the ML methodology known as
\textit{neural networks}, she may be surprised that this technique is
also parametric.  Again, more in Chapter \ref{chap:covars}.

\subsubsection{Comparison}

Consider (\ref{wthtage}), our model for the function of $t_1$ and $t_2$

\begin{equation}
E(\textrm{weight} ~|~ \textrm{height} = t_1, \textrm{age} = t_2)
\end{equation}


With the linear assumption (\ref{wthtage}), we will be estimating three
quantities, the $\beta_i$.  But with a nonparametric approach, we are
estimating infinitely many quantities, one for each value of the
$(t_1,t_2)$ pair.

In other words, \textbf{parametric methods are a form of dimension reduction}.
On the other hand, this reduction comes at the expense of relying on the
assumption of linearity in (\ref{wthtage}).  However, this is not so
restrictive as it may seem, because:

\begin{itemize}

\item There are ways to assess the validity of the assumption.  This is
covered in almost any book on regression, such as mine
(N. Matloff, \textit{Statistical Regression and Classification: from
Linear Models to Machine Learning}, CRC, 2017).

\item One can add polynomial terms, as seen in the next section.

\item Assumptions tend to be less important in prediction contexts than
in estimation.  In the RS context, for instance, a rough model may be
fine if we wish to take into account gender in predicting ratings, but
might be insufficient if we want to estimate the actual magnitude of
gender effect.

\end{itemize} 

\subsection{The lm() Function in R}

In R, the workhorse regression estimator is the \textbf{lm()} function.
Let's apply this to the MovieLens data, predicting rating from age and
gender.  We'll define the latter as 1 for male, 0 for female.  We find
that our estimated regression function of rating on age and gender is

\begin{equation}
\textrm{estimated mean rating} = 3.3599 + 0.005311 ~ \textrm{age} 
- 0.0069 ~ \textrm{gender}
\end{equation}

(Note the word \textit{estimated}! These are not the true unknown
population values, just estimates based on sample data.)

Actually, this shows that age and gender are pretty weak predictors of
movie rating, which you will recall is on a scale of 1 to 5.  A 10-year
difference in age raises the predicted rating only by about 0.05!  The
effect of gender is small too.  And while it is interesting to see that
older people tend to give slighly higher ratings, as do women, we must
keep in mind that the magnitude of the effect here is
small.\footnote{You may be familiar with the term \textit{statistically
significant}.  It is generally recognized today that this term can be
quite misleading.  This is beyond the scope of this book, but suffice it
to say that although age and gender are statistically significant above
(details available via adding the call \textbf{summary(lmout)} to the
code below), their practical importance as predictors here is
essentially nil.  See R. Wasserstein and N. Lazar, The ASA's Statement
on p-Values: Context, Process, and Purpose, \textit{The American
Statistician}, June 2016.}  Of course, the gender effect may be large in
other RS datasets.

Here is the annotated code:

\begin{lstlisting}

# read (user,item,rating,transID) data; name the columns
ratings <- read.table('u.data') 
names(ratings) <- c('usernum','movienum','rating','transID') 
# read demographic data
demog <- read.table('u.user',sep='|') 
names(demog) <- c('usernum','age','gender','occ','ZIP') 
# merge (database 'join' op)
u.big <- merge(ratings,demog,by.x=1,by.y=1) 
u <- u.big[,c(1,2,3,5,6)] 
# fit linear model
lmout <- lm(rating ~ age+gender,data=u) 

\end{lstlisting}

Here's the output:

\begin{lstlisting}
> lmout

Call:
lm(formula = rating ~ age + gender, data = u)

Coefficients:
(Intercept)          age      genderM  
3.359894        0.005311    -0.006904  
\end{lstlisting}

Let's take a closer look at that \textbf{genderM}
coefficient.\footnote{The gender variable had been coded in the data as
'M' and 'F', and R chose the first one that showed up in the data, 'M',
as its base.}  Take for instance 28-year-old men and women; what are
their mean ratings, according to this model?

\begin{lstlisting}
> lmout$coef %*% c(1,28,1)
        [,1]
[1,] 3.50169
> lmout$coef %*% c(1,28,0)
         [,1]
[1,] 3.508593
\end{lstlisting}

(Note that the first '1' is needed to pick up the 3.359894.)

So, on average, 28-year-old women give ratings 3.508593 - 3.50169 =
0.006903 higher than men of that age.  And except for roundoff error,
that is the -0.006904 value was see in the output above.  

\subsection{Details of Linear Regression Estimation}
\label{lmdetails}

In the weight-height-age example, say, we form 

\begin{equation}
\label{rss}
r =
\sum_{i=1}^n [W_i - (b_0 + b_1 H_i + b_2 A_i)]^2
\end{equation}

where $W_i$ is the weight of the i$^{th}$ person in our sample data and
so on. This is the sum of squared prediction errors.  We take derivatives
with respect to the $b_k$ to minimize, then set $\widehat{\beta}_k$ to
the minimizing $b_k$.  

Though R will do the minimizing for us, it is worth having an idea how
it works, especially as more practice in following matrix-centric
derivations.  To get a glimpse of this, we look at a matrix formulation, as
follows.  Let $A$ denote the matrix of predictor values, with a 1s
column tacked on at the left.  In the above example, row 12, say, of $A$
would consist of a 1, followed by the height and age of the 12$^{th}$
person in our sample.  Let $D$ denote the vector of weights, so that
$D_{12}$ is the weight of the 12$^{th}$ person in our sample.  Finally,
let $b$ denote the vector of the $b_k$.  Say we have data on 100 people.
Then $A$ will have 100 rows, and $D$ will have length 100.

Use the above as a concrete guide to your thinking, but keep in mind the
general case:  If we have $p$ predictors and $n$ data points, then $A$
and $D$ will have sizes $n \times (p+1)$ and $n$

Then

\begin{equation}
\label{}
r = (D - A b)' (D - Ab)
\end{equation}

(Write it out to see this.  Doing so will be crucial to understanding
the material below and many points in the rest of the book.)

Write the \textit{gradient} of $r$ with respect to $b$,

\begin{equation}
\frac{\partial r}{\partial b} = (
\frac{\partial r}{\partial b_0},
\frac{\partial r}{\partial b_1},
...,
\frac{\partial r}{\partial b_p}
)'
\end{equation}

where $p+1$ is the number of predictor variables.\footnote{Note the
representation here of a column vector as the transpose of a row vector.
We will often do this, in order to save space on the page.  And, any
reference to a \textit{vector} will be to a column vector unless stated
otherwise.}

It can be shown that for a vector $u$, 

\begin{equation}
\frac{\partial u'u}{\partial u} = 2u
\end{equation}

(analogous to the scalar relations $d (u^2) /du = 2u$; again, this is
seen by writing the expressions out).

Setting $u = D - Ab$ and applying the Chain Rule (adapted for
gradients), we get

\begin{equation}
\frac{\partial r}{\partial b} = 
\frac{\partial r}{\partial u} 
\frac{\partial u}{\partial b} =
2(D - Ab) ~ \frac{\partial (D - Ab)}{\partial b}=
2 (-A') (D - Ab) 
\end{equation}

Setting the gradient to 0 and solving for $b$, we have

\begin{equation}
\label{adaab}
0 = A'D - A'Ab
\end{equation}  

so that the minimizing $b$, giving us $\widehat{\beta}$, is

\begin{equation}
\label{famouslm}
b = (A'A)^{-1} A'D
\end{equation}

This famous formula is what \textbf{lm()} computes in finding the
$\widehat{\beta}_k$.

Note that we cannot simply multiply both sides of (\ref{adaab}) by
$(A')^{-1}$, as $A'$ is nonsquare and thus noninvertible.

Note too that in our age/gender MovieLens example above, we should not
have variables for both male and female.  If we did, we have

\begin{equation}
A =
\left(
\begin{array}{rrrr}
1 & A_1 & M_1 & F_1 \\
1 & A_2 & M_2 & F_2 \\
... \\
1 & A_{100000} & M_{100000} & F_{100000} \\
\end{array}
\right )
\end{equation}

where $A_i$ is the age of the $i^{th}$ person in our data, and one of
$M_{i}$ and $F_{i}$ is 1 and the other 0, according to the
gender of this person.  (Recall that there are 100000 data points in tis
dataset.)  The problem is this:  The third and fourth columns of $A$
would then sum to a vector of all 1s, the same as in the first column.
So the columns of $A$ will be linearly dependent, and the rank will be 3
instead of 4.  The same will then be true for $A'A$, so that $(A'A)^{-1}$
will not exist in (\ref{famouslm}).

In other words, not only would the F column be unnecessary, it would be
prolematic.

\subsubsection{Polynomial Terms}
\label{poly}

People tend to gain weight during middle age, but often they lose weight
when they become elderly.  So (\ref{wthtage}), which is linear in the
age variable, may be rather unrealistic; we might believe a quadratic
model for mean weight as a function of age is better:

\begin{equation}
\label{wthtage2}
\textrm{mean weight} = \beta_0 + \beta_1 ~ \textrm{height} + 
\beta_2 ~ \textrm{age} +
\beta_3 ~ \textrm{age}^2
\end{equation}

A key point is that this is still a linear model! When we speak of a
linear model --- the 'l' in ``lm()'' -- we mean linear in the $\beta_i$.
If in (\ref{wthtage2}) we, say, multiply all the $\beta_i$ by 3, the
entire sum grows by a factor of 3, hence the linearity in the $\beta_i$.

Of course we may wish to add a quadratic term for height as well, and
for that matter, a product term height $\times$ age.  And since any
model is merely an approximation, we might consider using higher and
higher order polynomials.  We do have to worry about overfitting though;
see Section \ref{overfit}.

We'll have a long example in Section \ref{prgengex}.

\subsection{Categorical Variables (Predictor, Response)}

A \textit{categorical} variable is one that codes categories.  In our RS
context, for instance, a user's postal code --- ZIP Code, in the US ---
may be a helpful predictor of a rating.  Yet it cannot be treated in
\textbf{lm()}, say as a numeric variable.  After all, the ZIP Code
90024, for example, is not ``twice as good'' as 45002; they are just ID
codes.

\subsubsection{Dummy Variables as Predictors}

So, what do we do if we wish to use a categorical variable as a
predictor?  The answer is that we break the variable into a set of
\textit{indicator variables}, colloquially known as \textit{dummy
variables}.  These have the values 1 and 0, with 1
\underline{indicating} the trait in question, 0 meaning not.

Say for instance in RS we have the categorical variable State for users
in a US state.  We would define 50 dummy variables, one for each state.
For instance, the one for California would have the value 1 if the user
lives in California, 0 otherwise.

Note carefully though that we would only use 49 of the dummies, not 50.
We could for instance exclude Wyoming.  Why? Because if the other 49
dummies are all 0, then we know this user must be in Wyoming.  The
Wyoming dummy would then be redundant. Not only do we want to avoid
redundancy on dimension reduction grounds, but also that redundancy
would result in the matrix $A$ in (\ref{famouslm}) being less than full
rank, so $(A'A)^{-1}$ would not exist.

\textit{Categorical variables in R:}

In R, categorical variables are stored as objects of the class
\textbf{'factor'}.  The latter is a class designed specifically for
representing categorical data.  It consists of integer codes for the
categories, with character-string names.  The various categories are
called \textit{levels}.  

For example, say we are dealing with eye color, and that the only colors
are brown, blue and green, and have data on four people:

\begin{lstlisting}
> s <- c('brown','blue','green','blue')
> class(s)
[1] "character"
> sf <- as.factor(s)
> levels(sf)
[1] "blue"  "brown" "green"
> sf[2]
[1] blue
Levels: blue brown green
> as.numeric(sf)
[1] 2 1 3 1
\end{lstlisting}

In our MovieLens example above, let's take a look at
the data frame \textbf{demog}:

\begin{lstlisting}
> for (i in 1:5) 
+    print(class(demog[,i]))
[1] "integer"
[1] "integer"
[1] "factor"
[1] "factor"
[1] "factor"
\end{lstlisting}

The last three columns are factors.\footnote{Even the first two could
have been stored as factors, but were not coded as so.}  Let's see how
many occupations there are:

\begin{lstlisting}
> levels(demog$occ)
[1] "administrator" "artist"        "doctor"        "educator"     
[5] "engineer"      "entertainment" "executive"     "healthcare"   
[9] "homemaker"     "lawyer"        "librarian"     "marketing"    
[13] "none"          "other"         "programmer"    "retired"      
[17] "salesman"      "scientist"     "student"       "technician"   
[21] "writer"       
\end{lstlisting}

In a regression application, we'd form 21 dummies, but use only 20 of
them (any 20).

However, the designers of R (and its predecessor S), in their wisdom,
set things up to save us some time and trouble.  We can just specify the
factors we wish to use, and R will form the dummies for us, being
careful to drop one of them.

This is what happened, for instance, in our example above in which we
regressed rating against age and gender, with output

\begin{lstlisting}
Coefficients:
(Intercept)          age      genderM  
3.359894        0.005311    -0.006904  
\end{lstlisting}

R noticed that gender was an R factor, with levels M and F.  It created
dummies for M and F, but just retained the former, as the first case in
the data had gender as M.

So, the estimated coefficient -0.006904 meant that the ``maleness''
impact on mean rating has that value.  Men give slightly lower
ratings than women do, for fixed age.

In this manner, the estimated regression coefficient of a dummy variable
is the effect, all other predictors fixed, of this trait
\textit{relative to the excluded trai}, say femaleness above.

\subsubsection{Interaction Terms}

Say we are in some RS context in which age and gender are substantial
factors in predicting rating.  Suppose also that we suspect men become
more liberal raters as they age while women become more reserved in
their ratings.  Then a model like this might work well:

\begin{equation}
\label{mfinteraction}
\textrm{mean rating} = \beta_0 + \beta_1 ~ \textrm{age} + 
\beta_2 ~ \textrm{male} +
\beta_3 ~ \textrm{age} \times \textrm{male}
\end{equation}

where \textit{male} is a dummy variable.  To see why this might be
appropriate, consider what the above equation reduces to for men and
women:

\textit{men:}

\begin{equation}
\textrm{mean rating} 
= (\beta_0+\beta_2) + (\beta_1+\beta_3) ~ \textrm{age} 
\end{equation}

\textit{women:}M

\begin{equation}
\textrm{mean rating} = \beta_0 + \beta_1 ~ \textrm{age} 
\end{equation}

So the male and female lines have different slopes (and different
intercepts), allowing for the differential age effect we surmise.
Of course, once we compute the $\widehat{\beta}_i$ from the data, it
may well turn out that our differential aging trends may not be
confirmed.\footnote{One must take sampling variability into account, say
by forming confidence intervals for the $\beta_i$.  As noted earlier, do
not use significance testing for this.  At any rate, these aspects are
beyond the scope of this book.}

The term $\textrm{age} \times \textrm{male}$ is called an
\textit{interaction term}.  Note that interaction terms can be formed
from any predictor, not just dummy variables.  Also, one can form triple
products for three-way interactions and so on, though this could greatly
increase the complexity of the model and thus risk overfitting.


\subsubsection{Dummy Variables as Response Variables}

In many cases, the response variable may be categorical.  In the RS
context, for instance, a rating may simply be binary, i.e.\
like/dislike.  Or even click/not click --- does a user click on a Web page
location?  Let's use this as our example.

We are generally interested in the probability of a click.  That
actually fits a regression context, as follows.  Code a click as 1 and
nonclick as 0.  Since the expected value of a variable of this type is
the probability of a 1, and since a regression function by definition
is an expected value, taking Click as our response variables does
involve a regression function.

So, if our predictors were age and gender, say, we might entertain
formulating our regression model as

\begin{equation}
\label{clickagegen}
\textrm{probability of click} = \beta_0 + \beta_1 ~ \textrm{age} +
\beta_2 ~ \textrm{gender}
\end{equation}

One problem, though, is that a probability should be in [0,1] yet the
right-hand side of (\ref{clickagegen}) can conceivably be anywhere in
$(-\infty,\infty)$.  For this and other reasons the usual parametric
model for a binary response $Y$ is the \textit{logistic}:  For $p$
predictors $X_i$, our model is

\begin{equation}
P(Y  = 1 ~|~ X_1=t_1,...,X_p=t_p) =
\frac{1}{1+\exp{-(\beta_0+\beta_1 t_1+...+\beta_p t_p})}
\end{equation}

This is called a \textit{generalized linear model}, as it has the linear
form $\beta_0+\beta_1 t_1+...+\beta_p t_p$ embedded inside
another function, in this case the logistic function $g(s) =
1/(1+e^{-s})$.

Note that the latter function, often called \textit{logit} for short,
has values only in [0,1], as desired, and is increasing in $s$, thus
retaining the monotonic notion of linear models.\footnote{These
properties form the intuitive motivation for using logit models.
Another motivation is this: Let $X$ denote the vector of predictor
variables, and let $Y$ be the response variable, with the two classes 0
and 1.  If within each class, $X$ has a multivariate normal
distribution, with the same covariance matrix in each class.}

The $\beta_i$ are estimated by an R function \textbf{glm()}, similar to
\textbf{lm()}.\footnote{The class of the return value is \textbf{'glm'},
which is a subclass of \textbf{'lm'.}
}
Let's model a user giving a movie a rating of 4 or
higher:

\begin{lstlisting}
> r45 <- as.integer(u$rating >= 4)  # a binary value, 1 or 0
> u$r45 <- r45
> glmout <- glm(r45 ~ age+gender,data=u,family=binomial)
> glmout

Call:  glm(formula = r45 ~ age + gender, data = u)

Coefficients:
(Intercept)          age      genderM  
-0.002510       0.006886    -0.011189  
...
\end{lstlisting}

The argument \textbf{family = binomial} tells R that we want the
logistic model, not some other generalized linear model, such a model
known as \textit{Poisson regression}.\footnote{By the way, the argument
\textbf{family} must be an object of class \textbf{'function'}.  Inside
\textbf{glm()}, there will be a call \textbf{family()}.  R has a
built-in function \textbf{binomial()}, which is called here.}

\subsection{R's predict(), a Generic Function}

A key aspect to R's object orientation is \textit{generic} functions.
Take \textbf{plot()}, for instance.  Its action will depend on the class
of object it is applied to.  If we call the function on a vector, we get
a histogram.  But if we call it on a two-column matrix, we get a scatter
diagram.

What happens is that when \textbf{plot()} is called, R will check what
class of object the caller supplied as an argument.  If the object is of
class \textbf{"x"}, then the original call will be \textit{dispatched}
to \textbf{plot.x()}, a plotting function tailored to that class.  (Of
course, that means one needs to have been written and available.)

R's \textbf{predict()} is another example of a generic function, used to
predict new cases.  In the MovieLens example above, say we want to
predict the rating given by a 30-year-old man.  We could simply plug
30 and 1 into the estimated regression function, say using
\textbf{coef()} to get the $\widehat{\beta}_i$:

\begin{lstlisting}
> coef(lmout)
(Intercept)          age      genderM 
3.359894442  0.005310673 -0.006903502 
> coef(lmout) %*% c(1,30,1)  # linear algebra-style matrix multiply
      [,1]
[1,] 3.512311
\end{lstlisting}

Alternatively (and in many settings, more conveniently):

\begin{lstlisting}
> newdata <- data.frame(age=30,gender='M')
> predict(lmout,newdata)
    1 
3.512311 
\end{lstlisting}

Recall that we had assigned the output of \textbf{lm()} to
\textbf{lmout}, which will have class \textbf{'lm'}.  So, the call to
\textbf{predict()} above was dispatched to \textbf{predict.lm()}.

What about \textbf{glm()}?  There is a function \textbf{predict.glm()},
which normally should be called with the argument \textbf{type =
'response'}.  The latter means we want the return values to be the
estimated values of the regression function, i.e.\ the conditional
probabilities of response 1, given the values of the predictors.

\subsection{More Than Two Levels in Categorical Response}

What if our response variable is categorical but with more than two
levels?  In the click/nonclick setting, suppose the user has a choice of
five things to click, and must choose one.  Then the response is
categorical with five levels.  

There are two major approaches.  To explain, we'll use the following
very simple example. Say there are dogs, cats and foxes on a field, and
they sometimes step on a sensor, so we know their weights but do not see
them.  Say we have data on 10000 data points, in which we do know the
species.  Our data frame, \textbf{df}, has 10000 rows and 4 columns.  In
the columns, say the names are 'Weight', 'Dog', 'Cat' and 'Fox', with
the last three being dummies.  Say we have 5000 dogs, 2000 cats and 3000
foxes.  Then for instance 2000 of the rows in \textbf{df} would be cats.

\textit{One-vs.All (OVA) Method}

One would run run three logistic models:

\begin{lstlisting}
gdog <- glm(Dog ~ .,data=df[,1:2])  # dog vs. all else
gcat <- glm(Cat ~ .,data=df[,c(1,3)])  # cat vs. all else
gfox <- glm(Fox ~ .,data=df[,c(1,4])  # fox vs. all else
\end{lstlisting}

Then for each new animal we encounter of unknown species, we call
\textbf{predict()} three times, yielding three estimated conditional
probabilities.  If the one for cat, say, is largest, we guess Cat.

\textit{All vs. All (AVA) Method}

Here again we'd run multiple logit models, in pairs as follows:

\begin{lstlisting}
gdogcat <- 
   glm(Dog ~ .,data=df[df$dog+df$cat==1,1:2])  # dog vs. cat
gdogfox <- 
   glm(Dog ~ .,data=df[df$dog+df$fox==1,1:2])  # dog vs. fox
gcatfox <- 
   glm(Cat ~ .,data=df[df$cat+df$fox==1,1:3])  # cat vs. fox
\end{lstlisting}

Then for each new animal we encounter of unknown species, we call
\textbf{predict()} three times, again yielding three estimated
conditional probabilities.  Say in the first one, Cat ``wins,'' i.e.\
the conditional probability is less than 0.5.  Say Dog wins in the
second, and Cat wins in the third.  Since Cat had the most wins, we
predict Cat.

\textit{Comparison}

At first, OVA seems much better than AVA.  If we have $m$ levels, that
means running $C(m,2) = O(m^2)$ parwise logit models, rather than $m$
for OVA.  However, that is somewhat compensated by the fact that each
pairwise model will be based on less data, and some analysts contend
that AVA can have better accuracy.  It remains a bit of a controversy.

\section{Bias, Variance, Overfitting and p-Hacking}
\label{overfit}

By far the most vexing issue in statistics and machine learning is that
of \textit{overfitting}.  

\subsection{What Is Overfitting?}
\label{norelation}

Suppose we have just one predictor, and $n$ data points.  If we fit a
polynomial model of degree $n-1$, the resulting curve will pass through
all $n$ points, a ``perfect'' fit.  For instance:

\begin{lstlisting}
> x <- rnorm(6) 
> y <- rnorm(6) # unrelated to x!
> df <- data.frame(x,y) 
> df$x2 <- x^2 
> df$x3 <- x^3 
> df$x4 <- x^4 
> df$x5 <- x^5 
> df
           x          y         x2           x3
1 -1.1855131  0.2881291 1.40544120 -1.666168894
2 -1.7838769 -2.0741740 3.18221664 -5.676682627
3 -0.7124510 -0.4253678 0.50758640 -0.361630431
4  0.1676111 -0.1949265 0.02809348  0.004708779
5  1.2462926 -0.7348481 1.55324535  1.935798245
6  0.3741604  1.9521667 0.13999601  0.052380963
            x4            x5
1 1.975265e+00  -2.341702414
2 1.012650e+01 -18.064433938
3 2.576440e-01  -0.183558689
4 7.892437e-04   0.000132286
5 2.412571e+00   3.006769615
6 1.959888e-02   0.007333126
> lmo <- lm(y ~ .,data=df) 
> lmo

Call:
lm(formula = y ~ ., data = df)

Coefficients:
(Intercept)            x           x2           x3  
    -1.3127       4.7632      11.4809       0.5781  
         x4           x5  
    -6.9685      -2.4938  
> lmo$fitted.values 
         1          2          3          4          5 
 0.2881291 -2.0741740 -0.4253678 -0.1949265 -0.7348481 
         6 
 1.9521667 
> y
[1]  0.2881291 -2.0741740 -0.4253678 -0.1949265 -0.7348481
[6]  1.9521667
\end{lstlisting}

Yes, we ``predicted'' \textbf{y} perfectly, \textbf{even though there
was no relation between the response and predictor variables).}
Clearly that ``perfect fit'' is illusory, ``noise fitting.''  Our
ability to predict future cases would not be good.  This is
\textit{overfitting}.


Let's take a closer look, in an RS context.  Say we believe
(\ref{mfinteraction}) is a good model for the setting described in that
section, i.e.\ men becoming more liberal raters as they age but women
becoming more conservative.  If we omit the interaction term, than we
will underpredict older men and overpredict older women.  This biases
our ratings.

On the other hand, we need to worry about sampling variance.  Consider
the case of opinion polls during an election campaign, in which the goal
is to estimate $p$, the proportion of voters who will vote for Candidate
Jones.  If we use too small a sample size, say 50, our results will
probably be inaccurate.  This is due to sampling instability:  Two
pollsters, each randomly sampling 50 people, will sample different sets
of people, thus each having different values of $\widehat{p}$, their
sample estimates of $p$.  For a sample of size 50, it is likely that
their two values of $\widehat{p}$ will be substantially different from
each other, whereas if the sample size were 5000, the two estimates
would likely be close to each other.  In other words, the
\underline{variance} of $\widehat{p}$ is too high if the sample size is
just 50.\footnote{The repeatable experiment here is randomly choosing 50
people.  Each time we perform this experiment, we get a different set of
50 people, thus a different value of $\widehat{p}$.  The latter is a
random variable, and thus has a variance.}

In a parametric regression setting, increasing the number of terms
roughly means that the sampling variance of the $\widehat{\beta}_i$ will
increase.

So we have the famous \textit{bias/variance tradeoff}:  As we use more and
more terms in our regression model (predictors, polynomials, interaction
terms), the bias decreases but the variance increases.  This ``tug of
war'' between these decreasing and increasing quantities typically
yields a U-shaped curve:  As we increase the number of terms from 1,
mean absolute prediction error will at first decrease but eventually
will increase.  Once we get to the point at which it increases, we are
\textit{overfitting}.

This is particularly a problem when one has many dummy variables. For
instance, there are more than 42,000 ZIP Codes in the US; to have a
dummy for each would almost certainly be overfitting.

\subsection{Can Anything Be Done about It?}

So, where is the ``happy medium,'' the model that is rich enough to
capture most of the dynamics of the variables at hand, but simple enough
to avoid variance issues?  Unfortunately, \textbf{there is no good answer to
this question.}

One quick rule of thumb is that one should have $p < \sqrt{n}$, where
$p$ is the number of predictors, including polynomial and interaction
terms (not to be confused with the quantity of the same name in our
polling example above), and $n$ is the number of cases in our sample.
But this is certainly not a firm rule by any means.  One may be able to
get good prediction with a considerably larger value of $p$, especially
if \textit{regularization} methods are used (Chapter \ref{chap:svd}).

From the polynomial-ftting example in Section \ref{norelation}, we see
the following key point:

\begin{quote}
An assessment of predictive ability, based on predicting the same data
on which our model is fit, tends to be overly optimistic and may be
meaningless or close to it.
\end{quote}

This motivates the most common approach to dealing with the
bias/variance tradeoff, \textit{cross validation}.  In the simplest
version, one randomly splits the data into a \textit{training set} and a
\textit{test set}.\footnote{The latter is also called a \textit{holdout
set} or a \textit{validation set}.}  We fit the model to the training
set and then, pretending we don't know the ``Y'' (i.e. response) values
in the test set, predict those values from our fitted model and the
``X'' values (i.e.\ the predictors) in the test set.  We then
``unpretend,'' and check how well those predictions worked. 

The test set is ``fresh, new'' data, since we called \textbf{lm()} or
whatever only on the training set.  Thus we are avoiding the ``noise
fitting'' problem.  We can try several candidate models, then choose the
one that best predicts the test data.

(Note carefully that after fitting the model via cross-validation, we
then use the full data for later prediction.  Splitting the data for
cross-validation was just a temporary device for model selection.)

Cross-validation is essentially the standard for model selection, and it
works well if we only try a few models.  Problems can occur if we try
many models, as seen in the next section.

\subsection{The Problem of P-hacking}

The (rather recent) term \textit{p-hacking} refers to the following
abuse of statistics.\footnote{The term \textit{abuse} here will not
necessarily connote intent. It may occur out of ignorance of the
problem.}

Say we have 250 pennies, and we wish to determine whether any are
unbalanced, i.e.\ have probability of heads different from 0.5.  We do
so by tossing each coin 100 times.  If we get fewer than 40 heads or
more than 60, we decide this coin is unbalanced.\footnote{For those who
know statistics:  This gives us a Type I error rate of about 0.05, the
standard used by most people.}  The problem is that, even if all the
coins are perfectly balanced, we eventually will have one that has fewer
than 40 or greater than 60 heads, just by accident.  \textbf{We will
then \underline{falsely} declare this coin to be unbalanced.}

Or, to give a somewhat frivolous example that still will make the point,
say we are investigating whether there is any genetic component to a
person's sense of humor.  Is there a Humor gene?  There are many, many
genes to consider.  Testing each one for relation to sense of humor is
like checking each penny for being unbalanced: Even if there is no Humor
gene, then eventually, just by accident, we'll stumble upon one that
seems to be related to humor.\footnote{For those with background in
statistics, the reason this is called ``p-hacking'' is that the
researcher may form a significance test for each gene, computing a
p-value for each test.  Since under the null hypothesis we have a 5\%
chance of getting a ``significant'' p-value for any given gene, the
probability of having at least one significant result out of the
thousands of tests is quite high, even if the null hypothesis is true in
all cases.  There are techniques called \textit{multiple inference} or
\textit{multiple comparison} methods, to avoid p-hacking in performing
statistical inference.  See for example \textit{Multiple Comparisons:
Theory and Methods}, Jason Hsu, 1996, CRC.}

Though the above is not about prediction, it has big implications for
the prediction realm.  In ML there are various datasets on which
analysts engage in contests, vying for the honor of developing the model
with the highest prediction accuracy, say for classification of images.
If there is a large number of analysts competing for the prize, then
even if all the analysts have models of equal accuracy, it is likely
that one is substantially higher than the others, just due to an
accident of sampling variation.  This is true in spite of the fact that
they all are using the same sample; it may be that the ``winning''
analyst' model happens to do especially well in the given data, and may
not be so good on another sample from the same population.  So, when some
researcher sets a new record on a famous ML dataset, it may be that the
research really has found a better prediction model --- or it may be
that it merely looks better, due to p-hacking.

The same is true for your own analyses.  If you try a large number of
models, the ``winning'' one may actually not be better than all the
others.

\section{Extended Example}
\label{prgengex}

Let's illustrate this on the dataset \textbf{prgeng}, assembled from the
2000 US census.  It consists of wage and other information on programmers and
engineers in Silicon Valley.  This dataset is included in the R
\textbf{polyreg} package, which fits polynomial models as we saw in
Section \ref{poly} above.\footnote{Available from
\url{github.com/matloff}.}

\begin{lstlisting}
> getPE()  # produces data frame pe
> pe1 <- pe[,c(1,2,4,6,7,12:16,3)]  # choose some predictors
> head(pe1)
       age sex wkswrkd ms phd occ1 occ2 occ3 occ4 occ5
1 50.30082   0      52  0   0    0    0    1    0    0
2 41.10139   1      20  0   0    0    1    0    0    0
3 24.67374   0      52  0   0    0    0    1    0    0
4 50.19951   1      52  0   0    1    0    0    0    0
5 51.18112   0       1  0   0    1    0    0    0    0
6 57.70413   1       0  0   0    1    0    0    0    0
wageinc
1   75000
2   12300
3   15400
4       0
5     160
6       0
\end{lstlisting}

By the way, note the dummy variables.  We have just two levels for
education, so anyone with just a bachelor's degree or less, or a
professional degree, with be ``other,'' coded by \textbf{ms} and
\textbf{phd} both having the values 0.  Similarly, there are actually
six occupations, hence five dummies; the sixth occupation is coded by
having 0s for the five occupation dummies.

Well, then, let's see how well we can predict wage income. First, split
the data:

\begin{lstlisting}
> set.seed(9999)
> testidxs <- sample(1:nrow(pe1),1000)
> testset <- pe1[testidxs,]
> trainset <- pe1[-testidxs,]
\end{lstlisting}

Then fit the model and predict:

\begin{lstlisting}
> lmout <- lm(wageinc ~ .,data=pe1)  # predict wage from all others
> predvals <- predict(lmout,testset[,-11])
> mean(abs(predvals - testset[,11]))
[1] 26494.62
\end{lstlisting}

An average, our prediction is off by more than \$26,000, not so good,
but not the main point here.

Let's try a quadratic model.  Again, we could form the squared terms
ourselves, but \textbf{polyreg} makes it convenient.

\begin{lstlisting}
> pfout <- polyFit(pe1,deg=2)
> mean(abs(predict(pfout,testset[,-11]) - testset[,11]))
[1] 25885.43
\end{lstlisting}

Ah, some improvement, with a second-degree model.  Note that this
includes the squares of all predictors, and their products, e.g.\ age
times weeks worked.  Altogether there are now $p = 46$ predictors, up
from our original 10:

\begin{lstlisting}
> dim(pfout$poly.xy)
[1] 20090    47
\end{lstlisting}

How about a cubic model? 

\begin{lstlisting}
> pfout <- polyFit(pe1,deg=3)
> mean(abs(predict(pfout,testset[,-11]) - testset[,11]))
[1] 25410.9
\end{lstlisting}

Even better, though it may be largely a difference due to sampling
variation.  We are now up to $p = 118$.

\begin{lstlisting}
> pfout <- polyFit(pe1,deg=4)
getPoly time:  1.12 0.056 1.177 0 0 
lm() time:  0.88 0.024 0.906 0 0 
> mean(abs(predict(pfout,testset[,-11]) - testset[,11]))
[1] 25380.65
Warning message:
In predict.lm(object$fit, plm.newdata) :
  prediction from a rank-deficient fit may be misleading
\end{lstlisting}

About the same, but with an ominous warning.  R found that the matrix
$A'A$ in (\ref{famouslm}) was close to nonfull-rank, thus nearly
singular (noninvertible).  Now $p = 226$.

Remarkably, this improvement continued:

\bigskip

\begin{tabular}{|r|r|}
\hline
deg & MAPE \\
\hline
6 & 25245.83 \\
\hline
10 & 24679.86 \\
\hline
15 & 24123.59 \\
\hline
\end{tabular}

And at degree 15, $p$ was a whopping 3791.  By the way, in that last
computation, it took more than 2 minutes to simply generate the
polynomials --- those 3791 columns --- and almost 20 minutes to process
the call to \textbf{lm()}.

Yet, even with this large model, we have not yet reached the point of
overfitting.  Eventually, though, MAPE will begin to rise.

