Intermediate Econometrics

• If \(Z \sim \mathcal{N}(0,1)\) then \(X = \mu + \sigma Z\) is distributed as \(X\sim \mathcal{N}(\mu, \sigma^2)\)
• Viceversa, if \(X\sim \mathcal{N}(\mu, \sigma^2)\) then the r.v. \(Z=\frac{X-\mu}{\sigma} \sim \mathcal{N}(0,1)\)

• We assume that \(Y_1, \ldots, Y_n\) are random variables independent and identically distributed according to the \(\mathcal{N}(\mu, \sigma^2)\) distribution
• Parameter: \(\theta = (\mu, \sigma^2) \in \Theta = \mathbb{R} \times \mathbb{R}_{+}\)
• The most popular estimators of \(\mu\) and \(\sigma^2\) are respectively \[\overline Y = \frac{1}{n} \sum_{i=1}^{n} Y_i \quad \quad S^2 = \frac{1}{n-1} \sum_{i=1}^{n} (Y_i-\overline Y)^2\] that is the sample mean and the sample variance, respectively
• The estimators above are unbiased and consistent (meaning that the variance tends to zero when \(n \to \infty\))

• \(\overline Y\) and \(S^2\) are independent
• \(\overline Y \sim \mathcal{N}(\mu, \sigma^2/n)\) and \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\), where \(\chi^2_{\nu}\) is the Chi-square distribution with \(\nu\) degrees of freedom

• For \(0<\alpha <1\) \[\rm{Pr}(t_{n-1;\alpha/2} \leq T \leq t_{n-1;1-{\alpha/2}})=1-\alpha \]
• The realization of the confidence interval is \[IC^{1-\alpha}_{\mu} = (\bar y - t_{n-1; 1-\alpha/2} s/\sqrt{n}, \bar y - t_{n-1; 1-\alpha/2} s/\sqrt{n})\]
  
• where \(-t_{n-1;1-\alpha/2}=t_{n-1;\alpha/2}\) and \(t_{n-1;1-\alpha/2}\) represent the quantiles of order \(\alpha/2\) and \(1-\alpha/2\) of a Student’s t distribution with \(n-1\) degrees of freedom, respectively.

• Let’s assume to be interested on the inference on a parameter \(\theta\) (unknown)

• Given a random sample \(y=(y_1, \ldots, y_n)\) of size \(n\) from \(Y\) which is distributed according to the model \(f(y; \theta)\), with \(\theta \in \Theta \subset R^d, d \geq 1\), we define the likelihood function for \(\theta\) the function as \(\mathcal{L}(\theta) = f(y_1, \ldots, y_n; \theta)\)

• It’s not a probability function: It’s a function of \(\theta\) which is defined on \(\Theta\) getting values in \(\mathbb{R}^{+}\)

• Under a random sampling, due to the independence condition of the random variables (and for the condition of identical distribution), we have \[\mathcal{L}(\theta) = f(y_1, \ldots, y_n; \theta) = \prod_{i=1}^{n}f(y_i; \theta)\]

• For convenience, it is better to work with the log-likelihood function

\[\ell(\theta) = \log \mathcal{L}(\theta) = \sum_{i=1}^{n} \log f(y_i; \theta)\]

• A value \(\hat \theta \in \Theta\) such that \(\ell(\hat \theta) \geq \ell(\theta)\) for all \(\theta \in \Theta\) is said maximum likelihood estimate (the random variable \(\hat \theta(Y)\) is said maximum likelihood estimator)
• Under regularity conditions (what we consider) the maximum likelihood estimate is obtained by solving \[\ell'(\theta) = \frac{\partial \ell(\theta)}{\partial \theta} = 0\] where the vector \(\ell'(\theta)\) is the score vector
• Finally, \(\hat \theta\) is consistent and has asymptotic distribution \(\hat \theta \stackrel{\cdot} \sim \mathcal{N}(\theta, j(\hat \theta)^{-1})\) where \[j(\theta) = -\ell''(\theta) = -\frac{\partial^2 \ell(\theta)}{\partial \theta \partial \theta^\top}\]

\[\mathcal{L}(\theta) = \mathcal{L}(\mu, \sigma^2) = \prod_{i=1}^{n}f(y_i; \mu, \sigma^2)= \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2\sigma^2} (y_i - \mu)^2} = \left(\frac{1}{2\pi\sigma^2}\right)^{n/2} e^{-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2}\] \[\ell(\mu, \sigma^2) = \log \mathcal{L}(\theta) = -\frac{n}{2} \log(2\pi) - \frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2\]

• From \(s_1 = 0\) we obtain \(\hat \mu = \bar y\)
• From \(s_2 = 0\) (and substituting \(\mu\) with \(\hat \mu\)), we obtain \(\hat \sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(y_i - \bar y)^2\)

• The settings above can be straightforwardly extended to linear regression models
• Let’s consider a simple linear regression \(Y_i = \beta_1 + \beta_2 x_i +\varepsilon_i\), where \(\varepsilon_i\sim\mathcal{N}(0, \sigma^2)\) independently
  
• Note that the difference is that we do not have the identically distribution because \(Y_i \sim \mathcal{N}(\mu_i, \sigma^2)\) where \(\mu_i=\beta_1 + \beta_2 x_i\)
• Let’s adapt the negative log-likelihood function to this case

• As we noticed the \(\beta\) estimates obtained via maximum likelihood are the same of those obtained via OLS.
• Indeed, from \[\mathcal{L}(\beta, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi} \sigma} e^{-\frac{1}{2\sigma^2} (y_i - \beta_1 x_{i1} - \ldots - \beta_p x_{ip})^2}\] \[ \propto (\sigma^2)^{-n/2} e^{-\frac{1}{2\sigma^2} \sum_{i=1}^{n}(y_i - \beta_1 x_{i1} - \ldots - \beta_p x_{ip})^2}\] \[ \propto (\sigma^2)^{-n/2} e^{-\frac{1}{2\sigma^2} (y - X \beta)^\top (y-X\beta)}\] \[\ell(\beta, \sigma^2) = - \frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2} (y - X \beta)^\top (y-X\beta)\]
• Note that the term that does not depend on the parameter has been removed and we introduce the proprionality symbol \(\propto\)

• For fixed \(\sigma^2\), \(\ell(\beta, \sigma^2)\) is maximum when the residual sum of square \(RSS(\beta) = (y - X^\top \beta)^\top (y-X\beta)\) is minimum, that is

\[\hat \beta = \operatorname*{argmin}_{\beta} \mathrm{RSS}(\beta) = \operatorname*{argmax}_{\beta} \ell({\beta}, \sigma^2)\]

• Then, the maximum likelihood estimate is equal to the OLS estimate
  \[\hat \beta_{ML} = \hat \beta_{OLS} = (X^\top X)^{-1} X^\top y\]

• By substituing \(\hat \beta\) in \(\ell(\beta, \sigma^2)\) we have \[\ell(\hat \beta, \sigma^2) = - \frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2} (y - X^\top \hat \beta)^\top (y-X\hat \beta)\] \[= - \frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2} RSS(\hat \beta) = - \frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}e^2_i\]
• By getting the derivative with respect to \(\sigma^2\) and equating to zero, we obtain

\[\hat \sigma^2= \frac{1}{n}\sum_{i=1}^{n}e^2_i = \frac{1}{n} e^\top e\]

• Since \(Y\sim \mathcal{N}_n(\mu, \sigma^2I)\), the estimator \(\hat \beta = (X^\top X)^{-1} X^\top Y\) is a linear transformation of a multivariate normal. So

\[{\hat \beta} \sim \mathcal{N}_p(\beta, \sigma^2(X^\top X)^{-1})\]

• Due to the results on the multivariate normal distribution, we know that also the marginals are normally distributed and in particular

\[\hat \beta_r \sim \mathcal{N} (\beta_r, \sigma^2(X^\top X)^{-1}_{rr}), \quad r=1, \ldots, p\] having denoted with \((X^\top X)^{-1}_{rr}\) the \(r\)-th diagonal element of \((X^\top X)^{-1}\).

• The estimator \(\sigma^2\) is biased, while the unbiased estimator is \[S^2 = \frac{n}{n-p}\hat \sigma^2\] for which \[\frac{(n-p)S^2}{\sigma^2} \sim \chi^2_{n-p}\]

• From the results just outlined, we can obtain pivotal quantities to make inference on a single coefficient \[t_r = \frac{\hat \beta_r - \beta_r}{\sqrt{S^2(X^\top X)^{-1}_{rr}}} \sim t_{n-p}\]
• Confidence interval of level \(1-\alpha\) for \(\beta_r\) \[IC^{1-\alpha}_{\beta_r}=(\hat \beta_r - t_{n-p; 1- \alpha/2} \sqrt{S^2((X^\top X)^{-1}_{rr})}, \hat \beta_r + t_{n-p; 1- \alpha/2} \sqrt{S^2((X^\top X)^{-1}_{rr})})\]
• Hypothesis test (nullity of the single coefficient): \[\begin{cases}H_0: \beta_r = 0 \\H_1: \beta_r \neq 0 \end{cases}\] \[t_r = \hat \beta_r/\sqrt{S^2(X^\top X)^{-1}_{rr}} \stackrel{H_0} \sim t_{n-p}\] \[\mathcal{R}_\alpha = \{|t^{obs}_r| > t_{n-p; 1-\alpha/2}\}\] \[p-value = P(|t_{n-p}| > |t^{obs}_{r}|) = 2P(t_{n-p}\leq -|t^{obs}_{r}|)\]

• Let’s suppose we want assess the model overall, that is no one of the predictors are explaining the variability of \(Y\). The hypothesis system is \[\begin{cases}H_0: \beta_2 = \ldots, \beta_p = 0 \\ \text{at least one of} \quad H_1: \beta_r \neq 0 \quad \text{for} \,\, r = 1, \ldots, p\end{cases}\]
• In this case we distinguish between \[ \text{Full model}(\mathcal{M_1}): Y_i = \beta_1 + \beta_2 x_{i2} + \ldots + \beta x_{ip} + \varepsilon_i\] \[ \text{Reduced model}(\mathcal{M_0}): Y_i = \beta_1 + \varepsilon_i\]
• Under \(\mathcal{M1}\) the estimate of the variance of the error term component is \(\hat \sigma^2 = \frac{1}{n}e^\top e\), while Under \(\mathcal{M0}\) is simply \(\tilde \sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(y_i - \bar y)^2\)
• The test statistic is \[F = \frac{\frac{\tilde \sigma^2 - \hat \sigma^2}{p-1}}{\frac{\hat \sigma^2}{n-p}} \stackrel{H_0}\sim F_{p-1, n-p}\]
• The p-value is given by \(p-value = Pr(F_{p-1;n-p} > f^{obs})\)