Zhejiang Universitypan.hohoweiya.xyz/regression-slides/chap 3.pdf · Zhejiang University September...

transcript

LSE LSE�5� �åLSE �ä BCC� GLSE õ��5 *�O PC�O Stein

1nÙ £8ëê��O

Tianxiao Pang

Zhejiang University

September 28, 2016

Tianxiao Pang 1nÙ £8ëê��O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

1 ��¦�O

2 ��¦�O�5�

3 �å��¦�O

4 £8�ä

5 Box-CoxC�

6 2Â��¦�O

7 õ��5

8 *�O

9 Ì¤©�O

10 SteinØ �O

�O£8ëê��Ä��{´��¦{(Least SquaresMethod). �Ùcn!?ØXÛA^��¦{¦£8ëê��¦�O, ¿ïÄù«�O�Ä�5�. 31oÚ1Ê!, ·�?Ø£8�.�Ä�b��·^5±9�ùb�Ø·^�, éêâAT��C�(Box-CoxC�). 318!·�ò?Ø2Â��¦�O. 31Ô!·�ò?Ø�«AÏ�gCþ, ¿?Ø§XÛ��¦�O�5�³. 1l!Ú1Ê!·�ò?Øü«#��O�{: *�OÚÌ¤©�O. ��!{ü0�SteinØ �O.

lùpm©, ·�b½gCþØ´�ÅCþ, Ï�§�� ±�<��. Ó�·��½: ��gCþ^��i1L«,��ÏCþ´�ÅCþ, �^��i1L«(k��L«��*ÿ�).

��¦�O

^yL«ÏCþ, x1, · · · , xpL«(�U)éykK��p�gCþ. b�§��m÷vXe��5'Xª:

y = β0 + β1x1 + · · ·+ βpxp + e, (3.1.1)

Ù¥e´�ÅØ�, β0, · · · , βpL«��ëê. ¡β0�£8~ê, ¡β1, · · · , βp�£8Xê. k�r§�Ú¡�£8Xê. b½®²k��

(xi1, · · · , xip, yi), i = 1, · · · , n,

K§�÷v

yi = β0 + β1xi1 + · · ·+ βpxip + ei, i = 1, · · · , n. (3.1.2)

b�Ø��ei, i = 1, · · · , n÷vGauss-Markovb�.

e^ÝL«, K��

y2...yn

1 x11 · · · x1p

1 x21 · · · x2p...

......

...1 xn1 · · · xnp

β1...βp

e2...en

½��

Y = Xβ + e. (3.1.3)

Ù¥Yń× 1��Å*ÿ�þ(ÏCþ�þ), Xń× (p+ 1)�®��OÝ(b�p+ 1 ≤ n), β´(p+ 1)× 1��ëê�þ, eń× 1��ÅØ��þ.

òGauss-Markovb��¤Ý/ª:

E(e) = 0,Cov(e) = σ2In. (3.1.4)

ò(3.1.3)Ú(3.1.4)Ü�3�å, =��Ä��5£8�.:

Y = Xβ + e, E(e) = 0, Cov(e) = σ2In. (3.1.5)

·�^��¦�{Ïéβ��O, ù��OÏd�¡��¦�O(LSE: Least Squares Estimator). ù��{´Ïé��β��O, ¦� ��þe = Y −Xβ��Ý�²�‖Y −Xβ‖2��.

Q(β) = ‖Y −Xβ‖2 = (Y −Xβ)′(Y −Xβ)

= Y′Y − 2Y

′Xβ + β

′Xβ.

éβ¦�, -Ù�u", ��§|

X′Xβ = X

′Y . (3.1.6)

¡ù��§|��5�§|(½�K�§|). ù��§|k��)�¿�^�´X

′X��´p+ 1, ù�duX��´p+ 1(=X

´�÷��). Ï�X ´�±<��, ¤±·�ob½X´�÷��. u´·��(3.1.6)��)

β = (X′X)−1X

′Y . (3.1.7)

±þ�?Ø�U`²β´Q(β)��7:, ��7Ò´��:.eyβ(¢´Q(β)��:.

é?¿�β, k

‖Y −Xβ‖2 = ‖Y −Xβ +X(β − β)‖2

= ‖Y −Xβ‖2 + (β − β)′X

′X(β − β)

+2(β − β)′X

′(Y −Xβ). (3.1.8)

Ï�β÷v�5�§|(3.1.6), ¤±X′(Y −Xβ) = 0. ùy²

é?¿�β, k

‖Y −Xβ‖2 = ‖Y −Xβ‖2 + (β − β)′X

′X(β − β). (3.1.9)

duX′X´�K½Ý, ¤±(β − β)

′X(β − β) ≥ 0. u´

‖Y −Xβ‖2 ≥ ‖Y −Xβ‖2.

�y.Tianxiao Pang 1nÙ £8ëê��O

Pβ = (β0, β1, · · · , βp)′, ·��±��(²�)£8�§:

Y = Xβ ½ y = β0 + β1x1 + · · ·+ βpxp. (3.1.10)

ù��§´Ø´£ãy�x1, · · · , xp�ý¢'X, �I�?�Ú�ÚO©Û, 3�±�?n.

~3.1.1��5£8. b�gCþ�k��, P�x. ��(xi, yi), i = 1, · · · , n. u´k�5£8�.

yi = β0 + β1xi + ei, i = 1, · · · , n.

ù��5�§|�n

n∑i=1

��OÝX´�÷��, =xi, i = 1, · · · , nØ��,n∑i=1

(xi − x)2 6= 0, u´β0Úβ1�LSE�

β0 = y − β1x,

∑ni=1(xi − x)(yi − y)∑n

i=1(xi − x)2=:

SxySxx

3£8©Û¥, ·�k�r�©êâ?1¥%zÚIOz. -

n∑i=1

xij , j = 1, · · · , p.

ò(3.1.2)U��

yi = α+ β1(xi1 − x1) + · · ·+ βp(xip − xp) + ei,

i = 1, · · · , n. (3.1.11)

ùp, α = β0 + β1x1 + · · ·+ βpxp. ¡(3.1.11)�¥%z�.. P

x11 − x1 x12 − x2 · · · x1p − xpx21 − x1 x22 − x2 · · · x2p − xp

......

xn1 − x1 xn2 − x2 · · · xnp − xp

. (3.1.12)

K(3.1.11)�U��

Y = α1n +Xcβ + e =(1n Xc

)(αβ

)+ e. (3.1.13)

ùp, β = (β1, · · · , βp)′. 5¿�

1′nXc = 0 (3.1.14)

Ïd�5�§|��(n 0

0 X′cXc

)(αβ

X′cY

), (3.1.15)

�d/�¤ {nα = 1

′nY ,

X′cXcβ = X

(3.1.16)

u´£8ëê�LSE�{α = y,

β = (X′cXc)

−1X′cY .

(3.1.17)

Ïd, éu¥%z�5£8�.(3.1.11), £8~ê�LSEo´ÏCþ��þ�, £8Xêβ�LSE β = (X

′cXc)

−1X′cY�du

l�5£8�.Y = X′cβ + e¥O�β�LSE. 3¢SA^¥, O

�(X′cXc)

−1o´'O�(X ′X)−1��B�:, �·�o´AO'%£8Xê, ¤±¥%z´kÐ?�.

~3.1.2��5£8(Y). ò~3.1.1¥��5£8�.?1¥%z, �

yi = α+ β1(xi − x) + ei, i = 1, · · · , n. (3.1.18)

dúª(3.1.17)�LSEα = y,

∑ni=1(xi − x)yi∑ni=1(xi − x)2

∑ni=1(xi − x)(yi − y)∑n

i=1(xi − x)2.

(3.1.19)

,, ·��±égCþ�IOz?n. P

n∑i=1

(xij − xj)2, j = 1, · · · , p,

zij =xij − xjsj

, i = 1, · · · , n; j = 1, · · · , p.

-Z = (zij)n×p, §äk5�:

1′nZ = 0,

R = Z′Z = (rij),

∑nk=1(xki − xi)(xkj − xj)

sisj, i, j = 1, · · · , p (3.1.20)

�gCþxi�xj��'Xê. ¤±R´gCþ��'XêÝ.

égCþ?1IOz�Ð?: (1)�±^5©Û£8gCþ�m��''X; (2)IOz�Øþj�K�, Bué£8Xê�O��ÚO©Û.

²LIOz��5£8�.�

yi = α+xi1 − x1

s1β1 + · · ·+ xip − xp

spβp + ei, (3.1.21)

½�¤Ý/ª

Y = α1n +Zβ + e =(1n Z

)(αβ

��ëê�LSE�

α = y, β = (β1, · · · , βp)′

= (Z′Z)−1Z

′Y .

£8�§�

y = α+x1 − x1

s1β1 + · · ·+ xp − xp

spβp. (3.1.22)

~3.1.3��Á�Nì��ðøA9þ, ¦Ù�±ð§, eL¥,gCþxL«Nì±��íü �m�²þ§Ý(◦C), yL«ü �mS�Ñ��ðþ(L), �*ÿ25 ��mü . ã3.1.1´ùêâ�Ñ:ã, éù|êâ, A^¥%z�5£8�.(3.1.18),·��

y = 9.424, x = 52.6,

£8ëê�LSE�

α = y = 9.424, β1 = −0.0798.

¤±£8�§�

y = 9.424− 0.0798(x− 52.6),

½�¤y = 13.623− 0.0798x.

L3.1.1: �ðêâ

SÒ y(L) x(◦C) SÒ y(L) x(◦C)1 10.98 35.3 14 9.57 39.12 11.13 29.7 15 10.94 46.83 12.51 30.8 16 9.58 48.54 8.40 58.8 17 10.09 59.35 9.27 61.4 18 8.11 706 8.73 71.3 19 6.83 707 6.36 74.4 20 8.88 74.58 8.5 76.7 21 7.68 72.19 7.82 70.7 22 8.47 58.1

10 9.14 57.5 23 8.86 44.611 8.24 46.4 24 10.36 33.412 12.19 28.9 25 11.08 28.613 11.88 28.1

ã3.1.1 Ñ:ã

yx=read.table(”ex p33 data.txt”)y=yx[, 1]x=yx[, 2]mydata=data.frame(y,x)plot(y∼x)lm.sol=lm(y∼x,data=mydata)abline(lm.sol)summary(lm.sol)

��£8�§:

y = 13.623− 0.0798x.

��¦�O�5�

��¦�O(LSE)äk�ûÐ�5�:

½n (3.2.1)

éu�5£8�.(3.1.5), LSE β = (X′X)−1X

′Yäke�5�:

(a) E(β) = β;(b) Cov(β) = σ2(X

′X)−1.

y²: (a)´�E(Y ) = Xβ, ¤±

E(β) = (X′X)−1X

′ · E(Y ) = β.

(b)Ï�Cov(Y ) = Cov(e) = σ2In, ¤±

Cov(β) = Cov((X′X)−1X

′Y )

= (X′X)−1X

′Cov(Y )X(X

′X)−1

= (X′X)−1X

′σ2InX(X

′X)−1 = σ2(X

′X)−1.

�c´p+ 1��~ê�þ, éu�5¼êc′β(ù´��ëê),

·�¡c′β�c

′β�LSE.

íØ (3.2.1)

(a) E(c′β) = c

′β;

(b) Cov(c′β) = σ2c

′X)−1c.

=é?¿��5¼êc′β, c

′β�c

′β�Ã �O, ��

σ2c′(X

′X)−1c. Ï�c

′β = c

′X)−1X

′Y�y1, · · · , yn��5

¼ê, ¤±c′β�c

′β��5Ã �O(�5�O��´*

ÿy1, · · · , yn��5¼ê). ·��±�EÑc′β�Ù§�5Ã

�O. ù�¤c′β��5Ã �Oa.

½n (3.2.2, Gauss-Markov)

éu�5£8�.(3.1.5), 3c′β�¤k�5Ã �O¥, ��

¦�Oc′β´��5Ã �O(BLUE: best linear

unbiased estimator).

y²: �a′Y�c

′β��5Ã �O. u´é��p+ 1��

þβ,c′β = E(a

′Y ) = a

′Xβ,

a′X = c

′. (3.2.1)

Ï�Var(a′Y ) = σ2a

′a = σ2‖a‖2, ·�é‖a‖2�©):

‖a‖2 = ‖a−X(X′X)−1c+X(X

′X)−1c‖2

= ‖a−X(X′X)−1c‖2 + ‖X(X

′X)−1c‖2

+2c′(X

′X)−1X

′(a−X(X

′X)−1c). (3.2.2)

d(3.2.1)�í�

2c′(X

′X)−1X

′(a−X(X

′X)−1c) = 2c

′X)−1(X

′a−c′) = 0.

díØ3.2.1(b)�í�

σ2‖X(X′X)−1c‖2

= σ2c′(X

′X)−1X

′X(X

′X)−1c

= σ2c′(X

′X)−1c

= Var(c′β).

u´, ·�y²éc′β�?��5Ã �Oa

Var(a′Y ) = Var(c

′β) + σ2‖a−X(X

′X)−1c‖2

≥ Var(c′β), (3.2.3)

�Ò¤á��=�‖a−X(X′X)−1c‖ = 0, =a = X(X

′X)−1c,

d�a′Y = c

′β. ½n�y.

3�5£8�.(3.1.5)¥�k��ëêσ2(Ø��), §�N�.Ø��, 3£8©Û¥åX��^. ·�y35�Oσ2.

e = Y −Xβ´Ø��þ, Ø�*ÿ. ^β�Oβ, ¡

e = Y −Xβ = Y − Y (3.2.4)

�í�(residual)�þ. Px′i��OÝX�1i1, K

ei = yi − x′iβ, i = 1, · · · , n (3.2.5)

�1ig*ÿ�í�. g,/, ·��±òew�e��O. ·�ò^

RSS = e′e =

n∑i=1

e2i (3.2.6)

5�Eσ2�Ã �Oþ.Tianxiao Pang 1nÙ £8ëê��O

RSS=Residual Sum of Squares, L«í�²�Ú. §�N¢Sêâ�nØ�.(3.1.5)� l§Ý½ö`[Ü§Ý. RSS��L«êâ��.[Ü��Ð.

½n (3.2.3)

(a) RSS = Y′(In −X(X

′X)−1X

′)Y =: Y

′(In −H)Y ;

(b) σ2 = RSS/(n− p− 1)´σ2�Ã �Oþ.

5: ¡H = X(X′X)−1X

′�lf(hat)Ý, §´��é¡��

y²: (a)

RSS = e′e = (Y −Xβ)

′(Y −Xβ)

= [(In −X(X′X)−1X

′)Y ]

′[(In −X(X

′X)−1X

′)Y ]

= Y′(In −X(X

′X)−1X

′)Y .

(b)dE(Y ) = Xβ,Cov(Y ) = σ2In±9½n2.2.1�

E(RSS) = E[Y′(In −X(X

′X)−1X

′)Y ]

= β′X

′(In −X(X

′X)−1X

′)Xβ

+σ2tr(In −X(X′X)−1X

= σ2[n− tr(X(X′X)−1X

′)].

�â,�5�tr(AB) = tr(BA)�

tr(X(X′X)−1X

′) = tr((X

′X)−1X

′X) = tr(Ip+1) = p+ 1.

u´E(RSS) = σ2(n− p− 1).

XJb�Ø��þeÑl��©Ù, =e ∼ N(0, σ2In), @o·��±��βÚσ2�õ��5�.

½n (3.2.4)

éu�5£8�.(3.1.5), eØ��þe ∼ N(0, σ2In), K(a) β ∼ N(β, σ2(X

′X)−1);

(b) RSS/σ2 ∼ χ2(n− p− 1);(c) β�RSS�pÕá.

y²: (a) 5¿�Y ∼ N(Xβ, σ2In)±9β = (X′X)−1X

′Y´Y

��5C�, @od½n2.3.4B�í�(a).

(b) �â½Â

RSS = Y′(In −H)Y = Y

′(In −H)Y = Y

′NY ,

ùpN = In −H, §´��é¡��Ý. 5¿�NX = 0, ¤±

RSS = (Xβ + e)′N(Xβ + e) = e

′Ne. (3.2.7)

5¿�e ∼ N(0, σ2In), ¤±�â½n2.4.3·��Iy²N��n− p− 1. |^��Ý��u§�,ù�5�, ·��

rk(N) = tr(In −X(X′X)−1X

= n− tr(X(X′X)−1X

= n− tr((X′X)−1X

= n− p− 1.

(c) Ï�β = β + (X′X)−1X

′e, RSS = e

′Ne, 5¿�

(X′X)−1X

′N = 0,

¤±d½n2.4.5��(X′X)−1X

′e�RSS�pÕá, =β�RSS�

pÕá.

�β�1��©þ�~ê�β0�, �c = (0, · · · , 0, 1, 0, · · · , 0)′, Ù

¥1 3c�1i+ 1� �, Kc′β = βi. 2Pβ = (β0, · · · , βp)

′, u

´c′β = βi. 2^(A)iiL«ÝA�1(i, i)��, @o·Xeí

íØ (3.2.2)

éu�5£8�.(3.1.5), ee ∼ N(0, σ2In), K(a) βi ∼ N(βi, σ

2[(X′X)−1]i+1,i+1);

(b)3βi, i = 1, · · · , p��5Ã �O¥, βi, i = 1, · · · , p´��ö.

ò½n3.2.1Ú½n3.2.4A^u¥%z�.(3.1.13), Kk

íØ (3.2.3)

éu¥%z�.(3.1.13), 5¿ùp�β = (β1, · · · , βp)′, k

(a) E(α) = α,E(β) = β, ùpα = y, β = (X′cXc)

−1X′cY ;

)= σ2

0 (X′cXc)

(c)e?�Úb�e ∼ N(0, σ2In), K

α ∼ N(α,σ2

n), β ∼ N(β, σ2(X

′cXc)

−1),

�α�β�pÕá.

R2 =ESS

TSS, (3.2.8)

n∑i=1

(y − yi)2 = (Y − y1n)′(Y − y1n)

¡�£8²�Ú(½)º²�Ú: Explained Sum of Squares),

n∑i=1

(yi − y)2 = (Y − y1n)′(Y − y1n)

¡�o �²�Ú(½¡�o²�Ú: Total Sum of Squares).¡R2��½Xê½ÿ½Xê, ¡R =

√R2�E�'Xê.

d�5�§|�±y²

n∑i=1

(yi − y)2

n∑i=1

(yi − yi + yi − y)2

n∑i=1

(yi − yi)2 +

n∑i=1

(yi − y)2 + 2

n∑i=1

(yi − yi)(yi − y)

=n∑i=1

(yi − yi)2 +

n∑i=1

(yi − y)2.

=TSS = ESS + RSS.

ÏdR2Ýþ£8gCþx1, · · · , xpéÏCþy�[Ü§Ý�Ð�. 0 ≤ R2 ≤ 1, §��, L²y�ÃgCþk��'X.

e¦^¥%z�.(3.1.13), @oESS�ÏLe�úªO�:

ESS = β′X

′cY = Y

′Xc(X

′cXc)

−1X′cY ,

ùpβ = (β1, · · · , βp)′. ¯¢þ, dY = α1n +Xcβ9úª

(3.1.17)��Y − y1n = Y − α1n = Xcβ.

ESS = (Y − y1n)′(Y − y1n) = (Xcβ)

′Xcβ

= β′X

′c ·Xc(X

′cXc)

−1X′cY

= β′X

′cY .

éu��5£8�.

yi = β0 + β1xi + ei, i = 1, · · · , n.

�±y²R2 = r2, Ù¥r��.¥gCþ�ÏCþ��'Xê.

¯¢þ, e·�r�.¥%z:

yi = α+ β1(xi − x) + ei, i = 1, · · · , n,

@o��

∑ni=1(xi − x)yi∑ni=1(xi − x)2

ESS = β1

n∑i=1

(xi − x)yi =

[∑ni=1(xi − x)yi

]2∑ni=1(xi − x)2

[∑ni=1(xi − x)(yi − y)

]2∑ni=1(xi − x)2

R2 =ESS

[∑ni=1(xi − x)(yi − y)

]2∑ni=1(xi − x)2

∑ni=1(yi − y)2

~3.2.1�â²��, 3<��p��^�e, ÙÉØ�Â Øy�Nx1!c#x2k'. yÂ813¶If�ÿþêâ, �eL. Áïáy'ux1, x2��5£8�§.

L3.2.1: ÉØêâ

SÒ xi1 xi2 yi SÒ xi1 xi2 yi1 152 50 120 8 158 50 1252 183 20 141 9 170 40 1323 171 20 124 10 153 55 1234 165 30 126 11 164 40 1325 158 30 117 12 190 40 1556 161 50 125 13 185 20 1477 149 60 123

|^¥%z�.

yi = α+ β1(xi1 − x1) + β2(xi2 − x2) + ei, i = 1, · · · , 13.

²O��

13∑i=1

xi1 = 166.8, x2 =1

13∑i=1

xi2 = 38.85, y =1

13∑i=1

yi = 130,

−14.08 11.1516.92 −18.854.92 −18.85−1.08 −8.85−8.08 −8.85−5.08 11.15−17.08 21.15−8.08 11.153.92 1.15−13.08 16.15−2.08 1.1523.92 1.1518.92 −18.85

�5�§|X′cXcβ = X

′cY�{

2078.92β1 − 1533.85β2 = 1607,−1533.85β1 + 2307.69β2 = −715.

)�β1 = 1.068, β2 = 0.4, α = y = 130. ¤±£8�§�

y = α+ β1(x1 − x1) + β2(x2 − x2)

= 130 + 1.068× (x1 − 166.8) + 0.4× (x2 − 38.85)

= −62.963 + 1.068x1 + 0.4x2.

d, ��

ESS = β′X

′cY = 1430.276, TSS = 1512,

¤±R2 = 1430.276/1512 = 0.9459.

yx=read.table(”ex p39 data.txt”)x1=yx[, 1]x2=yx[, 2]y=yx[, 3]mydata=data.frame(y,x1,x2)lm.sol=lm(y∼x1+x2,data=mydata)summary(lm.sol)

dR$1(J�: y = −62.963 + 1.068x1 + 0.4x2, R2 = 0.9461.

�å��¦�O

éβØ\?Û�å^��/e, ·�?Ø§�LSE±9§�Ä�5�. �3�AÏ|Ü, ~Xb�u�¯K, ·�I�¦�k�½�å^��LSE.

Aβ = b (3.3.1)

´��N�5�§|(=�§|k)), Ù¥A´k × (p+ 1)®�Ý, ��k, b´k × 1®��þ. ·�^Lagrange¦f{¦�5£8�.(3.1.5)3÷v�5�å(3.3.1)��LSE.=3(3.3.1)ù�^�e¦β¦Q(β) = ‖Y −Xβ‖2��.

�E9Ï¼ê

F (β,λ) = ‖Y −Xβ‖2 + 2λ′(Aβ − b)

= (Y −Xβ)′(Y −Xβ) + 2λ

′(Aβ − b),

Ù¥λ = (λ1, · · · , λk)′�Lagrange¦f�þ. é¼êF (β,λ)'

uβ¦�¿-§�u0, �

−X ′Y +X

′Xβ +A

′λ = 0. (3.3.2)

y3, ·�I�)d(3.3.1)Ú(3.3.2)|¤�éá�§|.

·�^βcÚλcL«ù��§|�). d(3.3.2)�

βc = (X′X)−1X

′Y − (X

′X)−1A

′λc

= β − (X′X)−1A

′λc. (3.3.3)

�\(3.3.1)�

b = Aβc = Aβ −A(X′X)−1A

′λc,

��d/�¤

A(X′X)−1A

′λc = Aβ − b. (3.3.4)

duA��k, ¤±A(X′X)−1A

′´k × k�_Ý, Ïd(3.3.4)

k��)λc = (A(X

′X)−1A

′)−1(Aβ − b),

2ò§�\(3.3.3)�

βc = β − (X′X)−1A

′(A(X

′X)−1A

′)−1(Aβ − b). (3.3.5)

eyβc(¢´�5�åAβ = beβ�LSE. ù�Iy²(a) Aβc = b;(b)é��÷vAβ = b�β, Ñk‖Y −Xβ‖2 ≥ ‖Y −Xβc‖2.

d(3.3.5)�í�(a). �y(b), ò‖Y −Xβ‖2©)�

‖Y −Xβ‖2

= ‖Y −Xβ‖2 + ‖X(β − β)‖2

= ‖Y −Xβ‖2 + ‖X(β − βc + βc − β)‖2

= ‖Y −Xβ‖2 + ‖X(β − βc)‖2 + ‖X(βc − β)‖2,(3.3.6)

ùp�í�^�eã'X: (Y −Xβ)′X = 0±9é��÷

vAβ = b�β,

(β − βc)′X

′X(βc − β) = λ

′cA(βc − β) = λ

′c(Aβc −Aβ) = 0.

(3.3.6)L²: é��÷vAβ = b�β, ok

‖Y −Xβ‖2 ≥ ‖Y −Xβ‖2 + ‖X(β − βc)‖2, (3.3.7)

�Ò¤á��=�X(βc − β) = 0, =β = βc(Ï�X�÷�). Ïd3(3.3.7)¥^βc�Oβ, �Ò¤á. =

‖Y −Xβc‖2 = ‖Y −Xβ‖2 + ‖X(β − βc)‖2. (3.3.8)

y3, (Ü(3.3.7)Ú(3.3.8)B�í�(b).

·�rβc¡�β��å��¦�O, u´ke��½n:

½n (3.3.1)

éu�5£8�.(3.1.5), ÷v(3.3.1)��åLSE�

βc = β − (X′X)−1A

′(A(X

′X)−1A

′)−1(Aβ − b),

Ù¥β = (X′X)−1X

′Y´Ã�å^�e�LSE.

~3.3.13U©ÿþ¥, éU�¥�n�( :�¤�n�/ABC�n�S�θ1, θ2, θ3?1ÿþ, ��y1, y2, y3, du�3ÿþØ�, ¤±I�éθ1, θ2, θ3?1�O, ·�|^�5�.L«k'�þ

y1 = θ1 + e1,y2 = θ2 + e2,y3 = θ3 + e3,θ1 + θ2 + θ3 = π.

�¤Ý/ª {Y = Xβ + e,Aβ = b,

ùpY = (y1, y2, y3)′, β = (θ1, θ2, θ3)

′, X = I3, A = (1, 1, 1),

b = π. 5¿�β = (X′X)−1X

′Y = Y , A^½n3.3.1, ²O��

3∑i=1

yi − π)

θi = yi −1

3∑i=1

yi − π), i = 1, 2, 3

�θi��åLSE.

£8�ä

éu�5£8�.(3.1.5):

Y = Xβ + e, E(e) = 0, Cov(e) = σ2In,

·�3?Ø��ëê��¦�O±9�Oþ�©Ù�, �Xe��Ä��b�:

(a)�5b�: ÏCþ�gCþäk�5�''X;(b)��à5b�: Var(ei) = σ2, i = 1, · · · , n;(c)Ø�'5b�: Cov(ei, ej) = 0, i 6= j.(d)��5b�: ei ∼ N(0, σ2), i = 1, · · · , n.

XJùb�Ø¤á, @o·�±c?Ø��¦�O±9§�ÚO5�Òk�U´Ø¤á�. Ïd, 3¢S¯K¥, �·�k�1êâ�, I��·��êâ´Ä÷v½öÄ�÷vùb�. ù´�.�ä�SN.

Ï�ùb�Ñ��ÅØ�ek', í��þe�w¤´e��O, Ïd·��±ÏLí�5©Û4�Ä�b�´Ä¤á. �Ï�ù��Ï, ùÜ©SN��¡�í�©Û.

ØI�é�.?1�ä, ·��I�éêâ��?1�ä, §�¹É~:�äÚrK�:�ä.

3£8©Û¥, ¤¢É~:(q�l+:)´�éQ½�. lé��êâ:. �§e��O(�½Â´��(J�, �8�vkÚ��½Â. 8c, éÉ~:��61�w{´: (1)É~:´�@�ý�õêêâ:²wØ�N�êâ:. ù�É~:�n)�¤b½©Ù¥�4à:;(2)É~:Ò´@À/:, =´��ý�õêêâ:Ø´5gÓ�©Ù��Oêâ:.

É~:�·\òéëê��OE¤K�, Ïd·�I�uÿÑÉ~:¿ò§íØ.

êâ8¥�rK�:´�éÚOíä(ëê�O!b�u��)�)��K��êâ:, 3d�§¥, A�é£8ëêβ��¦�Ok��K��êâ:.

éz�|êâ:(x′i, yi), ·�F"§é£8ëê��Ok�½�

K�, �qF"ù«K�ØU��. ù�, ·��²�£8�§Òk�½�½5. ÄK, XJ�O�ü|êâé�OkÉ~��K�, �·�GØùêâ��, ÒU��5�Éé��²�£8�§.

Ïd3£8©Û¥, ·�I��z|êâéëê�O�K��. ùÜ©SN�¡�K�©Û.

uÿÑrK�:�, ·�I�Ø�ùêâ´Ä�~, eØ�~KíØ�, ÄK�ÄÂ8�õ�êâ½æ^�è�O�{± �/DºrK�:é�O�K�.

É~:ÚrK�:´ü�ØÓ�Vg. l�¡�©Û�±wÑ,§��mQk�½�éX�k�½�«O.

rK�:�UÓ�´É~:��UØ´, ��, É~:�UÓ�q´rK�:��UØ´.

Ïd, £8�ä�¹êâ��ä��.��äùüÜ©SN. Ù¥êâ��ä�)É~:�äÚrK�:�ä, �.��ä�¹�5�ä!��à5�ä!Ø�'5�äÚ��5�ä4 Ü©SN.

k5?Øí�©Û:

éu�5£8�.:

Y = Xβ + e, E(e) = 0, Cov(e) = σ2In, (3.4.1)

·�^

ei = yi − x′iβ, i = 1, · · · , n (3.4.2)

L«1ig*ÿ�í�, ¿r§w¤´ei��O. e�.(3.4.1)�(, @oeiAäkei��5G.

PY = Xβ, ¡Y�[Ü��þ, ¡yi = x′iβ�1i�[Ü�. ´�

Y = X(X′X)−1X

′Y = HY , (3.4.3)

Ù¥H = X(X′X)−1X

′�¡�lfÝ, §´é¡��Ý.

í�e��L«�

e = Y − Y = (I −H)Y = (I −H)e. (3.4.4)

I −H�´��é¡��Ý.

½n (3.4.1)

e(3.4.1)¤á, K(a) E(e) = 0, Cov(e) = σ2(I −H);(b) Cov(Y , e) = 0.(c)e?�Úb�e ∼ N(0, σ2I), K

e ∼ N(0, σ2(I −H)).

y²: N´, Ñ.

Ï�Var(ei) = σ2(1− hii)(hiiL«ÝH�1i�é��), �à5, ùkNuei�¢SA^. Ïd·��Ä¤¢�Æ)zí�

ri =ei

1− hii, i = 1, · · · , n, (3.4.5)

ùpσ2 = RSS/(n− p− 1) = e′e/(n− p− 1).

=¦3e ∼ N(0, σ2I)�^�e, ri�©ÙE,'�E,, ��Cq/@�ri�pÕá�ÑlN(0, 1)(��FW(1987)). d½n3.4.1��{ri, i ≥ 1}�{yi, i ≥ 1}Õá({ei, i ≥ 1}�{yi, i ≥ 1}�Õá).

í�ã´±,«í�(Æ)zí�ri½ÊÏí�ei)�p�I, ±?ÛÙ§�þ�î�I�Ñ:ã. c®�Ñí��Ø�ei��OAT�ei��Ø�, ��âí�ã5G´Ä�Ak�5��,Ò�±é�.b��Ün5Jø�k^�&E.

e¡·�±[Ü�yi�î�I, Æ)zí�ri�p�I�í�ã�~?Øí�ã�äNA^. ��J�´, Ï~�¹e, ±ÊÏí�ei�p�IÚ±Æ)zí�ri�p�I�í�ã/G��Ó, ±,�gCþxj�î�I½ö±SÒi�î�IÚ±[Ü�yi�î�I�í�ã/G��Ó.

�5�ä : e�5b�¤á, @oeiØ�¹5ggCþ�?Û&E, Ïdí�ãØA¥y?Ûk5K�/G, ÄKknd~¦�5b�Ø¤á.

��à5�ä : e��à5, @oí�ãþ�:´/þ!0ÑÙ�, ÄK, í�ãÏ~¬¥y//i.0½/�/i.0½üöo k��/G.

Ø�'5�ä : eÕá5¤á, @oí�ãþ�:Ø¥y5K5, ÄK, Ñ:ãò¥y”8ì5”½/ì��50�/G.

��5�ä : e��5¤á, @oÆ)zí�ri�Cqw¤´�pÕá�ÑlN(0, 1). ¤±, 3±ri�p�Iyi�î�I�í�ãþ, ²¡þ�:(yi, ri), i = 1, · · · , n ��Aá3°Ý�4�Y²�|ri| ≤ 2«�S(ù�ªÇA395%�m), �Ø¥y?Ûª³.

·��±^Æ)zí��QQ ã5��5�ä. �|Nþ�n�êâ'u,�©ÙF (x)�QQãÒ´±êâ�i/n© ê�p�I, ±F (x)�i/n© ê�î�I�Ñ:ã.

XJêâ´5gF (x)��{ü�Å��, KùÑ:A��3�^��þ,ÏdÏL�äQQã´ÄCq3�^��þ��¤��êâ8´ÄÑl¤�½©Ù��ÚO�â.

d, ·��±Shapiro-Wilk�{5��5u�.

��5�ä : e��5¤á, @oÆ)zí�ri�Cqw¤´�pÕá�ÑlN(0, 1). ¤±, 3±ri�p�Iyi�î�I�í�ãþ, ²¡þ�:(yi, ri), i = 1, · · · , n ��Aá3°Ý�4�Y²�|ri| ≤ 2«�S(ù�ªÇA395%�m), �Ø¥y?Ûª³.

·��±^Æ)zí��QQ ã5��5�ä. �|Nþ�n�êâ'u,�©ÙF (x)�QQãÒ´±êâ�i/n© ê�p�I, ±F (x)�i/n© ê�î�I�Ñ:ã.XJêâ´5gF (x)��{ü�Å��, KùÑ:A��3�^��þ,ÏdÏL�äQQã´ÄCq3�^��þ��¤��êâ8´ÄÑl¤�½©Ù��ÚO�â.

d, ·��±Shapiro-Wilk�{5��5u�.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: Ù5G��5b�Ä��, Ïd�@��5b�e ∼ N(0, σ2I)´Ün�.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: í�ã¥y/i., Ïd@��à5b�ØÜn.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: í�ã¥y�/i., Ïd@��à5b�ØÜn.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: ã/¥yk5K�/G, Ïd@��5b�ØÜn, í�¥A¹kgCþ�&E.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: í�ã¥y8ì5, Ïd@�Ø�'5ØÜn.

0.0 0.2 0.4 0.6 0.8 1.0

©Û: í�ã¥yì��5, Ïd@�Ø�'5ØÜn.

lí�ã�äÑ5�U�/;¾0, �Ò´,b�^�Ø¤á,·�I�é¯K/éwe�0:

XJkwG¦·�~¦�5b�Ø¤á, @o·��±�Ä3£8gCþ¥O\,gCþ��g�, Xx2

1, x22½��x1x2�,

äNO\=gCþ�=�, IÀ¢S�J ½;

XJkwG¦·�~¦��à5b�Ø¤á, @o�±�ÄéÏCþ�·��C�¦#CþäkCq��(��½5C�), ½öæ^2Â��¦�O;

XJkwG¦·�~¦Ø�'5b�Ø¤á, @o�±�ÄéÏCþ�/�©0, ¦#CþäkCqÕá5;

XJkwG¦·�~¦��5b�Ø¤á, @·��±�ÄéÏCþ��5C�–Box-CoxC�. ��J�´, Box-CoxC�´�«nÜ£n�Y, e�!2�[0�.

��½5C� :

��ÅCþY�þ��µ, ��σ2, b��þ��¼ê, =b�σ2 = g(µ), Ù¥g´��®��¼ê. ~X, eY ∼ B(n, p),KE(Y ) = np, Var(Y ) = np(1− p) = µ(1− µ/n). yI�{��C�Z = f(Y ), ¦�Var(Z)�~ê. ùI�éÑ¼êf�L�ª. (ùp, ·�^��i1YÚZL«�ÅCþ, ��i1yÚzL«��Å�Cþ)

Pz = f(y), ¿-§3y = µ?TaylorÐm, �Cqª

f(y) = f(µ) + f′(µ)(y − µ).

òyU¤�ÅCþ, k

Z = f(Y ) = f(µ) + f′(µ)(Y − µ),

Ù��c = Var(Z) = [f′(µ)]2Var(Y ) = [f

′(µ)]2g(µ). �f

′(µ) =√

c/g(µ), K��

f(y) =

∫ √c/g(y)dy.

e¡´A�A~:

(1) g(µ)�µ¤�'�, Pg(µ) = aµ, K∫ √c

aydy = 2

ay + c

Ñ�~êØO, ��²��C�: Z =√Y .

(2)�g(µ)�µ2¤�'�, Pg(µ) = aµ2, �y > 0, K∫ √c

ay2dy =

aln y + c

Ñ�~êØO, ��éêC�: Z = lnY .

(3)�g(µ)�µ4¤�'�, Pg(µ) = aµ4, K∫ √c

ay4dy = −

Ñ�~êØO, ��êC�: Z = 1/Y .Tianxiao Pang 1nÙ £8ëê��O

3A^þ, Äklí�ãoÑ/��eσ2�µ�U�3�A«'X(=�O¼êg(·)), ,�lúª

f(y) =

∫ √c/g(y)dy

¦ÑéA�C�. éA«C�L�êâ©O��¦?n, �#�í�ã, w=�«C��í�ãÃ��à5��î. l¥ÀÑ�Ð��½5C�.

~3.4.1�ïÄ^>p¸z��^>þy�z�o^>þx�'X, yÂ8,�53rêâ.

L3.4.1: ^>þêâ

i x y yi ei z =√y zi ei

1 679 0.790 1.669 -0.879 0.889 1.229 -0.3402 292 0.440 0.244 0.196 0.663 0.860 -0.1973 1012 0.560 2.896 -2.336 0.748 1.547 -0.7984 493 0.790 0.984 -0.194 0.889 1.052 -0.1635 582 2.700 1.312 1.388 1.643 1.137 0.5066 1156 3.640 3.426 0.214 1.908 1.684 0.2247 997 4.730 2.840 1.890 2.175 1.532 0.6438 2189 9.500 7.230 2.270 3.082 2.668 0.4149 1097 5.340 3.209 2.131 2.311 1.628 0.68310 2078 6.850 6.822 0.028 2.617 2.562 0.05511 1818 5.840 5.864 -0.024 2.417 2.315 0.10212 1700 5.210 5.430 -0.220 2.283 2.202 0.08013 747 3.250 1.920 1.330 1.803 1.294 0.50914 2030 4.430 6.645 -2.215 2.105 2.517 -0.412

15 1643 3.160 5.220 -2.060 1.778 2.148 -0.37016 414 0.500 0.693 -0.193 0.707 0.977 -0.27017 354 0.170 0.472 -0.302 0.412 0.920 -0.50718 1276 1.880 3.868 -1.988 1.371 1.798 -0.42719 745 0.770 1.912 -1.142 0.877 1.292 -0.41520 435 1.390 0.771 0.619 1.179 0.997 0.18221 540 0.560 1.157 -0.597 0.748 1.097 -0.34822 874 1.560 2.388 -0.828 1.249 1.415 -0.16623 1543 5.280 4.851 0.429 2.298 2.052 0.24524 1029 0.640 2.958 -2.318 0.800 1.563 -0.76325 710 4.000 1.784 2.216 2.000 1.259 0.74126 1434 0.310 4.450 -4.140 0.557 1.949 -1.39227 837 4.200 2.251 1.949 2.049 1.380 0.67028 1748 4.880 5.606 -0.726 2.209 2.248 0.03929 1381 3.480 4.255 -0.775 1.865 1.898 -0.03330 1428 7.580 4.428 3.152 2.753 1.943 0.81031 1255 2.630 3.791 -1.161 1.622 1.778 -0.15632 1777 4.990 5.713 -0.723 2.234 2.275 -0.04233 370 0.590 0.531 0.059 0.768 0.935 -0.16734 2316 8.190 7.698 0.492 2.862 2.789 0.073

35 1130 4.790 3.330 1.460 2.189 1.659 0.53036 463 0.510 0.874 -0.364 0.714 1.023 -0.30937 770 1.740 2.004 -0.264 1.319 1.316 0.00338 724 4.100 1.835 2.265 2.025 1.272 0.75339 808 3.940 2.144 1.796 1.985 1.352 0.63340 790 0.960 2.078 -1.118 0.980 1.335 -0.35541 783 3.290 2.052 1.238 1.814 1.328 0.48642 406 0.440 0.664 -0.224 0.663 0.969 -0.30643 1242 3.240 3.743 -0.503 1.800 1.766 0.03444 658 2.140 1.592 0.548 1.463 1.209 0.25445 1746 5.710 5.599 0.111 2.390 2.246 0.14446 468 0.640 0.892 -0.252 0.800 1.028 -0.22847 1114 1.900 3.271 -1.371 1.378 1.644 -0.26548 413 0.510 0.690 -0.180 0.714 0.976 -0.26249 1787 8.330 5.750 2.580 2.886 2.285 0.60150 3560 14.940 12.280 2.660 3.865 3.974 -0.10951 1495 5.110 4.675 0.435 2.261 2.007 0.25452 2221 3.850 7.348 -3.498 1.962 2.699 -0.73653 1526 3.930 4.789 -0.859 1.982 2.036 -0.054

A^��¦{, �£8�§

y = −0.83130 + 0.00368x.

yx=read.table(”ex p47 data.txt”)x=yx[, 1]y=yx[, 2]mydata=data.frame(y,x)lm.sol=lm(y∼x,data=mydata)summary(lm.sol)

�í�©Û:

y.fit=predict(lm.sol)e.hat=y-y.fit /*½öe.hat=residuals(lm.sol)*/e.std=rstandard(lm.sol)plot(e.hat∼y.fit)plot(e.std∼y.fit)plot(e.hat∼x)

ÊÏí�ã��±ÏLXe�ª��:

plot(lm.sol,which=1)

lí�ã�wÑ, ù´��/i.í�ã, ´��à5Ø�ÎÜ��wG. �ÄéÏCþy�C�, }Áz =

√y, �£8�§

z = 0.5822 + 0.000953x.

z=sqrt(y)mydata2=data.frame(z,y,x)lm.sol2=lm(z∼x,data=mydata2)summary(lm.sol2)z.fit=predict(lm.sol2)e.hat=z-z.fit /*½öe.hat=residuals(lm.sol2)*/plot(e.hat∼z.fit) /*½ö^plot(lm.sol2,which=1)*/

#�í�ãØ¥y?Û²w5K5, ùL²·�¤^�C�´Ü·�. ��£8�§�

y = z2 = (0.5822+0.000953x)2 = 0.339+0.0011x+0.00000091x2.

�e5·��êâ��ä: É~:�äÚrK�:�ä.

É~:�ä : duÆ)zí�ri�Cqw¤´�pÕá�ÑlN(0, 1), @o|ri| > 2´��VÇ¯�, u)�VÇ��0.05. Ïd, ek,�|ri| > 2, ·�Òknd~¦éA��:(x

′i, yi)´

rK�:�ä : kÚ?�PÒ, ^Y(i), X(i)Úe(i)©OL«lY , XÚe¥GØ1i1¤��þ½Ý. GØ1i|êâ�,�e�n− 1|êâ��5£8�.�

Y(i) = X(i)β + e(i), E(e(i)) = 0, Cov(e(i)) = σ2In−1. (3.4.6)

rlù��.¦��β�LSEP�β(i), K

β(i) = (X′

(i)X(i))−1X

(i)Y(i). (3.4.7)

�þβ − β(i)�N1i|êâé£8Xê�O�K��, �§´��þ, ØBuA^©Û, A�Ä§�,«êþz¼ê. Cookål´Ù¥A^�2��«.

Äk·�5¦β − β(i)�°(L�ª. �d, Ik0��ð�ª:

�A�n× n�_Ý, uÚvþ�n× 1�þ, @ok

(A− uv′)−1 = A−1 +

A−1uv′A−1

1− v′A−1u.

Px′i��OÝX�1i1. @oX = (x1, · · · ,xn)

′. |^þãÚ

n, ��

(i)X(i))−1 = (X

′X − xix

′i)−1

= (X′X)−1 +

(X′X)−1xix

′i(X

′X)−1

1− hii,(3.4.8)

Ù¥hii = x′i(X

′X)−1xi�lfÝH�1i�é��, �¡

�m\:. e,hii��, K�¡�pm\:. qÏ�

(i)Y(i) =∑j 6=ixjyj =

n∑j=1

xjyj − xiyi = X′Y − xiyi,

β(i) = (X′

(i)X(i))−1X

(i)Y(i)

′X)−1 +

(X′X)−1xix

′i(X

′X)−1

1− hii

′Y − xiyi)

= β − (X′X)−1xiyi +

(X′X)−1xix

′iβ

1− hii

−(X′X)−1xihiiyi1− hii

= β − (X′X)−1xiyi1− hii

′X)−1xix

′iβ

1− hii

= β − (X′X)−1xiei1− hii

. (3.4.9)

β − β(i) =(X

′X)−1xiei1− hii

CookÚ?e��ål:

Di(M , c) =(β(i) − β)

′M(β(i) − β)

Ù¥M´�½��½Ý, c´�½��~ê. N´wÑ

Di(M , c) =e2i

c(1− hii)2· x′

i(X′X)−1M(X

′X)−1xi.

�M = X′X, c = (p+ 1)σ2, K

Di =e2i

(p+ 1)σ2(1− hii)2· hii =

p+ 1· hii

1− hii· r2i .

½n (3.4.2)

Cookål

Di =(β(i) − β)

′X(β(i) − β)

(p+ 1)σ2

(p+ 1)σ2(1− hii)2· hii

p+ 1· hii

1− hii· r2i , i = 1, · · · , n, (3.4.10)

Ù¥hii�lfÝH�1i�é��, ri´Æ)zí�.

ù�½nL², 3O�Cookål��ÿ, ·��I�l��êâ��5£8�.�ÑÆ)zí�ri ÚlfÝ�é��hiiÒ�±, Ø7é?Û��Ø��êâ��5£8�.(3.4.6) ?1O�.

hii�¹Â: hiiÝþ1i|êâxi�¥%x = 1n

∑ni=1 xi�ål.

�Ä�.

Y = Xβ + e, E(e) = 0, Cov(e) = σ2In,

b�gCþ®¥%z, K

1 x11 − x1 · · · x1p − xp...

......

...1 xn1 − x1 · · · xnp − xp

1 (x1 − x)′

......

1 (xn − x)′

ùpx = (x1, · · · , xp)′, xi = (xi1, · · · , xip)

′. ²{üO��

X′X =

(n 00 L

Ù¥L =∑n

i=1(xi − x)(xi − x)′, �p��. ?�Ú��

hii =1

n+ (xi − x)

′L−1(xi − x).

e´��¥%z�5£8�., K

1 x1 − x...

...1 xn − x

, X′X =

(n 00 Sxx

), hii =

(xi − x)2

3úª(3.4.10)¥, 1/(p+ 1)�iÃ', Pi = hii1−hii´hii�üNO¼

ê, Ï�hiiÝþ1i|êâxi�¥%x = 1n

∑ni=1 xi�ål. Ï

dPi�x1i|êâålÙ§êâ��C. Cookål�PiÚr2i�

��¤û½. ½n3.4.2w�·�, pm\:�U´rK�:, ��UØ´; É~:�U´rK�:, ��UØ´; �XJ�|êâQ´pm\:q´É~:, @o§Ò´rK�:.

��Cookål��^±�½rK�:��.�´é(J�.

A^�&ý¥�±éCookålí�¥�M , c�À��½�nØ|±. 3Ø��b�e,

β ∼ N(β, σ2(X′X)−1),

|^íØ2.4.1�

(β − β)′X

′X(β − β)

σ2∼ χ2(p+ 1),

,��¡(n− p− 1)σ2

σ2∼ χ2(n− p− 1),

��ö�pÕá. ¤±

(β − β)′X

′X(β − β)

(p+ 1)σ2∼ F (p+ 1, n− p− 1). (3.4.11)

S ={β :

(β − β)′X

′X(β − β)

(p+ 1)σ2≤ Fα(p+ 1, n− p− 1)

}¡�β��&Y²�1− α��&ý¥. (3.4.10)�(3.4.11)¥�ÚOþé�q, �cö¿ØÑlF©Ù. , /Ïu�ö·��±éDi��Ñ±e�VÇ)º.

~X, eDi = F0.50(p+ 1, n− p− 1), KL²1i|êâ(x′i, yi)�

GØ�, β��Oβ(i)£Ä�β��&Y²�0.5��&ý¥>.

þ; eDj = F0.80(p+ 1, n− p− 1), Kβ(j)£Ä�β��&Y²�0.2��&ý¥>.þ. ��1i|êâé�O�K�'1j|êâ5��.

éu~3.4.1, ·��Xe�êâ�ä©Û:

yx=read.table(”ex p47 data.txt”)x=yx[,1]y=yx[,2]z=sqrt(y)mydata2=data.frame(z,x)lm.sol2=lm(z∼x,data=mydata2)summary(lm.sol2)z.fit=predict(lm.sol2)r.std=rstandard(lm.sol2)plot(r.std∼z.fit)text(z.fit,r.std,type=”1:53”)r.stdinfluence.measures(lm.sol2)

©Û: 126Ò��:�É~:. ek”�cü�”�:, K�ÑÆ)zí

��?1?�Ú(@.Tianxiao Pang 1nÙ £8ëê��O

o(: 126Ò��:�É~:, ·�I�u�êâ5 ´ÄkL�, eÃL�KíØd��:. d, 126Ò��:�rK�:,IÚå5¿.

~3.4.2�åÿÁêâ. L3.4.2´��Æ[ÿÁ�21��Ö�P¹, Ù¥x´�Ö�c#(ü : �), y´,«�å�I, ÏLùêâ, ·��ïá�å�c#Cz�'X.

L3.4.2�åÿÁêâ

SÒ x y SÒ x y1 15 95 12 9 962 26 71 13 10 833 10 83 14 11 844 9 91 15 11 1025 15 102 16 10 1006 20 87 17 12 1057 18 93 18 42 578 11 100 19 17 1219 8 104 20 11 86

10 20 94 21 10 10011 7 113

R §S:

x=c(15,26,10,9,15,20,18,11,8,20,7,9,10,11,11,10,12,42,17,11,10)y=c(95,71,83,91,102,87,93,100,104,94,113,96,83,84,102,100,

105,57,121,86,100)lm.sol=lm(y∼x)summary(lm.sol)influence.measures(lm.sol)

plot(lm.sol,which=2) /*QQã*/y.fit=predict(lm.sol)e=y-y.fitr.std=rstandard(lm.sol)plot(r.std∼y.fit,xlab=expression(hat(y)),ylab=“r”)text(y.fit,r.std,type=“1:25”)

o(: 119Ò��:�É~:, 118,19Ò��:´rK�:.dQQãÚí�ã, ·��É�5b�!��à5b�!Ø�'5b�Ú��5b�.

Box-CoxC�

éu*ÿêâ, e²L£8�ä��, §�Ø÷v�5b�!��à5b�!Ø�'5b�Ú��5b�¥��½eZ�,@o·�Ò�éêâæ�”£�”��. ¢�y², êâC�´?nk¯Kêâ��«Ð�{.

�!0�Box-CoxC�, §�Ì�A:´Ú\��ëê, ÏLêâ��OTëê, l (½Aæ��êâC�/ª. ¢�y²,Box-CoxC�éNõ¢SêâÑ´1�k��.

Box-CoxC�´éÏCþ�XeC�:

y(λ) =

{yλ−1λ , λ 6= 0,

ln y, λ = 0,(3.5.1)

ùpλ´��½�C�ëê.

Box-CoxC�´�xC�, §�)Nõ~��C�, ÃXéêC�(λ = 0), �êC�(λ = −1)Ú²��C�(λ = 1/2)��.

éÏCþ�n�*ÿ�y1, · · · , ynA^þãC�, ��C��þ

Y (λ) = (y(λ)1 , · · · , y(λ)

n )′.

·��(½C�ëêλ¦�Y (λ)÷v

Y (λ) = Xβ + e, e ∼ N(0, σ2I). (3.5.2)

=�¦C��þY (λ)�£8gCþ�mäk�5�''X,Ø�÷v��©Ù!��à5!�pÕá.

Ïd, Box-CoxC�´ÏLéëêλ�ÀJ, ��é�5êâ�”nÜ£n”, ¦Ù÷v��5£8�.�¤kb�^�.

·�^4�q,�{5(½λ. é�½�λ, βÚσ2, Y (λ)�q,¼ê�

2πσ)nexp

{− 1

2σ2(Y (λ) −Xβ)

′(Y (λ) −Xβ)

¤±Y�q,¼ê�

L(β, σ2) =1

2πσ)nexp

{− 1

2σ2(Y (λ) −Xβ)

′(Y (λ) −Xβ)

}|J |,

ùpJ�C��Jacobi1�ª

n∏i=1

dy(λ)i

n∏i=1

yλ−1i .

lnL(β, σ2)'uβÚσ2¦�¿-Ù�u", ��βÚσ2�MLE�{β(λ) = (X

′X)−1X

′Y (λ),

σ2(λ) = 1nY

(λ)′(I −H)Y (λ) ∆

= 1nRSS(λ,Y (λ)).

(3.5.3)

éA�q,¼ê��

Lmax(λ) = L(β(λ), σ2(λ))

= (2πe)−n/2 · |J | ·(RSS(λ,Y (λ))

)−n/2.(3.5.4)

ù´λ�¼ê,·�ÏL¦§��5(½λ. ù�duÏL¦lnLmax(λ)��5(½λ. Äk, k

lnLmax(λ) = −n2· ln[RSS(λ,Y (λ))

]+ ln |J |+ C

= −n2

ln[Y (λ)

|J |1/n(I −H)

Y (λ)

|J |1/n]

∆= −n

2ln[RSS(λ,Z(λ))

]+ C, (3.5.5)

Ù¥Z(λ) = (z(λ)1 , · · · , z(λ)

n )′

= Y (λ)/|J |1/n,

RSS(λ,Z(λ)) = Z(λ)′(I −H)Z(λ). (3.5.6)

(3.5.5)ªéBox-CoxC�3O�Åþ�¢y�5é��B, Ï��¦lnLmax(λ)��, �I¦RSS(λ,Z(λ))��. �,éJé�¦RSS(λ,Z(λ))��λ�)ÛL�ª, �é�X��½�λ�, ÏL¦��¦�O�£8§S, N´O�ÑéA�RSS(λ,Z(λ)). ,�xÑRSS(λ,Z(λ))'uλ�ã, lãþ�±Cq�éÑ¦RSS(λ,Z(λ)) ��λ.

Box-CoxC��äNÚ½:

1. é�½�λ�,O�z(λ)i , i = 1 · · · , n.

2. U(3.5.6)ªO�í�²�ÚRSS(λ,Z(λ)).

3. é�X��½�λ�, EþãÚ½, ��A�í�²�ÚRSS(λ,Z(λ))��G�. ±λ�î¶, RSS(λ,Z(λ)) �p¶, xÑ�A��. ^�*�{, éÑ¦RSS(λ,Z(λ))��λ.

~3.5.13~3.4.1¥, ·�éÏCþy�²��C�, ù��u¦^λ = 0.5�Box-CoxC�. ·�y35y¢ù��C�´Ü·�.eL�Ñ12�ØÓ�λ�éA�í�²�ÚRSS(λ,Z(λ)), {ü'��uy�λ = 0.5�í�²�ÚRSS(λ,Z(λ))��. ÏdCq/@�0.5Ò´C�ëêλ��`ÀJ.

L3.5.1

λ -2 -1 -0.5 0 0.125 0.25RSS 34101.04 986.04 291.59 134.10 119.20 107.21λ 0.375 0.5 0.625 0.75 1 2

RSS 100.26 96.95 97.29 101.69 127.87 1275.56

x<-c(679,292,1012,493,582,1156,997,2189,1097,2078,1818,1700,747,2030,1643,414,354,1276,745,435,540,874,1543,1029,710,1434,837,1748,1381,1428,1255,1777,370,2316,1130,463,770,724,808,790,783,406,1242,658,1746,468,1114,413,1787,3560,1495,2221,1526)y<-c(0.79,0.44,0.56,0.79,2.70,3.64,4.73,9.50,5.34,6.85,5.84,5.21,3.25,4.43,3.16,0.50,0.17,1.88,0.77,1.39,0.56,1.56,5.28,0.64,4.00,0.31,4.20,4.88,3.48,7.58,2.63,4.99,0.59,8.19,4.79,0.51,1.74,4.10,3.94,0.96,3.29,0.44,3.24,2.14,5.71,0.64,1.90,0.51,8.33,14.94,5.11,3.85,3.93)lm.sol<-lm(y∼x)library(MASS)boxcox(lm.sol,plotit=T,lambda=seq(-2,2,by=0.05))

2Â��¦�O

3c¡�?Ø¥, ·�o´b��5£8�.�Ø�´��à5�Ø�'�, =Cov(e) = σ2In.�´3Nõ¢S¯K¥, êâ Ø÷vù�b�(�ÏLí�ã�ä). ù�·�Ib�Ø��þ��Ý�Cov(e) = σ2Σ, ùpΣ´��½Ý. ù�Σ�¹��ëê, �·�ùpb�Σ ´��®��.

·��?Ø��5£8�.�

Y = Xβ + e, E(e) = 0, Cov(e) = σ2Σ. (3.6.1)

Ì�8�´�Oβ.

Ï�Σ´é¡�½Ý, ¤±�3n× n��P¦�

Σ = PΛP′,

ùpΛ = diag(λ1, · · · , λn), λi > 0, i = 1, · · · , n, ´Σ�A��.P

Σ12 = P diag(λ

121 , · · · , λ

12n )P

K��(Σ12 )2 = Σ, ¡Σ

12´Σ�²��. Σ−

12L«Σ

12�_Ý

·�r�5£8�.(3.6.1)?1��C�. ^Σ−12�¦(3.6.1). P

Z = Σ−12Y , U = Σ−

12X, ε = Σ−

Ï�Cov(ε) = Σ−12σ2ΣΣ−

12 = σ2In, u´·��Xe��5£

Z = Uβ + ε, E(ε) = 0, Cov(ε) = σ2In. (3.6.2)

ù´·�®?ØL��.. 3ù#�.¥, ��β�LSE�

β∗ = (U′U)−1U

′Z = (X

′Σ−1X)−1X

′Σ−1Y . (3.6.3)

·�¡β∗�β�2Â��¦�O(GLSE). ù��OäkûÐ�ÚO5�.

½n (3.6.1)

(a) E(β∗) = β;(b) Cov(β∗) = σ2(X

′Σ−1X)−1;

(c)é?¿�p+ 1��þc, c′β∗�c

′β��5Ã

y²: (a)

E(β∗) = (X′Σ−1X)−1X

′Σ−1E(Y ) = (X

′Σ−1X)−1X

′Σ−1Xβ = β.

(b)|^½n2.1.3,

Cov(β∗) = Cov[(X′Σ−1X)−1X

′Σ−1Y ]

= (X′Σ−1X)−1X

′Σ−1Cov(Y )((X

′Σ−1X)−1X

′Σ−1)

= σ2(X′Σ−1X)−1X

′Σ−1Σ((X

′Σ−1X)−1X

′Σ−1)

= σ2(X′Σ−1X)−1.

(c)�b′Y´c

′β�?��5Ã �O. éu�.(3.6.2),

c′β∗ = c

′U)−1U

′Z, b

′Y = b

12 Σ−

12Y = b

=c′β∗�c

′β�LSE, b

′Y = b

12Z�c

′β��5Ã �O. ¤±

é�.(3.6.2)A^Gauss-Markov½n�

Var(c′β∗) ≤ Var(b

12Z) = Var(b

′Y ),

�Ò¤á��=�c′β∗ = b

12Z = b

′Y .

½n3.6.1(c)Ò´��/e�Gauss-Markov½n, §L²3��5£8�.(3.6.1)¥, GLSE β∗´�`�(eΣ = In, KGLSEòz�LSE β). éu�.(3.6.1), N´y²βE´Ã �O, ��7´�`��5Ã �O, Ï�Var(c

′β∗) ≤ Var(c

′β). ùÒ´`,

éu��5£8�.(3.6.1), GLSEo´`uLSE.

�.(3.6.1)�{ü�~f´ÏCþ�ØÓ*ÿäkØ��/, =

Cov(e) = diag(σ21, · · · , σ2

ùp�σ2i , i = 1, · · · , nØ��. Px

′1, · · · ,x

′n©O´�OÝ

X�n�1�þ. N´íÑ,

β∗ =( n∑i=1

xix′i

)−1( n∑i=1

xiyiσ2i

). (3.6.4)

ü�Úª©O´xix′iÚxiyi�\�Ú(�Ñ�1/σ2

i ), Ïd�¡β∗�\��¦�O(WLSE).

σ2i ´��, ù�·�I��{¦�§��Oσ2

i , ,�3(3.6.4)¥^σ2

i�Oσ2i . ù«�O�{¡�üÚ�O(two-stage

estimate).

~3.6.1b�·�^�«°�¤ì3ü�¢�¿éÓ��þµ©O?1n1gÚn2gÿþ, Pùÿþ�©O�y11, · · · , y1n1Úy21, · · · , y2n2 . r§��¤�5£8�./ª{

y1i = µ+ e1i, i = 1, · · · , n1,y2i = µ+ e2i, i = 1, · · · , n2.

duü�¢�¿��*^�9¤ì�°ÝØÓ, �§��ÿþØ��Ø�. �

Var(e1i) = σ21, Var(e2i) = σ2

2, σ21 6= σ2

Pe = (e11, · · · , e1n1 , e21, · · · , e2n2)′, K

Cov(e) =

1In1 00 σ2

)= σ2

(θIn1 0

)∆= σ2

ùpθ = σ21/σ

22. b�θ®�, KΣ®�.

5¿�ùp��OÝX = (1, · · · , 1)′, u´µ�GLSE�

µ∗ =(n1

θ+ n2

)−1(1

n1∑i=1

n2∑i=1

n1∑i=1

y1i, y2 =1

n2∑i=1

ω1 =1

Var(y1)=n1

, ω2 =1

Var(y2)=n2

Kµ∗�U��

µ∗ =ω1

ω1 + ω2y1 +

ω1 + ω2y2.

=µ∗´ü�¢�¿*ÿ�þ��\�²þ, §�� ω1ω1+ω2

ω1+ω2��¢�¿ÿþ�Ø��Úÿþgêk'. Ø��

�, ÿþgê��, éA��Ò�.

µ∗�¹��ëêσ21Úσ

22, ØUGÃ¢SA^. ·��±�{�

Eσ21Úσ

22��O. ¯¢þ, ùü�¢�¿�*ÿêâ©O�¤�

5£8�.Yi = µ1ni + ei, i = 1, 2,

ùpYi = (y1i, · · · , y1ni)′, ei = (ei1, · · · , eini)

′. Ï�Cov(ei) =

σ2i Ini , ¤±ei, i = 1, 2÷vGauss-Markovb�. ¤±σ2

i�LSE�

σ2i =

ni − 1‖Yi − yi1ni‖2 i = 1, 2.

^σ2i , i = 1, 2�Oµ∗¥�σ2

i , i = 1, 2, =��µ�üÚ�O.

õ��5

£8Xê�LSEkNõ`û�5�, Ù¥��´Gauss-Markov½n, §L²3�5Ã �Oa¥, LSE´��äk��O. �´ù�`:, ¦�LSE3�5ÚO�.��OnØÚ¢SA^¥Ókýé��/ .

�´·�±c?Ø�LSEI�b��OÝX´�÷��, =�¦ÝX��þ�m´�5Ã'�. , , 3¢S¯K¥, du²~�?n¹k�õgCþ��.£8¯K, �²LCþ�m Ø´�á� ´�péX�, ùÑ�¦�OÝX��þ�mØ�U��5Ã'. éõ�¹e, �OÝX��þ�m�3õ��5/E��5(multi-collinearity)'X.

e�3Ø��0�p+ 1�~êc0, c1, · · · , cp¦�

c0 + c1xi1 + · · ·+ cpxip = 0, i = 1, · · · , n,

K¡gCþx1, · · · , xp�m�3X��õ��5/E��5'X.

3¢S¯K¥, ��õ��5/E��5'X¿Øõ�, ��Ñy�´�½§Ýþ��5.

½Â (õ��5/E��5'X)

e�3Ø��0�p+ 1�~êc0, c1, · · · , cp¦�

c0 + c1xi1 + · · ·+ cpxip ≈ 0, i = 1, · · · , n,

K¡gCþx1, · · · , xp�m�3Xõ��5/E��5'X.

é²Lêâï��, õ��5��/éõ, õ��5�/¬�õ��5£8©Û�5�oK�!XÛ�ägCþ�m�õ��5±9XÛ�Ñõ��5�K��¯Kò´�!�?Ø�Ì�SN.

·�kÚ\��Vg: þ�Ø�(MSE: Mean Squared Errors), §´^5µd��O`��IO�.

�θ��þ. θ�θ��O. ½Âθ�þ�Ø��

MSE(θ) = E‖θ − θ‖2 = E[(θ − θ)′(θ − θ)].

½n (3.7.1)

MSE(θ) = tr[Cov(θ)] + ‖Eθ − θ‖2.

y²: ØJwÑ

MSE(θ) = E[(θ − θ)′(θ − θ)]

= E[(θ − Eθ) + (Eθ − θ)]′[(θ − Eθ) + (Eθ − θ)]

= E(θ − Eθ)′(θ − Eθ) + E(Eθ − θ)

′(Eθ − θ)

∆= ∆1 + ∆2.

|^,�5�,

∆1 = E{tr[(θ − Eθ)′(θ − Eθ)]}

= E{tr[(θ − Eθ)(θ − Eθ)′]}

= tr[E(θ − Eθ)(θ − Eθ)′] = tr[Cov(θ)].

∆2 = E[(Eθ − θ)′(Eθ − θ)] = ‖Eθ − θ‖2´w,�. y..

ePθ = (θ1, · · · , θp+1)′, K

∆1 =

p+1∑i=1

Var(θi),

§´θ�©þ��Ú.

∆2 =

p+1∑i=1

(Eθi − θi)2,

3�5£8�.(3.1.5)¥, éβ�LSE β, k(a) MSE(β) = σ2

∑p+1i=1

(b) E‖β‖2 = ‖β‖2 + σ2∑p+1

i=11λi,

Ù¥λ1, · · · , λp+1 > 0�X′X�A��.

y²: (a)Ï�LSE β´Ã �O, ¤±∆2 = 0,

MSE(β) = ∆1 = tr[Cov(β)] = σ2tr[(X′X)−1].

Ï�X′X´é¡�½Ý, ¤±�3��P¦�

X′X = P diag(λ1, · · · , λp+1)P

ùpλ1, · · · , λp+1 > 0�X′X�A��. ¤±

(X′X)−1 = P diag(

λ1, · · · , 1

λp+1)P

|^,�5�êþ��

tr(X′X)−1 = tr

(diag(

λ1, · · · , 1

λp+1))

p+1∑i=1

¤±MSE(β) = σ2∑p+1

i=11λi

(b) Ï�

MSE(β) = E[(β − β)′(β − β)]

= E(β′β − 2β

′β + β

′β)

= E‖β‖2 − β′β,

E‖β‖2 = ‖β‖2 + MSE(β) = ‖β‖2 + σ2p+1∑i=1

(Ø(a)w�·�, XJX′X��k��A��~�, =�~�

Cu", @oMSE(β)Ò¬é�. lþ�Ø��IO5w, ��¦�OβØ´��Ð��O. ùÚGauss-Markov½n¿Øgñ, Ï�Gauss-Markov½n==�y��¦�O3�5Ã �Oa¥��5. �3X

′X��k��A��~��, ù��

��%é�, Ï ��é��þ�Ø�.

(Ø(b)w�·�, XJX′X��k��A��~�, @o��

o�, �X′X��k��A��~��, ��¦�OØ2´

��Ð��O.

PX = (1,x1, · · · ,xp), =xi�X�1i+ 1�. �λ�X′X��

A��, φ�ÙéA�A��þ, Ù�Ý�1, =φ′φ = 1.

eλ ≈ 0, K

‖Xφ‖2 = φ′X

′Xφ = λφ

′φ = λ ≈ 0.

u´Xφ ≈ 0. Pφ = (c0, c1, · · · , cp)′, K

c0 + c1x1 + · · ·+ cpxp ≈ 0. (3.7.1)

=�OÝX��þ�m(=gCþ�m)kõ��5'X.

��, e�OÝX��þ�mkõ��5'X, =(3.7.1)¤á, d�X

′XE´�½Ý�|X ′

X| ≈ 0. dd��

p+1∏i=1

λi = |X ′X| ≈ 0,

¤±��k��A��~�, �Cu".

�Ò´`��k��A��~��X��þ�mkõ��5'X´�d�. ù�¡�OÝX´¾�Ý.

õ��5��ä :

(1) ��)äÏf(VIF: variance inflation factor)�ä{

PR2j�gCþxjéÙ{p− 1�gCþ��½Xê, ½Â

VIFj =1

1−R2j

, j = 1, · · · , p.

duR2jÝþgCþxjéÙ{p− 1�gCþ�m��5�'§

Ý, x1, · · · , xp�m�õ��5�î, R2j��Cu1, VIFj�Ò

��. Ïd^VIF 5Ýþõ��5�§Ý´Ün�.

Ýþ�OK:

�k,�VIFj ≥ 10

�, @�gCþ�m�3î�õ��5; ½ö�

VIF =1

p∑i=1

VIFj � 1

�, @�gCþ�m�3î�õ��5.

(2)A��^�ê(CI: Condition Index)�ä{

��Øþj�K�, ·�b�gCþ�ÏCþ�*ÿ�þ®IOz. d��@��5£8�.vk�å�, �OÝX´n× p�Ý, X

′X´p�gCþ��'XêÝ.

A��ä{: XJX′Xkm�A��Cq�", @oXÒkm

�õ��5'X, ¿�ùm�õ��5'X�Xê�þÒ´ùm��Cu"�A��¤éA�IO��zA��þ.

�A��λmax, ��A��λmin, ¡

κj =λmaxλj

, j = 1, · · · , p

�A��λj�^�ê.

κ = maxjκj =

λmaxλmin

§�±^5ÝþÝX′X�A��Cq�"�§Ý, Ïd�±^

5�äõ��5´Ä�3±9õ��5�î§Ý.

^�ê�½OK:

e0 < κ < 100, K@�Ø�3õ��5;

e100 ≤ κ ≤ 1000, K@��3�r�õ��5;

eκ > 1000, K@��3î�õ��5.

L3.7.1

SÒ y x1 x2 x3 x4 x5 x61 10.006 8 1 1 1 0.541 -0.0992 9.737 8 1 1 0 0.130 0.0703 15.087 8 1 1 0 2.116 0.1154 8.422 0 0 9 1 -2.397 0.2525 8.625 0 0 9 1 -0.046 0.0176 16.289 0 0 9 1 0.365 1.5047 5.958 2 7 0 1 1.996 -0.8658 9.313 2 7 0 1 0.228 -0.0559 12.960 2 7 0 1 1.380 0.502

10 5.541 0 0 0 10 -0.798 -0.39911 8.756 0 0 0 10 0.257 0.10112 10.937 0 0 0 10 0.440 0.432

yx=read.table(”ex p62 data.txt”)y=yx[, 1]x1=yx[, 2]x2=yx[, 3]x3=yx[, 4]x4=yx[, 5]x5=yx[, 6]x6=yx[, 7]mydata=data.frame(y,x1,x2,x3,x4,x5,x6)lm.sol=lm(y∼ x1+x2+x3+x4+x5+x6,data=mydata)summary(lm.sol)library(DAAG)vif(lm.sol)

VIFÑÑ(J

x1 x2 x3 x4 x5 x6

182.0500 161.3600 266.2600 297.7100 1.9200 1.4553

Ï�co�VIPÑØ�u10, ¤±@�gCþ�m�3î�õ��5.

A��^�ê�ä{:

X=cbind(x1,x2,x3,x4,x5,x6)Xrho=cor(X)rhoeigen(rho)kappa(rho,exact=TRUE) /*%@´exact=FALSE, ù�k��O�Ø�*/

A��!A��þ�^�ê:

λmin = 0.001106051, éA�A��þ�

(0.44768, 0.42114, 0.541689, 0.57337, 0.00605, 0.00217)′.

¤±IOzgCþx∗i , i = 1, · · · , 6�m�3Xe�õ��5'X

0.44768x∗1 + 0.42114x∗2 + 0.541689x∗3 + 0.57337x∗4 + 0.00605x∗5

+0.00217x∗6 ≈ 0.

c0 + 0.44768x1 + 0.42114x2 + 0.541689x3 + 0.57337x4

+0.00605x5 + 0.00217x6 ≈ 0.

½��

c0 + 0.44768x1 + 0.42114x2 + 0.541689x3 + 0.57337x4 ≈ 0.

^�êκ = 2195.908 > 1000, �`²gCþ�m�3î�õ��5'X.

�Øõ��5��{:

O��Nþ, �Ø½�)gCþ��5�'5.

�gCþ�mäkõ��5�, ��ÑLSE²wC��¯K,Hoerlu1962cJÑ�«U?��¦�O�{, =*�O(Ridge Estimate). ��, HoerlÚKennard31970céT�O�?�Ú��[?Ø.

*�O�g�: �gCþ�m�3õ��5�, �OÝX´¾��, =|X ′

X| ≈ 0, l (X′X)−1�CÛÉ. �;�ù�y�,

�X′X \þ��~êé�ÝkI(k > 0),KÝ

(X′X + kI)−1

�CÛÉ��U5�'(X′X)−1�CÛÉ��U5��õ. Ïd

β(k) = (X′X + kI)−1X

′Y (3.8.1)

��ëêβ��OAT'��¦�O�½�.

é�½�0 < k <∞, ¡β(k) = (X′X + kI)−1X

′Y�£8X

¥βj(k)�kCz¤LyÑ5��¡�*,(ridge trace).

5: kØÓ, ·��ØÓ��O. Ïd*�Oβ(k)´��Oa.�k = 0�, β(k)Ò´Ï~�LSE. ��¹e, ·�Jå*�O,´Ø�)LSE�.

3?1*�O�c, ��Øþj�K�, ·�ob�gCþ�ÏCþþ®IOz, Ïdùp��OÝX´n× pÝ.

*�O�5� :

5�1: β(k)´β�k �O, =é?¿�0 < k <∞, E(β(k)) 6=β.

k 5´*�O��¦�O��ØÓ�?. ��O�þ�Ø�d��ÚÚ ��²�Ú|¤. ��3õ��5�, ��¦�O�,�± �Ü©�", �§��Ü©%é�, �ª��§�þ�Ø�é�. ·�Ú?*�O�8�´ã+Ã 5, ��Ü©��ÌÝ~�, �ªü$Ùþ�Ø�.

5�2: β(k)´��¦�Oβ��5C�.

y²: �I5¿�

β(k) = (X′X + kI)−1X

= (X′X + kI)−1X

′X · (X ′

X)−1X′Y

= (X′X + kI)−1X

′Xβ.

5�3: é?¿�k > 0, e‖β‖ 6= 0, K·�ok‖β(k)‖ < ‖β‖.=*�O´r��¦�Oβ��:�·Ý�Ø ��, *�O´��Ø k �O.

y²: �Äõ��5£8�.Y = Xβ + e, -

Z = XP , α = P′β,

Ù¥P��Ý÷v

P′(X

′X)P = Λ = diag(λ1, · · · , λp),

λ1, λ2, · · · , λp > 0�X′X�A��. ù�, õ��5£8�.�

��

Y = Zα+ e, E(e) = 0, Cov(e) = σ2In. (3.8.2)

·�¡(3.8.2)��5£8�.�;K/ª, ¡α�;K£8Xê.

5¿�Z′Z = P

′XP = Λ, ¤±

α = (Z′Z)−1Z

′Y = Λ−1Z

′Y .

β = (X′X)−1X

′Y = PΛ−1P

′Y = PΛ−1Z

′Y = Pα.

§��A�*�O©O�

α(k) = (Z′Z + kI)−1Z

′Y = (Λ + kI)−1Z

′Y ,

β(k) = (X′X + kI)−1X

= PP′(X

′X + kI)−1PP

= Pα(k).

‖β(k)‖ = ‖α(k)‖ = ‖(Λ + kI)−1Λα‖ < ‖α‖ = ‖β‖.

5: ´�þ�Ø�3�OÚëê��C�e�±ØC, ¤±;K£8Xê��¦�O(½*�O)Ú�£8Xê��¦�O(½*�O)k�Ó�þ�Ø�:

MSE(α) = MSE(β), MSE(α(k)) = MSE(β(k)). (3.8.3)

½n (3.8.1, *�O�35½n)

�3k > 0¦�MSE(β(k)) < MSE(β).

=�3k > 0, ¦�3þ�Ø�¿Âe, *�O`u��¦�O.

y²: d(3.8.3), �Iy²�3�3k > 0¦�

MSE(α(k)) < MSE(α). (3.8.4)

Pf(k) = MSE(α(k)), k ≥ 0. 5¿f(0) = MSE(α). e·�Uy²f(k)3[0,∞)þ´ëY¼ê�f

′(0) < 0, K7�3��

k > 0 ¦�(3.8.4)¤á.

5?Øf(k). 5¿�

E(α(k)) = (Λ + kI)−1Z′E(Y )

= (Λ + kI)−1Z′Zα

= (Λ + kI)−1Λα,

Cov(α(k)) = σ2(Λ + kI)−1Z′Z(Λ + kI)−1

= σ2(Λ + kI)−1Λ(Λ + kI)−1.

f(k) = MSE(α(k)) = tr[Cov(α(k))

]+ ‖E(α(k))−α‖2

= σ2p∑i=1

λi(λi + k)2

+ k2p∑i=1

(λi + k)2

∆= f1(k) + f2(k). (3.8.5)

w,f(k)´[0,∞)þ�ëY¼ê. q

f′1(k) = −2σ2

p∑i=1

λi(λi + k)3

, f′1(0) = −2σ2

p∑i=1

< 0 (3.8.6)

f′2(k) = 2k

p∑i=1

λiα2i

(λi + k)3, f

′2(0) = 0, (3.8.7)

¤±f′(0) = f

′1(0) + f

′2(0) = −2σ2

∑pi=1

1λ2i< 0. y..

*�O�35½n3nØþy²�3,�*�O`u��¦�O, ��éÑù�*ëêk´ØN´�. N´wÑ, nØþ, ù�k´e��§�):

f′(k) = −2σ2

p∑i=1

λi(λi + k)3

p∑i=1

λiα2i

(λi + k)3

p∑i=1

λi(kα2i − σ2)

(λi + k)3= 0. (3.8.8)

ù�)�6u��ëêαi, i = 1, · · · , pÚσ2, ¤±Ø�Ul)�§��Ý¼�*ëêk. ÚOÆ[�lÙ§å»JÑÀJ*ëêk��{.

*ëêÀJ�{ :

(1) Hoerl-Kennardúª

HoerlÚKennardJÑ�ÀJk�úª´

k =σ2

maxiα2i

. (3.8.9)

¼�ù�úª��{Xe: d(3.8.8)�, ekα2i − σ2 < 0éi = 1,

· · · , pÑ¤á, Kf′(k) < 0. u´�

k∗ =σ2

maxiα2i

�0 < k < k∗�, f′(k)o´�u", Ï f(k)3(0, k∗)þ´üN4

~¼ê, 2df(k)3[0,∞)�ëY5�f(k∗) < f(0). ,�2^αiÚσ2 �Oαi Úσ

2, B�(3.8.9).

(2) *,{

òβ1(k), · · · , βp(k)�*,x3�Üãþ, �â*,�Czª³ÀJk. ±e´A^ÀJk�OK:

�£8Xê�*�O��'�½;

^��¦�O�ÎÒØÜn�£8Xê, Ù*�O�ÎÒòC�Ün;

£8XêvkØÜn�ÎÒ;

í�²�ÚØ�þ,�õ.

~3.8.1{I²Ló�öF"ÏLISo��x1,�;þx2,o�¤þx3�ýÿ?�o�y, ±þCþ�ü þ��·{J. �dÂ81949-1959�11c�êâ, �eL.

c° x1 x2 x3 y1949 149.3 4.2 108.1 15.91950 161.2 4.1 114.8 16.41951 171.5 3.1 123.2 19.01952 175.5 3.1 126.9 19.11953 180.8 1.1 132.1 18.81954 190.7 2.2 137.7 20.41955 202.1 2.1 146.0 22.71956 212.4 5.6 154.1 26.51957 226.1 5.0 162.3 28.11958 231.9 5.1 164.3 27.61959 239.0 0.7 167.6 26.3

library(MASS)yx=read.table(”ex p68 data.txt”)x1=yx[, 1]x2=yx[, 2]x3=yx[, 3]y=yx[, 4]mean(x1);mean(x2);mean(x3);mean(y)sd(x1);sd(x2);sd(x3);sd(y)mydata=data.frame(y, x1, x2, x3)lm.sol=lm(y∼x1+x2+x3)summary(lm.sol)

¼��Cþ��þ�!��IO�êâ:

¦^��¦�{�(J:

£8�§: y = −10.128− 0.051x1 + 0.587x2 + 0.287x3.

O\�, ?�o�y�AO\. ¤±, £8Xê�ÎÒ�¢SØÎ.Ù�Ï´n�gCþ�m�3Xõ��5, ù�{ü/lx1,x2, x3��'XêÝwÑ:

Ïdx1�x3�3pÝ��'5.

�õ��5�ä, �±²(uygCþ�m�3Xõ��5:

^*�O�{Ïé*£8�§:

kéêâ?1IOz, R§S:

yxs=scale(yx)x1=yxs[, 1]x2=yxs[, 2]x3=yxs[, 3]y=yxs[, 4]mydata2=data.frame(x1,x2,x3,y)mydata2

rr.sol=lm.ridge(y∼0+x1+x2+x3,data=mydata2,lambda=c(seq(0,0.01,by=0.001),seq(0.02,0.1,by=0.01),seq(0.2,1,by=0.1)))rr.sol /∗w«*�O∗/plot(rr.sol) /∗�*,ã∗/matplot(rr.sol$lambda,t(rr.sol$coef),type=”l”,col=c(”red”,”blue”,

”black”),main=”ridge trace”,xlab=expression(lambda),ylab=expression(hat(beta)(lambda))) /∗�*,ã∗/

*�Oêâ:

�â*,ãÀJ*ëêk = 0.4, IOzCþ�*£8�§�

v = 0.416u1 + 0.213u2 + 0.531u3.

y − 21.891

4.544= 0.416× x1 − 194.591

30.000+ 0.213× x2 − 3.300

+0.531× x3 − 139.736

20.634,

=y = −8.655 + 0.063x1 + 0.587x2 + 0.117x3.

rr.sol=lm.ridge(y∼x1+x2+x3,data=mydata,lambda=c(seq(0,0.01,by=0.001),seq(0.02,0.1,by=0.01),seq(0.2,1,by=0.1)))rr.solplot(rr.sol)

ÀJ*ëêk = 0.4, �*£8�§:

y = −8.621 + 0.063x1 + 0.588x2 + 0.117x3.Tianxiao Pang 1nÙ £8ëê��O

2Â*�O :

PK = diag(k1, · · · , kp), ki ≥ 0, i = 1, · · · , p. ¡

β(k) = (X′X +K)−1X

�β�2Â*�O.

Ì¤©�O

Ì¤©(Principle Component)�O´dW.F.Massyu1965cJÑ�,�«k �O, 8�´��Ñ�OÝX�¾�Ý��¦�O�½5òC�é�ù�"�. Ì¤©�O�Ä�g�´: Äk/Ïu��C�ò£8gCþC�éA�Ì¤©(Ì¤©�*ÿ�þ´pØ�'�, l �Øõ��5¯K), ,�l¤k�Ì¤©¥À��Ü©��Ì¤©(å�ü��^)¿±§��#�£8gCþïá#�£8�., ^��¦�{�O#�.¥�£8Xê¿��£8�§. Äu��£8�§2ò§�=��©Cþ�£8�§.

��Øþj�K�, b�gCþ�ÏCþþ®IOz.

�Ä£8�.:

Y = Xβ + e, E(e) = 0, Cov(e) = σ2In, (3.9.1)

Ù¥X´n× p�OÝ. Pλ1 ≥ · · · ≥ λp > 0�X′X�A��,

φ1, · · · ,φp�éA�IO��zA��þ. K

Φ = (φ1, · · · ,φp)

�p× p��Ý�

Φ′X

′XΦ = diag(λ1, · · · , λp)

∆= Λ.

2PZ = XΦ, α = Φ′β, K�.(3.9.1)�U��

Y = Zα+ e, E(e) = 0, Cov(e) = σ2In. (3.9.2)

3þã��5£8;K�.(3.9.2)¥, #��OÝ

Z∆= (z1, · · · , zp) = (Xφ1, · · · ,Xφp),

zj = φ1jx1 + · · ·+ φpjxp, j = 1, · · · , p.

∑ni=1(zij − zj)2 = λj .

y²: Ï�Z′Z = Φ

′XΦ = Λ = diag(λ1, · · · , λp), ¤±

z′jzk = 0, ∀j 6= k

�z′jzj = λj , j = 1, · · · , p. qÏ�X´I5z�OÝ, ¤±

n∑i=1

zij =1

n∑i=1

p∑k=1

φkjxik =1

p∑k=1

n∑i=1

xik = 0.

n∑i=1

(zij − zj)2 =

n∑i=1

z2ij = z

′jzj = λj , j = 1, · · · , p.

λjÝþ1j�Ì¤©zj��CÄ��. Ï�λ1 ≥ · · · ≥ λp > 0,¤±·�¡z1�1�Ì¤©, z2�1�Ì¤©, · · · . ùp�Ì¤©�*ÿ�þ´pØ�'�, ¤±#�gCþz1, · · · , zp(½�OÝZ)Ø�3õ��5¯K.

d5�1��, z1 éÏCþ�)ºUå�r, z2 g�, · · · , zp�f.e�OÝX´¾�Ý, @ok�X

′X�A��é�, Ø�

b�λr+1, · · · , λp ≈ 0.

éΛ,α,Z,Φ�©¬:

(Λ1 00 Λ2

), α =

), Z = (Z1

...Z2), Φ = (Φ1...Φ2),

Ù¥Λ1�r × rÝ, α1�r × 1�þ, Z1�n× rÝ, Φ1�p× rÝ. GØZ2α2, �.(3.9.2)C�

Y ≈ Z1α1 + e, E(e) = 0, Cov(e) = σ2In. (3.9.3)

Z1Ø´¾�Ý, ¤±��A^��¦�O�α1�LSE

α1 = (Z1′Z1)−1Z1

′Y .

β = Φ

)= (Φ1,Φ2)

)= Φ1Λ

−11 Z

= Φ1Λ−11 Φ

′Y .

Ì¤©�O�L§ :

Ì¤©�O�5� :

y²: �âe�'X:

Z = (Z1...Z2) = (XΦ1

...XΦ2), Φ′1Φ1 = Ir, Φ

′1Φ2 = 0

′X = ΦΛΦ

′= Φ1Λ1Φ

′1 + Φ2Λ2Φ

¿5¿�X′(I −H) = 0(H�lfÝ), ��

β = Φ1Λ−11 Φ

′Y = Φ1Λ

−11 Φ

′Xβ

= Φ1Λ−11 Φ

′1Φ1Λ1Φ

′1β + Φ1Λ

−11 Φ

′1Φ2Λ2Φ

′2β

= Φ1Λ−11 Φ

′1Φ1Λ1Φ

′1β

= Φ1Φ′1β.

y²: �I5¿�E(β) = β=�.

y²: -I = diag(Ir,0), KdΦ�½Â�

Φ1Φ′1 = ΦIΦ

‖β‖ = ‖ΦIΦ′β‖ = ‖IΦ

′β‖ ≤ ‖Φ′

β‖ = ‖β‖.

½n (3.9.1)

MSE(β) < MSE(β).

y²: b�X′X��p− r�A��λr+1, · · · , λpé�Cu". Ø

MSE(β) = MSE

)]+∥∥∥E

)−α

∥∥∥2

= σ2tr(Λ−11 ) + ‖α2‖2.

MSE(β) = σ2tr(Λ−1) = σ2tr(Λ−11 ) + σ2tr(Λ−1

¤±MSE(β) = MSE(β) + (‖α2‖2 − σ2tr(Λ−1

u´MSE(β) < MSE(β)

��=�

‖α2‖2 < σ2tr(Λ−12 ) = σ2

p∑i=r+1

λi. (3.9.4)

Ï�α2 = Φ′2β, (3.9.4)��(β

Φ2Φ2′(βσ

)< tr(Λ−1

2 ), (3.9.5)

(a)é�½�ëêβÚσ2, �X′X��p− r�A��'��, Ì

'��¦�Ok��þ�Ø�.

Ì¤©�êr�À� :

(2)ÀJr¦�cr�A��Ú3p�A��oÚ¥¤Ó�'~(¡�\O�zÇ)��ýk�½��. �X, ÀJ��r¦�∑r

i=1 λi∑pi=1 λi

> 0.85.

k?1~1��¦�O�õ��5�ä:

yx=read.table(”ex p68 data.txt”)x1=yx[, 1]x2=yx[, 2]x3=yx[, 3]y=yx[, 4]economy=data.frame(x1,x2,x3,y)economylm.sol=lm(y∼x1+x2+x3,data=economy)summary(lm.sol)

X=cbind(x1,x2,x3)rho=cor(X)rholibrary(DAAG)vif(lm.sol)eigen(rho)kappa(rho,exact=TRUE)

economy.pr=princomp(∼x1+x2+x3,data=economy,cor=TRUE)summary(economy.pr,loadings=TRUE)

1n�A��λ3 = 0.05187378392 = 0.00269 ≈ 0.

éA�n�IO��zA��þ�

φ1 = (−0.706, 0,−0.707)′,

φ2 = (0,−0.999, 0)′,

φ3 = (0.707, 0,−0.707)′.

n�Ì¤©©O�

z1 = −0.706x1 − 0.707x3,

z2 = −0.999x2,

z3 = 0.707x1 − 0.707x3.

pre=predict(economy.pr)pre

?1Ì¤©�O:

z1=pre[, 1];z2=pre[, 2]yxs=scale(yx)y=yxs[, 4]mydata=data.frame(y,z1,z2)pc.sol=lm(y∼0+z1+z2,data=mydata)summary(pc.sol)

�Ì¤©£8�§:

u = −0.65787z1 − 0.1824z2

= −0.65787× (−0.706x∗1 − 0.707x∗3)− 0.1824× (−0.999x∗2)

= 0.46446x∗1 + 0.18222x∗2 + 0.46511x∗3.

y − 21.891

4.544= 0.46446× x1 − 194.591

30.000+ 0.18222× x2 − 3.300

+0.46511× x3 − 139.736

20.634,

=y = −7.768 + 0.070x1 + 0.502x2 + 0.102x3.

�{ ~ê� x1 x2 x3

��¦�O -10.128 -0.051 0.587 0.287

*�O(k = 0.04) -8.655 0.063 0.587 0.117

SteinØ �O

½Â (Stein�O)

3�5£8�.(3.1.5)¥, Ù£8Xêβ��¦�O�β =(X

′X)−1X

′Y . ·�¡

βs(c) = cβ

�Stein�O, Ù¥0 ≤ c ≤ 1�¡�Ø Xê.

SteinØ �O�5� :

5�1.�c 6= 1�, βs(c)´β�k !Ø �O.

5�2.�30 < c < 1, ¦�MSE(βs(c)) < MSE(β).

y²: βs(c)�þ�Ø��

MSE(βs(c)) = tr[Cov(βs(c))] + ‖E(βs(c))− β‖2

= c2σ2tr[(X′X)−1] + (c− 1)2‖β‖2

= c2σ2p∑i=1

λ−1i + (c− 1)2‖β‖2

∆= g(c).

ùp, ��Øþj�K�, ·�b�gCþÚÏCþþ®IOz, Ïd�OÝX�n× pÝ.

ég(c)'uc¦�¿-Ù�u"�)�c��`��

c∗ =‖β‖2

σ2∑p

i=1 λ−1i + ‖β‖2

< 1. (3.10.1)

dug(c)'uc��ê�u

2σ2p∑i=1

λ−1i + 2‖β‖2 > 0,

Ïdg(c) = MSE(βs(c))3c∗?��, ¿��c∗ ≤ c < 1�,

kMSE(βs(c)) < MSE(β).

Zhejiang Universitypan.hohoweiya.xyz/regression-slides/chap 3.pdf · Zhejiang University September...

Documents