Polynomial Regression by Gaussian Mixture Rodolphe Priam

Abstract. By modelling a data sample by a mixture of multidimensional Gaussian mixture, we get a natural expression for the nonlinear regression function. This framework permits to introduce the polynomial regression regularized by a mixture. Keywords: Data Analysis, Probabilistic mixture, Polynomial regression.

1

Introduction

The regression problem is to find an estimate for an unknown function f ∗ from which we only have a finite sample D = {(xi , yi ), i = 1, 2, · · · , n}, the input data xi and the ouput data yi , such as yi = f ∗ (xi ). In a statistical approach, we suppose that each data (xi , yi ) was generated by a joint random variable (X, Y ) whoseR distribution P is unknown too. It is well known that R E(Y |X) = argminf ||Y − f (X)||2 dP (X, Y ) is the true regression function. To estimate f ∗ , a common parametric solution is to P find the optimal parameter θ of the function fθ which minimizes the criterion i ||yi −fθ (xi )||2 under some smoothing hypothesis, and where fθ comes from an a priori family of functions. We need regularization because the goal is not to mimic the data sample but to predict new values. The more flexible model is not generally the best and a trade off between simple model and a complex one is necessary. In this work, we use a mixture model to regulate the compromise.

2

The regression finite mixture

We can naturally estimate P (X, Y ) by a mixture[McLachlan and Peel, 2000] of Gaussian densities before evaluating the conditional mean E(Y |X). The big interest with such a mixture is that it is a consistent estimator of any continuous density while it gives a closed form solution for the following developments. We write with X ∈ Rp , Y ∈ Rq , and a Gaussian g(; ak ) where ak = (mk , Σk ), the mean mk and the covariance matrix Σk :       k=K X mkx Σkxx Σkxy x ; ak ) with mk = Σk = πk g( Pa (x, y) = mky Σkyx Σkyy y k=1

−1 If (Xk , Yk ) ∼ g(; mk , Σk ) then E(Yk |Xk ) = mky + Σkyx Σkxx Xk , so we get[Sung, 2004] : Z X πk g(x; mkx , Σkxx )   P mky + Σkyx (Σkxx )−1 x EPa (Y |X = x) = ydPa (y|x) = l πl g(x; mlx , Σkxx ) k

2

Priam R.

This generative approach needs a good a as found by an maximum likelihood approach where the EM[Dempster et al., 1977] algorithm can seek the optimal value a ˆ. Next, we propose a solution by a polynomial interpolation.

3

The polynomial regression mixture

To get a simple expression, we suppose x and y one-dimensional vectors, replacing Σ matrices by σ 2 values. If fθ is a polynomial function with degree u ∈ N+ , we can write: ˆ = argminθ θ(a)

Z Z R

R

y−

t=u X t=0


2 θt xt Pa (x, y)dxdy

We derive under the integrand sign and find the linear problem:  ˆ  θ0 EPa (X 2 ) · · · EPa (X u ) u+1   ˆ  3 )   θ1  EPa (X ) · · · EPa (X .    ..  2u u+2 u+1 u ) · · · EPa (X ) ) EPa (X EPa (X ) EPa (X θˆu

  EPa (X 0 ) EPa (X 1 ) EPa (Y X 0 )  EPa (Y X 1 )   EPa (X 1 ) EPa (X 2 )    =  .. ..   . . 

EPa (Y X u )

and δt2 = δt+1 modulo 2=0 , we get : P t EPa (X t ) = δt1 × 1 3 5 · · · (t − 1) k πn k σkx P σ2 t δt1 (mky − σkxy EPa (Y X t ) = 1 3 5 · · · (t − 1) k πk σkx mkx ) − δt2 2

If δt1 = δt modulo

2=0

kx

4

2 σkxy 2 σkx

t σkx

o

Conclusion

We have shown that a solution exists for a polynomial interpolation. Work is currently done to compare this solution with the more usual form. This should gives new results about polynomial fitting as for instance quick statistical test to bound the polynomial degree.

References [Dempster et al., 1977]A.P. Dempster, N.M. Laird, and D.B. Rubin. Maximumlikelihood from incomplete data via the em algorithm. J. Royal Statist. Soc. Ser. B., 39, 1977. [McLachlan and Peel, 2000]G. McLachlan and D. Peel. Finite Mixture Models. John Wiley & Sons, 2000. [Sung, 2004]Hsi Guang Sung. Gaussian Mixture Regression and Classification. PhD thesis, Rice University, 2004.