Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks
10.1126/science.247.4945.978
Crossref journal-article
American Association for the Advancement of Science (AAAS)
Science (221)
Abstract

Learning an input-output mapping from a set of examples, of the type that many neural networks have been constructed to perform, can be regarded as synthesizing an approximation of a multidimensional function (that is, solving the problem of hypersurface reconstruction). From this point of view, this form of learning is closely related to classical approximation techniques, such as generalized splines and regularization theory. A theory is reported that shows the equivalence between regularization and a class of three-layer networks called regularization networks or hyper basis functions. These networks are not only equivalent to generalized splines but are also closely related to the classical radial basis functions used for interpolation tasks and to several pattern recognition and neural network algorithms. They also have an interesting interpretation in terms of prototypes that are synthesized and optimally combined during the learning stage.

Bibliography

Poggio, T., & Girosi, F. (1990). Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks. Science, 247(4945), 978–982.

Authors 2
  1. T. Poggio (first)
  2. F. Girosi (additional)
References 37 Referenced 756
  1. 10.1016/0025-5564(71)90051-4
  2. BERTERO, M, ILL-POSED PROBLEMS IN EARLY VISION, PROCEEDINGS OF THE IEEE 76: 869 (1988). (10.1109/5.5962) / PROCEEDINGS OF THE IEEE (1988)
  3. BROOMHEAD, D.S., Multivariable Functional Interpolation and Adaptive Networks, COMPLEX SYSTEMS 2: 321 (1988). / COMPLEX SYSTEMS / Multivariable Functional Interpolation and Adaptive Networks (1988)
  4. CASDAGLI, M, NONLINEAR PREDICTION OF CHAOTIC TIME-SERIES, PHYSICA D 35: 335 (1989). (10.1016/0167-2789(89)90074-2) / PHYSICA D (1989)
  5. Cybenko, G., Mathematics of Control, Signals, and Systems 2: 303 (1989). (10.1007/BF02551274) / Mathematics of Control, Signals, and Systems (1989)
  6. Duda R. O. Pattem Classification and Scene Analysis (1973).
  7. DURBIN, R, NEURAL COMPUT 1: 133 (1989). (10.1162/neco.1989.1.1.133) / NEURAL COMPUT (1989)
  8. FRANKE, R, SCATTERED DATA INTERPOLATION - TESTS OF SOME METHODS, MATHEMATICS OF COMPUTATION 38: 181 (1982). / MATHEMATICS OF COMPUTATION (1982)
  9. GIROSI F ARTIFICIAL INTELLIGENCE MEMORANDUM 1164 (1989).
  10. Hand D. J. Kernel Discriminant Analysis (1982).
  11. Kanerva P. Sparse Distributed Memory (1988).
  12. KEELER, J.D., COMPARISON BETWEEN KANERVA SDM AND HOPFIELD-TYPE NEURAL NETWORKS, COGNITIVE SCIENCE 12: 299 (1988). / COGNITIVE SCIENCE (1988)
  13. KIMELDORF, G, A CORRESPONDENCE BETWEEN BAYESIAN ESTIMATION ON STOCHASTIC PROCESSES AND SMOOTHING BY SPLINES, ANNALS OF MATHEMATICAL STATISTICS 41: 495 (1970). (10.1214/aoms/1177697089) / ANNALS OF MATHEMATICAL STATISTICS (1970)
  14. KOHONEN, T, SELF-ORGANIZED FORMATION OF TOPOLOGICALLY CORRECT FEATURE MAPS, BIOLOGICAL CYBERNETICS 43: 59 (1982). (10.1007/BF00337288) / BIOLOGICAL CYBERNETICS (1982)
  15. Lapedes A. Los Alamos National Laboratory Technical Report LA-UR-87-2662 (1982).
  16. MacQueen, J., Proceedings: 5th Berkeley Symposium on Mathematics, Statistics, and Probability 1: 281 (1967). / Proceedings: 5th Berkeley Symposium on Mathematics, Statistics, and Probability (1967)
  17. MARR, D, JOURNAL OF PHYSIOLOGY-LONDON 202: 437 (1969). (10.1113/jphysiol.1969.sp008820) / JOURNAL OF PHYSIOLOGY-LONDON (1969)
  18. MARROQUIN, J, PROBABILISTIC SOLUTION OF III-POSED PROBLEMS IN COMPUTATIONAL VISION, JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION 82: 76 (1987). (10.1080/01621459.1987.10478393) / JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION (1987)
  19. MICCHELLI, C.A., INTERPOLATION OF SCATTERED DATA - DISTANCE MATRICES AND CONDITIONALLY POSITIVE DEFINITE FUNCTIONS, CONSTRUCTIVE APPROXIMATION 2: 11 (1986). (10.1007/BF01893414) / CONSTRUCTIVE APPROXIMATION (1986)
  20. MOODY, J, NEURAL COMPUT 1: 281 (1989). (10.1162/neco.1989.1.2.281) / NEURAL COMPUT (1989)
  21. OMOHUNDRO, S.M., Efficient Algorithms with Neural Network Behavior, COMPLEX SYSTEMS 1: 273 (1987). / COMPLEX SYSTEMS / Efficient Algorithms with Neural Network Behavior (1987)
  22. PERRETT, D.I., VISUAL NEURONS RESPONSIVE TO FACES, TRENDS IN NEUROSCIENCES 10: 358 (1987). (10.1016/0166-2236(87)90071-3) / TRENDS IN NEUROSCIENCES (1987)
  23. Poggio, T., Nature 343: 263 (1990). (10.1038/343263a0) / Nature (1990)
  24. POGGIO T ARTIFICIAL INTELLIGENCE MEMORANDUM 1140 (1989).
  25. POGGIO, T, COMPUTATIONAL VISION AND REGULARIZATION THEORY, NATURE 317: 314 (1985). (10.1038/317314a0) / NATURE (1985)
  26. Poggio, T., Proceedings Image Understanding Workshop: 1 (1988). / Proceedings Image Understanding Workshop (1988)
  27. Powell, M. J. D., Algorithms for Approximation: 143 (1987). / Algorithms for Approximation (1987)
  28. Renals, S., Proceedings of the International Joint Conference on Neural Networks 1: 461 (1990). / Proceedings of the International Joint Conference on Neural Networks (1990)
  29. 10.1016/0005-1098(78)90005-5
  30. RUMELHART, D.E., LEARNING REPRESENTATIONS BY BACK-PROPAGATING ERRORS, NATURE 323: 533 (1986). (10.1038/323533a0) / NATURE (1986)
  31. Schumaker L. L. Spline Functions: Basic Theory (1981).
  32. SEJNOWSKI, T.J., Parallel Networks that Learn to Pronounce English Text, COMPLEX SYSTEMS 1: 145 (1987). / COMPLEX SYSTEMS / Parallel Networks that Learn to Pronounce English Text (1987)
  33. Tikhonov A. N. Solutions of Ill-Posed Problem (1977).
  34. Wahba, G., Approximation Theory III: 905 (1980). / Approximation Theory III (1980)
  35. Wahba G. Spines Modelsfor Observational Data 59 (1990). (10.1137/1.9781611970128)
  36. WOLPERT, D.H., A BENCHMARK FOR HOW WELL NEURAL NETS GENERALIZE, BIOLOGICAL CYBERNETICS 61: 303 (1989). (10.1007/BF00203178) / BIOLOGICAL CYBERNETICS (1989)
  37. Wolpert, D. H., Abstract of the First Annual International Neural Network Society Meeting: 474 (1988). / Abstract of the First Annual International Neural Network Society Meeting (1988)
Dates
Type When
Created 18 years, 10 months ago (Oct. 5, 2006, 6:10 p.m.)
Deposited 1 year, 7 months ago (Jan. 12, 2024, 11:05 a.m.)
Indexed 1 day, 19 hours ago (Aug. 21, 2025, 1:29 p.m.)
Issued 35 years, 6 months ago (Feb. 23, 1990)
Published 35 years, 6 months ago (Feb. 23, 1990)
Published Print 35 years, 6 months ago (Feb. 23, 1990)
Funders 0

None

@article{Poggio_1990, title={Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks}, volume={247}, ISSN={1095-9203}, url={http://dx.doi.org/10.1126/science.247.4945.978}, DOI={10.1126/science.247.4945.978}, number={4945}, journal={Science}, publisher={American Association for the Advancement of Science (AAAS)}, author={Poggio, T. and Girosi, F.}, year={1990}, month=feb, pages={978–982} }