DOI: 10.1098/rstb.1997.0101. Generative models for discovering sparse distributed representations

Generative models for discovering sparse distributed representations

10.1098/rstb.1997.0101

Crossref journal-article

The Royal Society

Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences (175)

Abstract

We describe a hierarchical, generative model that can be viewed as a nonlinear generalization of factor analysis and can be implemented in a neural network. The model uses bottom–up, top–down and lateral connections to perform Bayesian perceptual inference correctly. Once perceptual inference has been performed the connection strengths can be updated using a very simple learning rule that only requires locally available information. We demonstrate that the network learns to extract sparse, distributed, hierarchical representations.

Bibliography

Hinton, G. E., & Ghahramani, Z. (1997). Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 352(1358), 1177â1190.

Authors 2

Geoffrey E. Hinton (first)
Zoubin Ghahramani (additional)

References 26 Referenced 118

Barlow H. 1989 Unsupervised learning. Neural Comput. 1 295^311. (10.1162/neco.1989.1.3.295)
Becker S. & Hinton G. 1992 A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355 161^163. (10.1038/355161a0)
Bishop C. M. Svensen M. & Williams C. K. I. 1997 GTM: a principled alternative to the self-organizing map. Neural Comput. (In the press.) (10.1007/3-540-61510-5_31)
{'key': 'p_4', 'first-page': '38', 'article-title': 'Maximum likelihood from incomplete data via the EM algorithm', 'volume': '39', 'author': 'Dempster A.', 'year': '1977', 'journal-title': 'Jl R. Statist. Soc. B'} / Jl R. Statist. Soc. B / Maximum likelihood from incomplete data via the EM algorithm by Dempster A. (1977)
Devroye L. 1986 Non-uniform random variate generation. New York: Springer. (10.1007/978-1-4613-8643-8)
Durbin R. & Willshaw D. 1987 An analogue approach to the travelling salesman problem using an elastic net method. Nature 326 689^691. (10.1038/326689a0)
Everitt B. S. 1984 An introduction to latent variable models. London: Chapman & Hall. (10.1007/978-94-009-5564-6)
Frey B. J. 1997 Continuous sigmoidal belief networks trained using slice sampling. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press.
{'key': 'p_9', 'first-page': '741', 'article-title': 'Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images', 'volume': '6', 'author': 'Geman S.', 'year': '1984', 'journal-title': 'IEEE Trans. on Pattern Analysis and Machine Intelligence'} / IEEE Trans. on Pattern Analysis and Machine Intelligence / Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images by Geman S. (1984)
Gilks W. R. & Wild P. 1992 Adaptive rejection sampling for Gibbs sampling. Appl. Statist. 41 337^348. (10.2307/2347565)
Gregory R. L. 1970 The intelligent eye. London: Wiedenfeld & Nicolson.
Hinton G. E. & Sejnowski T. J. 1986 Learning and relearning in Boltzmann machines. In Parallel distributed processing: explorations in the microstructure of cognition. Vol. 1: foundations (ed. D. E. Rumelhart & J. L. McClelland). Cambridge MA: MIT Press.
Hinton G. E. & Sejnowski T. J. 1983 Optimal perceptual Inference. In Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition pp. 448^453. Washington DC.
Hinton G. E. Dayan P. Frey B. J. & Neal R. M. 1995 The wake ^sleep algorithm for unsupervised neural networks. Science 268 1158^1161. (10.1126/science.7761831)
Horn B. K. P. 1977 Understanding image intensities. Artif. Intell. 88 201^231. (10.1016/0004-3702(77)90020-0)
10.1162/neco.1991.3.1.79
10.1007/BF00337288
Lee D. D. & Seung H. S. 1997 Unsupervised learning by convex and conic coding. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press.
Lewicki M. S. & Sejnowski T. J. 1997 Bayesian unsupervised learning of higher order structure. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press.
Mumford D. 1994 Neuronal architectures for pattern-theoretic problems. In Large-scale neuroneal theories of the brain (ed. C. Koch & J. L. Davis) pp.125^152. Cambridge MA: MIT Press.
10.1016/0004-3702(92)90065-6
Neal R. M. & Dayan P. 1996 Factor analysis using delta-rule wake ^sleep learning. Technical report no. 9607 Dept. of Statistics University of Toronto Canada.
Olshausen B. A. & Field D. J. 1996 Emergence of simple-cell receptive ¢eld properties by learning a sparse code for natural images. Nature 381 607^609. (10.1038/381607a0)
Pearl J. 1988 Probabilistic reasoning in intelligent systems: networks of plausible inference. San Mateo CA: Morgan Kaufmann. (10.1016/B978-0-08-051489-5.50008-4)
10.1207/s15516709cog0901_5
Rumelhart D. E. Hinton G. E. & Williams R. J. 1986 Learning internal representations by back-propagating errors. Nature 323 533^536. (10.1038/323533a0)

Dates

Type	When
Created	22 years, 11 months ago (Oct. 1, 2002, 1:23 p.m.)
Deposited	4 years, 6 months ago (Feb. 20, 2021, 1:11 p.m.)
Indexed	1 month, 3 weeks ago (July 11, 2025, 6:45 a.m.)
Issued	28 years ago (Aug. 29, 1997)
Published	28 years ago (Aug. 29, 1997)
Published Online	28 years ago (Aug. 29, 1997)
Published Print	28 years ago (Aug. 29, 1997)

Funders 0

None

BibTeX

@article{Hinton_1997, title={Generative models for discovering sparse distributed representations}, volume={352}, ISSN={1471-2970}, url={http://dx.doi.org/10.1098/rstb.1997.0101}, DOI={10.1098/rstb.1997.0101}, number={1358}, journal={Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences}, publisher={The Royal Society}, author={Hinton, Geoffrey E. and Ghahramani, Zoubin}, editor={Anderson, J. and Barlow, H. B. and Gregory, R. L.}, year={1997}, month=aug, pages={1177–1190} }

JSON

{
  "indexed": {
    "date-parts": [
      [
        2025,
        7,
        11
      ]
    ],
    "date-time": "2025-07-11T10:45:02Z",
    "timestamp": 1752230702114
  },
  "reference-count": 26,
  "publisher": "The Royal Society",
  "issue": "1358",
  "license": [
    {
      "start": {
        "date-parts": [
          [
            1997,
            8,
            29
          ]
        ],
        "date-time": "1997-08-29T00:00:00Z",
        "timestamp": 872812800000
      },
      "content-version": "tdm",
      "delay-in-days": 0,
      "URL": "https://royalsociety.org/journals/ethics-policies/data-sharing-mining/"
    }
  ],
  "content-domain": {
    "domain": [],
    "crossmark-restriction": false
  },
  "published-print": {
    "date-parts": [
      [
        1997,
        8,
        29
      ]
    ]
  },
  "abstract": "<jats:p>We describe a hierarchical, generative model that can be viewed as a nonlinear generalization of factor analysis and can be implemented in a neural network. The model uses bottom\u2013up, top\u2013down and lateral connections to perform Bayesian perceptual inference correctly. Once perceptual inference has been performed the connection strengths can be updated using a very simple learning rule that only requires locally available information. We demonstrate that the network learns to extract sparse, distributed, hierarchical representations.</jats:p>",
  "DOI": "10.1098/rstb.1997.0101",
  "type": "journal-article",
  "created": {
    "date-parts": [
      [
        2002,
        10,
        1
      ]
    ],
    "date-time": "2002-10-01T17:23:02Z",
    "timestamp": 1033492982000
  },
  "page": "1177-1190",
  "source": "Crossref",
  "is-referenced-by-count": 118,
  "title": "Generative models for discovering sparse distributed representations",
  "prefix": "10.1098",
  "volume": "352",
  "author": [
    {
      "given": "Geoffrey E.",
      "family": "Hinton",
      "sequence": "first",
      "affiliation": [
        {
          "name": "Department of Computer Science, University of TorontoToronto, Ontario, Canada M5S 1A4"
        }
      ]
    },
    {
      "given": "Zoubin",
      "family": "Ghahramani",
      "sequence": "additional",
      "affiliation": [
        {
          "name": "Department of Computer Science, University of TorontoToronto, Ontario, Canada M5S 1A4"
        }
      ]
    }
  ],
  "member": "175",
  "published-online": {
    "date-parts": [
      [
        1997,
        8,
        29
      ]
    ]
  },
  "reference": [
    {
      "key": "p_1",
      "doi-asserted-by": "crossref",
      "unstructured": "Barlow H. 1989 Unsupervised learning. Neural Comput. 1 295^311.",
      "DOI": "10.1162/neco.1989.1.3.295"
    },
    {
      "key": "p_2",
      "doi-asserted-by": "crossref",
      "unstructured": "Becker S. & Hinton G. 1992 A self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355 161^163.",
      "DOI": "10.1038/355161a0"
    },
    {
      "key": "p_3",
      "doi-asserted-by": "crossref",
      "unstructured": "Bishop C. M. Svensen M. & Williams C. K. I. 1997 GTM: a principled alternative to the self-organizing map. Neural Comput. (In the press.)",
      "DOI": "10.1007/3-540-61510-5_31"
    },
    {
      "key": "p_4",
      "first-page": "38",
      "article-title": "Maximum likelihood from incomplete data via the EM algorithm",
      "volume": "39",
      "author": "Dempster A.",
      "year": "1977",
      "journal-title": "Jl R. Statist. Soc. B"
    },
    {
      "key": "p_5",
      "doi-asserted-by": "crossref",
      "unstructured": "Devroye L. 1986 Non-uniform random variate generation. New York: Springer.",
      "DOI": "10.1007/978-1-4613-8643-8"
    },
    {
      "key": "p_6",
      "doi-asserted-by": "crossref",
      "unstructured": "Durbin R. & Willshaw D. 1987 An analogue approach to the travelling salesman problem using an elastic net method. Nature 326 689^691.",
      "DOI": "10.1038/326689a0"
    },
    {
      "key": "p_7",
      "doi-asserted-by": "crossref",
      "unstructured": "Everitt B. S. 1984 An introduction to latent variable models. London: Chapman & Hall.",
      "DOI": "10.1007/978-94-009-5564-6"
    },
    {
      "key": "p_8",
      "unstructured": "Frey B. J. 1997 Continuous sigmoidal belief networks trained using slice sampling. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press."
    },
    {
      "key": "p_9",
      "first-page": "741",
      "article-title": "Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images",
      "volume": "6",
      "author": "Geman S.",
      "year": "1984",
      "journal-title": "IEEE Trans. on Pattern Analysis and Machine Intelligence"
    },
    {
      "key": "p_10",
      "doi-asserted-by": "crossref",
      "unstructured": "Gilks W. R. & Wild P. 1992 Adaptive rejection sampling for Gibbs sampling. Appl. Statist. 41 337^348.",
      "DOI": "10.2307/2347565"
    },
    {
      "key": "p_11",
      "unstructured": "Gregory R. L. 1970 The intelligent eye. London: Wiedenfeld & Nicolson."
    },
    {
      "key": "p_12",
      "unstructured": "Hinton G. E. & Sejnowski T. J. 1986 Learning and relearning in Boltzmann machines. In Parallel distributed processing: explorations in the microstructure of cognition. Vol. 1: foundations (ed. D. E. Rumelhart & J. L. McClelland). Cambridge MA: MIT Press."
    },
    {
      "key": "p_13",
      "unstructured": "Hinton G. E. & Sejnowski T. J. 1983 Optimal perceptual Inference. In Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition pp. 448^453. Washington DC."
    },
    {
      "key": "p_14",
      "doi-asserted-by": "crossref",
      "unstructured": "Hinton G. E. Dayan P. Frey B. J. & Neal R. M. 1995 The wake ^sleep algorithm for unsupervised neural networks. Science 268 1158^1161.",
      "DOI": "10.1126/science.7761831"
    },
    {
      "key": "p_15",
      "doi-asserted-by": "crossref",
      "unstructured": "Horn B. K. P. 1977 Understanding image intensities. Artif. Intell. 88 201^231.",
      "DOI": "10.1016/0004-3702(77)90020-0"
    },
    {
      "key": "p_16",
      "doi-asserted-by": "publisher",
      "DOI": "10.1162/neco.1991.3.1.79"
    },
    {
      "key": "p_17",
      "doi-asserted-by": "publisher",
      "DOI": "10.1007/BF00337288"
    },
    {
      "key": "p_18",
      "unstructured": "Lee D. D. & Seung H. S. 1997 Unsupervised learning by convex and conic coding. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press."
    },
    {
      "key": "p_19",
      "unstructured": "Lewicki M. S. & Sejnowski T. J. 1997 Bayesian unsupervised learning of higher order structure. In Advances in neural information processing systems 9 (ed. M. Mozer M. Jordan & T. Petsche). Cambridge MA: MIT Press."
    },
    {
      "key": "p_20",
      "unstructured": "Mumford D. 1994 Neuronal architectures for pattern-theoretic problems. In Large-scale neuroneal theories of the brain (ed. C. Koch & J. L. Davis) pp.125^152. Cambridge MA: MIT Press."
    },
    {
      "key": "p_21",
      "doi-asserted-by": "publisher",
      "DOI": "10.1016/0004-3702(92)90065-6"
    },
    {
      "key": "p_22",
      "unstructured": "Neal R. M. & Dayan P. 1996 Factor analysis using delta-rule wake ^sleep learning. Technical report no. 9607 Dept. of Statistics University of Toronto Canada."
    },
    {
      "key": "p_23",
      "doi-asserted-by": "crossref",
      "unstructured": "Olshausen B. A. & Field D. J. 1996 Emergence of simple-cell receptive \u00a2eld properties by learning a sparse code for natural images. Nature 381 607^609.",
      "DOI": "10.1038/381607a0"
    },
    {
      "key": "p_24",
      "doi-asserted-by": "crossref",
      "unstructured": "Pearl J. 1988 Probabilistic reasoning in intelligent systems: networks of plausible inference. San Mateo CA: Morgan Kaufmann.",
      "DOI": "10.1016/B978-0-08-051489-5.50008-4"
    },
    {
      "key": "p_25",
      "doi-asserted-by": "publisher",
      "DOI": "10.1207/s15516709cog0901_5"
    },
    {
      "key": "p_26",
      "doi-asserted-by": "crossref",
      "unstructured": "Rumelhart D. E. Hinton G. E. & Williams R. J. 1986 Learning internal representations by back-propagating errors. Nature 323 533^536.",
      "DOI": "10.1038/323533a0"
    }
  ],
  "container-title": "Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences",
  "original-title": [],
  "language": "en",
  "link": [
    {
      "URL": "https://royalsocietypublishing.org/doi/pdf/10.1098/rstb.1997.0101",
      "content-type": "application/pdf",
      "content-version": "vor",
      "intended-application": "text-mining"
    },
    {
      "URL": "https://royalsocietypublishing.org/doi/pdf/10.1098/rstb.1997.0101",
      "content-type": "unspecified",
      "content-version": "vor",
      "intended-application": "similarity-checking"
    }
  ],
  "deposited": {
    "date-parts": [
      [
        2021,
        2,
        20
      ]
    ],
    "date-time": "2021-02-20T18:11:13Z",
    "timestamp": 1613844673000
  },
  "score": 1,
  "resource": {
    "primary": {
      "URL": "https://royalsocietypublishing.org/doi/10.1098/rstb.1997.0101"
    }
  },
  "subtitle": [],
  "editor": [
    {
      "given": "J.",
      "family": "Anderson",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "given": "H. B.",
      "family": "Barlow",
      "sequence": "additional",
      "affiliation": []
    },
    {
      "given": "R. L.",
      "family": "Gregory",
      "sequence": "additional",
      "affiliation": []
    }
  ],
  "short-title": [],
  "issued": {
    "date-parts": [
      [
        1997,
        8,
        29
      ]
    ]
  },
  "references-count": 26,
  "journal-issue": {
    "issue": "1358",
    "published-print": {
      "date-parts": [
        [
          1997,
          8,
          29
        ]
      ]
    }
  },
  "alternative-id": [
    "10.1098/rstb.1997.0101"
  ],
  "URL": "http://dx.doi.org/10.1098/rstb.1997.0101",
  "relation": {},
  "ISSN": [
    "0962-8436",
    "1471-2970"
  ],
  "subject": [],
  "container-title-short": "Phil. Trans. R. Soc. Lond. B",
  "published": {
    "date-parts": [
      [
        1997,
        8,
        29
      ]
    ]
  }
}