BPadvertisementfrom

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Wednesday, 26 June 2013

Kolmogorov, Solomonoff, and de Finetti

Posted on 08:59 by Unknown
This is a nice historical article that connects a number of key figures in the history of probability and information theory. See also Frequentists and Bayesians, Jaynes and Bayes, and On the Origin of Probability in Quantum Mechanics.
Induction: From Kolmogorov and Solomonoff to De Finetti and Back to Kolmogorov
John J. McCall DOI: 10.1111/j.0026-1386.2004.00190.x

ABSTRACT This paper compares the solutions to “the induction problem” by Kolmogorov, de Finetti, and Solomonoff. Brief sketches of the intellectual history of de Finetti and Kolmogorov are also composed. Kolmogorov's contributions to information theory culminated in his notion of algorithmic complexity. The development of algorithmic complexity was inspired by information theory and randomness. Kolmogorov's best-known contribution was the axiomatization of probability in 1933. Its influence on probability and statistics was swift, dramatic, and fundamental. However, Kolmogorov was not satisfied by his treatment of the frequency aspect of his creation. This in time gave rise to Kolmogorov complexity. De Finetti, on the other hand, had a profound vision early in his life which was encapsulated in his exchangeability theorem. This insight simultaneously resolved a fundamental philosophical conundrum—Hume's problem, and provided the bricks and mortar for de Finetti's constructive probabilistic theory. Most of his subsequent research involved extensions of his representation theorem. De Finetti was against determinism and celebrated quantum theory, while Kolmogorov was convinced that in every seemingly indeterministic manifestation there lurked a hidden deterministic mechanism. Solomonoff introduced algorithmic complexity independently of Kolmogorov and Chaitin. Solomonoff's motivation was firmly focused on induction. His interest in induction was to a marked extent sparked by Keynes’ 1921 seminal book. This interest in induction has never faltered, remaining prominent in his most recent research. The decisive connection between de Finetti and Kolmogorov was their lifelong interest in the frequency aspect of induction. Kolmogorov's solution to the problem was algorithmic complexity. De Finetti's solution to his frequency problem occurred early in his career with the discovery of the representation theorem. In this paper, we try to explain these solutions and mention related topics which captured the interest of these giants.
Excerpt below from the paper. I doubt Nature (evolution) uses Solomonoff's Universal Prior (determined by minimum length programs), as it is quite expensive to compute. I think our priors and heuristics are much more specialized and primitive.
... There are a host of similarities and equivalences joining concepts like Shannon entropy, Kolmogorov complexity, maximum entropy, Bayes etc. Some of these are noted. In short, the psychological aspects of probability and induction emphasized by de Finetti, Ramsey and Keynes may eventually emerge from a careful and novel neuroscientific study. In this study, neuronal actors would interact as they portray personal perception and mem- ories. In this way, these versatile neuronal actors comprise the foundation of psychology.

... In comparing de Finetti and Kolmogorov one becomes entangled in a host of controversial issues. de Finetti was an indeterminist who championed a subjective inductive inference. Kolmogorov sought determinism in even the most chaotic natural phenomena and reluctantly accepted a frequency approach to statistics, an objective science. In the 1960s Kolmogorov questioned his frequency position and developed algorithmic complexity, together with Solomonoff and Chaitin. With respect to the frequency doubts, he would have been enlightened by de Finetti’s representation theorem. The KCS research remains an exciting research area in probability, statistics and computer science. It has raised a whole series of controversial issues. It appears to challenge both the frequency and Bayesian schools: the former by proclaiming that the foundations of uncertainty are to be found in algorithms, information and combinatorics, rather than probability; the latter by replacing the Bayes prior which differs across individuals in accord with their distinctive beliefs with a universal prior applicable to all and drained of subjectivity.
Email ThisBlogThis!Share to XShare to FacebookShare to Pinterest
Posted in ai, bayes, evolution, machine learning, mathematics, probability, statistics | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • PhD Comics: the movie
    PHD Movie Trailer from PHD Comics on Vimeo . I met Jorge Cham , the cartoonist who draws PhD Comics, a few years ago at Sci Foo. Cham was ...
  • Finding the Next Einstein
    Duke researcher Jonathan Wai interviewed me for his Psychology Today blog, Finding the Next Einstein . Below are my answers to two of his q...
  • Beanbags and causal variants
    Not only do these results implicate common causal variants as the source of heritability in disease susceptibility, but they also suggest th...
  • Sitzfleisch
    Freeman Dyson reviews the new biography of Oppenheimer by Ray Monk. I discussed the book already here . NYBooks : ... The subtitle, “A Life ...
  • A blog is born
    Raghu Parasarathy , a biophysicist at U Oregon, and my correspondent in this previous post on faculty blogging, has decided to try it out. ...
  • News from Microsoft Research Faculty Summit 2013
    Measuring the maximal commuting subset of observables uniquely determines the pure state of a quantum system (recently proved Kadison-Singer...
  • Talk cancelled
    This talk has been cancelled, for complex reasons that I will not discuss.
  • East Asian sociopaths?
    Some would assert that CEOs and other people in leadership positions are often warm sociopaths . Interestingly, it is claimed that there is ...
  • Swedish height in the 20th century
    Average height of Swedish military conscripts during the 20th century. Looks like an increase of roughly 1 cm per decade or about 1.5 SD in ...
  • The differences are enormous
    Luis Alvarez laid it out bluntly: The world of mathematics and theoretical physics is hierarchical. That was my first exposure to it. There...

Categories

  • ability (2)
  • academia (9)
  • affirmative action (8)
  • ai (13)
  • aig (1)
  • alan turing (3)
  • algorithms (2)
  • alpha (2)
  • american society (54)
  • art (3)
  • ashkenazim (1)
  • aspergers (4)
  • athletics (6)
  • autism (4)
  • autobiographical (13)
  • basketball (4)
  • bayes (1)
  • behavioral economics (4)
  • berkeley (5)
  • bgi (24)
  • biology (23)
  • biotech (6)
  • bjj (5)
  • black holes (4)
  • blade runner (2)
  • blogging (3)
  • books (5)
  • borges (2)
  • bounded rationality (10)
  • brainpower (57)
  • bubbles (3)
  • caltech (14)
  • cambridge uk (1)
  • careers (18)
  • charles darwin (1)
  • chet baker (2)
  • China (25)
  • christmas (1)
  • class (2)
  • cognitive science (35)
  • cold war (1)
  • complexity (1)
  • computing (9)
  • conferences (4)
  • cosmology (4)
  • creativity (3)
  • credit crisis (10)
  • crossfit (5)
  • cryptography (2)
  • data mining (4)
  • dating (2)
  • demographics (1)
  • derivatives (5)
  • determinism (1)
  • digital books (1)
  • dna (4)
  • economic history (5)
  • economics (38)
  • econtalk (2)
  • ecosystems (1)
  • education (5)
  • efficient markets (8)
  • Einstein (2)
  • elitism (14)
  • encryption (1)
  • energy (1)
  • entrepreneurs (3)
  • entropy (1)
  • environmentalism (1)
  • eugene (3)
  • evolution (19)
  • expert prediction (6)
  • fake alpha (2)
  • feminism (2)
  • Fermi problems (2)
  • feynman (7)
  • film (9)
  • finance (42)
  • fitness (3)
  • flynn effect (1)
  • foo camp (1)
  • football (5)
  • france (1)
  • free will (1)
  • freeman dyson (2)
  • fx (2)
  • game theory (1)
  • geeks (2)
  • gender (4)
  • genetic engineering (15)
  • genetics (79)
  • genius (24)
  • genomics (2)
  • geopolitics (7)
  • gilded age (13)
  • global warming (1)
  • globalization (23)
  • godel (2)
  • goldman sachs (2)
  • google (4)
  • happiness (2)
  • harvard (8)
  • harvard society of fellows (5)
  • hedge funds (4)
  • hedonic treadmill (1)
  • height (2)
  • higher education (38)
  • history (8)
  • history of science (12)
  • hormones (3)
  • hugh everett (2)
  • human capital (34)
  • humor (1)
  • income inequality (21)
  • india (2)
  • industrial revolution (1)
  • innovation (38)
  • intellectual history (10)
  • intellectual property (1)
  • intellectual ventures (1)
  • internet (4)
  • iq (16)
  • italy (4)
  • james salter (3)
  • japan (4)
  • jiujitsu (8)
  • keynes (1)
  • kids (13)
  • lewontin fallacy (1)
  • lhc (1)
  • literature (12)
  • luck (1)
  • machine learning (8)
  • malcolm gladwell (1)
  • manhattan (2)
  • many worlds (10)
  • mathematics (14)
  • meritocracy (7)
  • microsoft (2)
  • mma (10)
  • monsters (2)
  • moore's law (1)
  • movies (9)
  • MSU (18)
  • music (5)
  • mutants (2)
  • nathan myhrvold (1)
  • neal stephenson (1)
  • neanderthals (2)
  • nerds (3)
  • net worth (5)
  • neuroscience (7)
  • new yorker (1)
  • nicholas metropolis (1)
  • noam chomsky (2)
  • nobel prize (2)
  • nsa (2)
  • nuclear weapons (5)
  • obama (7)
  • olympics (4)
  • oppenheimer (7)
  • patents (1)
  • personality (9)
  • philip k. dick (1)
  • philosophy of mind (2)
  • photos (40)
  • physical training (13)
  • physics (73)
  • podcasts (10)
  • political correctness (6)
  • politics (4)
  • pop culture (2)
  • prisoner's dilemma (1)
  • privacy (2)
  • probability (5)
  • prostitution (2)
  • psychology (25)
  • psychometrics (31)
  • qcd (1)
  • quants (9)
  • quantum computers (2)
  • quantum field theory (3)
  • quantum mechanics (18)
  • race relations (10)
  • real estate (1)
  • realpolitik (6)
  • renaissance technologies (1)
  • research (3)
  • russia (2)
  • sad but true (2)
  • sci fi (8)
  • science (42)
  • sec (1)
  • security (5)
  • silicon valley (6)
  • singularity (1)
  • smpy (1)
  • social networks (2)
  • social science (12)
  • software development (2)
  • solar energy (1)
  • sports (13)
  • startups (19)
  • statistics (16)
  • success (2)
  • taiwan (1)
  • talks (16)
  • teaching (2)
  • technology (34)
  • television (2)
  • travel (24)
  • turing test (1)
  • ufc (8)
  • ultimate fighting (1)
  • universities (33)
  • university of oregon (6)
  • usain bolt (2)
  • venture capital (3)
  • volatility (1)
  • von Neumann (10)
  • wall street (2)
  • war (1)
  • warren buffet (1)
  • wwii (3)

Blog Archive

  • ▼  2013 (134)
    • ►  August (10)
    • ►  July (15)
    • ▼  June (22)
      • History will remember their names
      • Bert and Ernie, Out?
      • Solomonoff Universal Induction
      • Views of Hong Kong
      • Kolmogorov, Solomonoff, and de Finetti
      • HDL-C heritability from whole genomes: common vari...
      • Android Dreams
      • WDIST and PLINK
      • Ken Wilson, dead at 77
      • Beanbags and causal variants
      • Sinica podcast
      • China 3.0
      • Erdos and Tao
      • Spy vs Spy
      • The ratchet of power
      • If you can't fix it you've got to stand it
      • Horizons of truth
      • Weak Meat Strong Eat
      • Morgan Freeman on physics and physicists
      • All that is left is the wind in the pines
      • Lore
      • The genetics of humanness
    • ►  May (20)
    • ►  April (21)
    • ►  March (18)
    • ►  February (14)
    • ►  January (14)
  • ►  2012 (222)
    • ►  December (17)
    • ►  November (19)
    • ►  October (20)
    • ►  September (25)
    • ►  August (19)
    • ►  July (18)
    • ►  June (16)
    • ►  May (20)
    • ►  April (16)
    • ►  March (18)
    • ►  February (20)
    • ►  January (14)
  • ►  2011 (144)
    • ►  December (20)
    • ►  November (16)
    • ►  October (25)
    • ►  September (23)
    • ►  August (21)
    • ►  July (26)
    • ►  June (13)
Powered by Blogger.

About Me

Unknown
View my complete profile