Freeman Dyson - Fermi's rejection of our work (94/157)

Freeman Dyson recounts the episode [1] that inspired this paper in his web of life interviews (prepositioned to the fitting an elephant bit) [2] [1] [VIDEO] [2] [VIDEO]

> > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” For those who are interested, you can watch Freeman Dyson recount this conversation in his own words in an interview: [VIDEO]

Full paper here[1], and here's[2] the earlier paper about the model itself. They overloaded the CCC acronym, which has also been used for Penrose's Conformal cyclic cosmology[3], though can't say which is more speculative. I'm also reminded of Von Neumann's elephant[4], given that they vary the constants it doesn't seem that surprising I suppose that they can get a fit. Freeman Dyson learned this the hard way[5]. This is just one test, the article mentions a few others that they'll need to pass to be a serious contender. Will be interesting to see how it goes. [1]: [LINK] [2]: [LINK] [3]: [LINK] [4]: [LINK] [5]: [VIDEO]

I found the reference. It was an interview with Freeman Dyson where he said "quarks were invented", and I don't think it's a slip of the tongue. It's at 3:17 in this video of an interview with him [VIDEO] It's representative of a view that there's a thing (a particle) that explains another thing (a force) that was consistent with both theory and experiment. Thus, a quark could be a particle, and it was whatever the experimentalists and theorists said it was, however it was measured or contemplated. For a deeper meaning it becomes an exercise in hermeneutics i.e what does it mean when we say "particle"? That was the point of the original piece - there is no uncontested view of a particle's form, should one even exist. Each field, in order to advance, finds it useful to interpret it, or think about it, in different ways.

I haven't had a chance to read that, but that quote suggests I should (especially considering the author and the editors). I often refer to "Elephant Fitting" w.r.t these systems. I suspect you understand this, but I think most think it is just about overfitting. But the way problem isn't about the number of parameters, but that parameters need to be justified. As explained by Dyson here[0]. Vladimir's quote really reminds me of this. Fermi likewise was stressing the importance of theory. I think it is a profound quote, and you were (are?) lucky to have that friendship. I do think abstraction is at the heart of intelligence. François Chollet discusses it a lot, and he's far from alone. It seems to be well agreed upon in the neuroscience and cognitive science communities. I think this is essential to understand in our path forward to developing intelligent systems, because there are plenty of problems that need to be solved in which there is no algorithmic procedure. Where there is no explicit density function. Intractable, doubly intractable, and more problems. Maybe we're just too dumb, but it's clear there are plateaus where luck is needed to advance. I do not believe our current machines would be capable of closing a gap. [0] [VIDEO]

Interestingly a lot of the math and physics people in the ML community are considered "grumpy researchers." A joke apparent with this starter pack[0]. From my personal experience (undergrad physics, worked as engineer, came to CS & ML because I liked the math), there's a lot of pushback. - I've been told that the math doesn't matter/you don't need math. - I've heard very prominent researchers say "fuck theorists" - I've seen papers routinely rejected for improving training techniques with reviewers say "just tune a large model" - I see papers that show improvements when conditioning comparisons on compute restraints because "not enough datasets" or "but does it scale" (these questions can always be asked but require exponentially more work) - I've been told I'm gatekeeping for saying "you don't need math to make good models, but you need it to know why your models are wrong" (yes, this is a reference) - when pointing out math or statistical errors I'm told it doesn't matter - and much more. I've heard this from my advisor, dissertation committee, bosses[1], peers, and others (of course, HN). If my experience is short of being rare, I think it explains the grumpy group[2]. But I'm also not too surprised with how common it is in CS for people to claim that everything is easy or that leet code is proof of competence (as opposed to evidence). I think unfortunately the problem is a bit bigger, but it isn't unsolvable. Really, it is "easily" solvable since it just requires us to ma

I don't think this makes sense and I'm not quite sure why you went to ML, but that's okay. I am a machine learning researcher, but also frustrated with the state of machine learning, in part because, well... you can probably see how "proof by empirical evidence" is dialed up to 11. Sorry, long answer incoming. It is far from complete too but I think it will help build strong intuition around your questions. Will knowledge transfer? That entirely depends on the new problem. It also entirely depends on how related the problem is. But also, what information was used to solve the pre-transfer state. Take LLMs for example. There's lots of works that have shown them being difficult to train for solving calculations. Where they will do well on problems with the same number of digits but this will degrade rapidly as number of digits increase. It can be weird to read some of these papers as there will sometimes be periodic relationships with the number of digits but that should give us information about how they're encoding the problems. But that lack of transferability indicates that despite the problem solving and what we'd believe is actually just the same problem, doesn't mean it is. So you have to be really careful here, because us humans are really fucking good at generalization (yeah, we also suck, but a big part is our proficiency makes us recognize where we lack. But also, this is more a "humans can" more than "humans do" type of thing. So be careful when comparing). This gen

> I feel like extensive pretraining goes against the spirit of generality. What do you mean by generality? Pretraining is fine. It is even fine for pursuit to AGI. Humans and every animal has "baked in" memory. You're born knowing how to breath and have latent fears (chickens and hawks). Generalization is the ability to learn on a subset of something and then adapt to the entire (or a much larger portion) of the superset. It's always been that way. Humans do this, right? You learn some addition, subtraction, multiplication, and division and then you can do novel problems you've never seen before that are extremely different. We are extremely general here because we've learned the causal rule set. It isn't just memorization. This is also true for things like physics, and is literally the point of science. Causality is baked into scientific learning. Of course, it is problematic when someone learns a little bit of something and thinks they know way more about it, but unfortunately ego is quite common. But also, I'm a bit with you. At least with what I think you're getting at. These LLMs are difficult to evaluate because we have no idea what they're trained on and you can't really know what is new, what is a slight variation from the training, and this is even more difficult considering the number of dimensions involved (meaning things may be nearly identical in latent space though they don't appear so to us humans). I think there's still a lot of ML/AI research that can and SHO