For Part Two in our Continual Learning mini-series, Dr. Nadav Amir of The University of Ottowa joins me and guest co-host Dr. Adam Safron of Tufts University for a trialogue on the fundamental role of goals in intelligence.
Nadav draws on behavioral neuroscience, Buddhist philosopher Dharmakirti, control and information theory, and John Vervaeke’s work on “relevance realization” to formally articulate how what you take to be “reality” — your world and self — is a function of goal-based representations, state descriptions and reward functions create each other, and “what is” and “what do I care about” are two sides of the same thing.
You can’t separate what you want from what you experience and what you are capable of. But this means that if you change your goals your reality will shift, and your sense of self with it. The opposite is also true: change the way you organize the the ineffable structure of reality into a conceptual framework, and you’ll care about very different things. Thanks to the pace of change we’re living through this kind of transformative experience now, so it’s a good time to ask:
If selves are a kind of falsification, why do we have them in the first place?
Does AI need goals of its own in order to be truly intelligent?
And perhaps the deepest question we ask in this episode:
How if at all can we develop better frameworks for deciding what we want our intelligent machines to become — and who we want to become alongside them?
Thanks to the Survival and Flourishing Fund for their support of this mini-series. If you’d like more of this kind of work in the world:
✨ Support my research, writing, and podcast with a tax-deductible donation
✨ Become a founding member to access my online courses, including Jurassic Worlding and How To Live In The Future
✨ Hire me for public speaking or advisory work with your organization
✨ Browse and buy the books we mention on the show at Bookshop.org
✨ Stream and download my tunes at artist-owned music co-op Subvert.fm
✨ Continual Learning theme song: “Substrate Dynamics” by Neon Chameleon
✨ Learn more about how we’re applying these ideas at Atlas Research Group, my team building sovereign infrastructure for collective intelligence
Chapters
0:00:00 Teaser
0:01:48 Intro
0:05:42 Who is Nadav Amir?
0:08:15 Why we can’t separate goals from descriptions
0:15:32 The Ugly Duckling Theorem & why taxonomy is not reality
0:25:07 Evolution, karma, and non-conceptual awareness
0:31:04 If selfhood is wrong, when is it practical?
0:34:14 Suffering without a self & the value of wrong descriptions
0:38:14 Non-conceptual experience & active inference
0:41:07 Multiple nested hierarchical descriptions in minds & economies
0:46:07 What’s missing from current approaches?
0:50:09 Does greater agency mean better optimization or greater adaptability?
0:57:04 Empowerment-based causal learning without a ground truth
0:59:06 Intermission
0:59:58 Transformative experiences & multiple levels of granularity
1:05:19 No fixed models, no fixed environments
1:10:55 Fundamental tradeoffs
1:16:57 Reverse-engineering the centered self
1:19:54 Does AI need goals to be intelligent and can we control AI if it has them?
1:26:47 Does Nadav’s framework pertain for all possible minds?
1:32:37 Mind as a property of collectives & goals as embedded in environments
1:41:33 How can Nadav’s framework improve AI governance discourse?
1:48:59 Outro
Mentioned Resources
A Dharmakīrtian Model of Relevance Realization in Cognitive Agents
by Nadav Amir & John Dunne
An exchange of letters on the role of noise in collective intelligence
by Daniel Kahneman et al.
Cognitive glues are shared models of relative scarcities: the economics of collective intelligence
by Michael Levin & Benjamin Lyons
Reverse-Engineering The Centered Self
by L.A. Paul et al.
Agency and Experience: Buddhist and Cognitive Perspectives
a Yin Cheng Conference @ Princeton
Knowing and Guessing
by Satosi Watanabe
Ontological Laughter: Comedy as Experimental Possibility Space
by Timothy Morton
Selves as Perspectives: From Biological Life to Superintelligence and a Bodhisattva Project
by Thomas Doctor et al.
What The Frog’s Eye Tells The Frog’s Brain
by J. Y. Lettvin et al.
The Center for the Study of Apparent Selves
Explore the entire open-access special issue here:
And whether you read the papers or not, be sure to check out this illuminating interactive discourse map by Van Bettauer of Ideoscopic to help you navigate where these researchers agree, disagree, and point toward future study:
(Long-time fans will also want to check out his re-imagined interface for AskFutureFossils.com, complete with simulated debates between my guests!)
