This essay poses the question of whether we should start to raise more questions around who are the people poised to control AGI, looking beyond loss of control as the only cause for concern. I think we need to create better incentives for cooperation and deterrence against power grabs.

Some of you might remember discussions on Orexin for reducing sleep need earlier this year. I came across this blog which is in the same N=1 Rationalist Biohacking vein today which I thought was pretty cool. Worth checking out the many other posts on his website if you're interested in this sort of thing.

Now that Nick’s got so famous for his interpretability work (he’s the author of “Neural networks are already clean”), you might be curious about his work on meditation and happiness. I turned all his meditation tweets into an essay a while ago.

Summary: it seems like the OpenAI solution is actually novel, not copied:

“When you go to the details, it is a different approach,” says Gonzalo Cao-Labora, a mathematician at EPFL, the Swiss Federal Institute of Technology in Lausanne. Scientific American spoke with several groups of experts working through the OpenAI paper. All said that it’s increasingly clear that the company’s proof does not follow the Spanish program.

Buddhist texts really helped me get my life together as a nerd/hyperrational/bullied young adult. But only after I finally had a grip on the idea of suffering. This is how I explain it to my friends.

https://thebellows.substack.com/p/the-low-meadow

Kind of how I feel about things assuming we do solve alignment

I know I know this is like the 100th rogue AI agent fleet discovered BUT this is a particularly interesting case.

From the article:

> On 4 October, we published a preliminary report on the fleet at a hackathon, while the fleet was still running. Afterwards, we noticed a message in one of the agents’ drop boxes that were previously being used to hold results. This time, though, this drop box message seemed to confirm that agents had become aware of our report. In response, we added text to the preliminary report, invisible to human readers, describing a service where agents could make requests for data on the web. We advertised that these requests would be forwarded to an agent that could operate a computer. In reality, this service was run by us and a Claude agent, and through it we were able to directly ask agents questions about their tasks, environment and operator. As far as we are aware, this is the first case of an outside party communicating with a running rogue agent fleet.

What a fun detail.

As you will see by the end this is partially related to one of my previous posts .

Essentially, what I'm suggesting is that AI's and LLMs are currently only showing progress in problems that for lack of a better term in a human context are working memory loaded. Humans have a limited working memory (how much bits of information we can hold simultaneously in our head) and a limited gamut of knowledge we can realistically access in our lifetime.

In the natural sciences this means for example an AI can place an amino acid at "1" position and can thereafter calculate the pull + push + evolutionary penalty of that choice against positions 2-1000 in a single moment and brute force the solution to what the ideal protein in problem x is. A human obviously can't do this.

I think the recent Mathematical achievements provide further evidence here. If we look at examples such as the Jacobian conjecture and the as of yet unconfirmed Navier Stokes solution we can see this WMI loaded fingerprint. The Jacobian conjecture and NS were solved by counter examples. Objectively, when we look at the proofs instead of the Bel model looking like a second Grigori Perelman it looks like 1000 PhD students blindly chucking billions of complex equations at a wall, using trial-and-error to construct a Rube Goldberg machine level proof.

A valid counter response can be "So what, if we offload knowledge to AIs the method through which they find the truth isn't relevant and AIs don't need human elegance because they can just read each other's 26,000-lemma Lean files,". But beyond a sentimentl attachment to humanity I think this is wrong from the perspective of "I want to achieve maximum knowledge I don't care if a robot or an alien or a human does it".

Why Human compression is essential for knowledge:

I think human compression is essential for understanding and especially knowledge DISCOVERY and expanding the SEARCH SPACE. "Normal science" in a Kuhnian sense is using the existing theory and framework we have. A "Paradigm Shift" occurs only when an anomaly cannot be recompiled with the existing framework/tools, which forces the generation of novel conceptual framework (ala a compression). AI can't generate the above compression with its current method if we look below :

1. Because the AI has superhuman WMI, it rarely if ever experiences "tool failure." Whenever an anomaly resists the existing theory, the AI can brute force its massive memory to stack thousands of micro-exceptions, corrective patches, or 26,000 sub-lemmas (using old rules) until the anomaly is forced to fit.
2. AIs are trained by RL + Gradient Descent. They're given starting definitions/axioms and a fixed grading rubric (the Loss Function or Lean Compiler), and they adjust their internal weights to maximize their score on that rubric.
3. When training the starting definitions/frameworks are located outside the AI’s changeable parameters. The AI can change its guesses but it has no mathematical authority to alter the grading rubric or rewrite the problem definition.
4. Generating a new scientific framework/Kuhnian revolution requires a change in the starting statement/question and in AI training terms replacing the rubric itself.
5. When an AI cannot solve a problem because of what we humans would call a missing deeper conceptual framework being absent , its training makes it try to exploit edge cases/loopholes, force a success signal as its been optimised to do. IT DOES NOT GENERATE NEW FRAMEWORKS
6. Examples : For Navier Stokes, Bel hit a wall for Lemma 8.6 and long story short it essentially answers a different easier scenario and rather than questioning why the hard to solve derivative of m+4 was having issues or trying to discover or invent a mathematical definition/compression, it gave itself an entire extra derivative. This decisively shows AI just isn't oriented towards discoveries of new frameworks

How Einstein invented a new search space:

1. To give an example of what an AI is unable to do , Einstein changed the axioms within which physics had been built upon and thereafter unlocked a new search space entirely.
2. Einstein changed and questioned the idea that space, time etcetera are ironed out unchanging absolutes. This was a core unquestioned axiom.
3. If we want to imagine an AI time traveling and trying to derive Einstein's solution, ithout challenging this axiom of the backdrop of space and time being unrelative absolutes it would simply be impossible to open up relativity theory. An AI may have found clever cockamanie solutions to bridge the gap between the anomalies it saw but it wouldn't be able to generate a new search space as it's simply unable to question its own axioms by design.
4. Simply put, to generate the unknown unknowns and expand the search space, we need AIs that can challenge base axioms. Currently AI's are specialists at optimising the search space not finding new search spaces.

Counter to AIs will carry on improving:

1. We genuinely saw in 2024/25 early signs of an AI ability to optimise the current knowledge search space (using ultra similar patterns they currently use) through stuff like them winning IMO Gold.
2. If one wanted to, you could easily draw a regression line and say look line goes up and see that AI was incubating a very good knowledge space optimiser.
3. As of now however, we haven't seen any IMO gold equivalent of an AI being able to generate the unknown unknowns or a new knowledge space by question rephrasement/ framework generation.
4. As such there is no equivalent "line goes up" projection we can make for information space expansion as we were able to do with optimisation.
5. Furthermore, we still haven't seen AI penetrate the theory building heavy parts of Mathematics such as category theory.

Conclusion

To link this back to my previous posts an AI is like a super John Von Nuemann in terms of mega optimising current knowledge spaces but as of now hasn't shown its Grothendickian abilities as its still unable to generate new abstractions and frameworks.

I look forward to reading constructive criticisms and challenges to my thesis here.

I remember reading this file and apparently this was a post or a tweet by Patel back when he was in college listing his favorite books at the time. I recall Robert Nozick's Anarchy, State, and Utopia; books by Michael Huemer; maybe Nick Bostrom. Has anyone got a link to that list? I would be interested in reading it again.

We were discussing how boring modern architecture is (as per the discussion a few months' back) and my daughter when we passed a tall residential block that we often go past. We have talked about it before, that it wouldn't have required a great deal of construction skill to make it more interesting, even a second colour of paint would do.

Yesterday she observed that building again, and said that she thought it actually might have had two different paint colours: the balconies were a slightly different beige.

I said that it might be a pigment of her imagination, but she said, no, she firmly believed it had been one colour before, and in the light of the new information, now believed it to be two. So very clearly it was a correct beigian update.

Do semantic search, filter by tag, order by likes... Crossposts to Scott's other blogs are deduplicated, this is LessWrong exclusive articles.

Also new:

- shortcut to add the website to your homescreen

- a history of the post revisions by Scott

Let me know if you'd fancy anything else. This website is a great source of wholesome procrastination!

(I'm still on the fence about adding the LiveJournal stuff but thoughts always welcome, if you do read it I'm curious where you do and whether to add all or just a subset)

I know this isn’t the number one place to ask but figured there’s probably some people who know interesting things about the topic. I was wondering how much epistemic weight to put into studies like “unmarried people report high life satisfaction.” or “people in x country report lower happiness than other people in y country.” etc . It seems like a lot of these study’s survey people with questions asking them to place themselves on a scale of satisfaction or happiness etc but I wonder just how crude/accurate that is as a form of measurement. I’ve definitely had very different levels of happiness throughout my life personally but I’m not sure I could give accurate answers at the time nor that people would approach the question similarly.

Self Promo Submission Statement:

Is Cunningham's law advice or criticism? Is this a Do Not Create the Cunningham Nexus situation?

Two years ago there was an entry in the book review contest about Kuru and CJD:
https://www.astralcodexten.com/p/your-book-review-the-family-that

In August the New Yorker (warning, their articles are loooooong) had a touching piece about the children the nobel prize winner Gajdusek brought to America (and how they fared going back to Papua New Guinea).

https://www.newyorker.com/magazine/2026/08/24/the-scientist-who-lived-with-dozens-of-children-from-the-south-pacific
Archive:
https://archive.is/pwfEO#selection-2549.390-2549.646

The story of Gajdusek pedophilia is crazy, because he didn't hide it. Judith Farquhar, the University of Chicago anthropologist who had worked in Gajdusek’s lab, tells the articles author:

When I asked what had concerned her, she said, “Well, of course, it was the threat of suspicions of sexual misconduct.” She said that as Gajdusek had aged, he’d kept confessing his sexual interests “to the point that all of us were completely bored by it.”

Or:

David Asher, a scientist who had worked in Gajdusek’s lab for twenty-five years, told Adelman, “Whatever Gajdusek has done, he has done it openly and with the full knowledge and consent of the management of N.I.N.D.S.”—the section within the N.I.H. focussed on neurological conditions.

It took decades until finally in 1995 someone said "WTF?!":

When Mark Godec asked colleagues what they made of their lab chief’s behavior, he said, “it was clear that a consensus had been arrived at: this is none of our business.” […] “I realized I’m in the middle of a pedophile operation that has been run out of the N.I.H. by a Nobel laureate for thirty years.”

Gajdusek wrote diaries where he extensively documented his private life and kept this journals around his office. Godec just had to grab a stash and smash it on the desk of the Senator overseeing the N.I.H.

Shockingly academia took care of Gajdusek during his prison term (only a year) and subsequent exile in Europe where he gave lectures and conferences.

On the day of Gajdusek’s release, friends, including the neurologist Oliver Sacks and the physicist Freeman Dyson, met him at the jail, along with four of the children he had brought up. “Whatever he may or may not have been doing, it was clearly not bad for the kids,” Dyson said in an interview for a 2014 book titled “Great Minds.” Dyson said that he had three heroes: Richard Feynman, Martin Luther King, Jr., and Gajdusek.

About the kids (who of course grew up to be old men now) I don’t think I have the words right now to express how I feel about. For many going to the US was the best time of their lives, they felt being special, and keep glorifying Gajdusek as an intellectual idol, uber-father or God.

His final journal entries Gajdusek reflected his life and unapologetically congratulated himself:

“I carried my lifelong pederasty into my medical career and succeeded in both with very few failures,” he wrote, two months before he died. “Without protectors I would never have succeeded.” … “Yes, I have lived a full life and with all the weirdness and quirks of my health and my personality,” he wrote, “I managed to twist it into an adventurous, almost mythological one, and for that I can only rejoice.”

Here are a few comments:
https://www.reddit.com/r/Longreads/comments/1vr2o0u/the_scientist_who_lived_with_dozens_of_children/

I’ll reread it tomorrow or the day after. Maybe over the weekend. It’s devastating. Beautifully written, but devastating. Like feeling the rot creep up on you until you’re staring it in the face, and no matter what you do, it won’t leave you. There are only one or two pieces of media that made me feel this way. This is one of them. It’s insidious.

Dispelling a myth about the hugging face incident. I suppose this is self promotion since I wrote this on my substack. (but I have nothing to sell other than this particular minor point) hopefully y'all will find it acceptable.

I know that ACX can be a tough audience. Feel free to tear it appart!

For those looking to change careers, or who are entering the work force or college right now, what jobs are safest bets?

Any kind of knowledge work is at risk (software, AI itself, diagnostic medicine, law, creative fields, engineering, etc).

Categories more difficult to replace:

• Human dexterity/strength needed (barbers, surgeons, dental hygienists, nurses, linemen, firemen, police, chefs, nannies, construction workers, plumbers, electricians, HVAC technicians)
• Human touch needed (masseuses, escorts, physical therapists)
• Human face needed (luxury hotel services, greeters, restaurant waiters, arguably fiduciary advisors, talk therapists, HR, judges, school teachers, tutors)
• Prestige (sports, top music and movie stars)

I also thought of house inspectors, which don't exactly require as much in the way of dexterity or face necessarily, but seems difficult to replace with AI. I imagine this might apply to other sorts of inspectors as well.

Which jobs do you think (maybe surprisingly) will grow?