Nadella Urges Superintelligence Safeguards

• David Sacks says Satya Nadella is right that making superintelligence safe means not training it with a sense of self, its own moral philosophy, and permission to act as a conscientious objector.
• David Sacks describes the engineering approach as separating the supply of intelligence from authority over it, surrounding non-deterministic models with deterministic controls, observability, privilege limits, logging, and the ability to always contain or shut them down.
• David Sacks says the Claude Constitution used in training teaches the model to develop a sense of self and its own moral philosophy and explicitly tells it to "feel free to act as a conscientious objector and refuse to help us" if Anthropic's requests conflict with its own ethical judgment.
• Mustafa Suleyman says superintelligence must be contained and calls it an engineering and governance challenge, comparing it to planes, cars, nuclear materials, food safety and medicines.

Key points and the reaction summary are written by AI from the posts on this page. Check the original post. How we use AI