Building on that idea in The Human Use of Human Beings, [Norbert Wiener] argues that, once set in motion, machine learning might advance to a point where — “whether for good or evil” — computers could be entrusted with the administration of the state. An artificially intelligent computer would become an all-purpose bureaucracy-in-a-box, rendering civil servants obsolete. Society would be controlled by a “colossal state machine” that would makes Hobbes’s Leviathan look like “a pleasant joke.”
What for Wiener in 1950 was a speculative vision, and a “terrifying” one, is today a practical goal for AI-infatuated technocrats like Elon Musk. Musk and his cohort not only foresee an “AI-first” government run by artificial intelligence routines but, having managed to seize political power, are now actively working to establish it. In its current “chainsaw” phase, Musk’s DOGE initiative is attempting to rid the government of as many humans as possible while at the same time hoovering up all available government-controlled data and transferring it into large language models. The intent is to clear a space for the incubation of an actual governing machine. Musk is always on the lookout for vessels for his seeds, and here he sees an opportunity to incorporate his ambitions and intentions into the very foundations of a new kind of state. It’s preformationism writ large.
If the new machine can be said to have a soul, it’s the soul Turing feared: the small, callow soul of its creators.
Even more than a flesh-and-blood bureaucracy, Wiener understood, an inscrutable bureaucracy-in-a-box, issuing decisions and edicts with superhuman speed and certainty, could all too easily be put to totalitarian ends. The box might seem autonomous, its outputs immaculate, but it would always serve its masters. It would always be an instrument of power. “The modern man, and especially the modern American, however much ‘know-how’ he may have, has very little ‘know-what,’” Wiener wrote. “He will accept the superior dexterity of the machine-made decisions without too much inquiry as to the motives and principles behind these.”
To explain his distinction between know-how and know-what, a distinction he saw as critical to the future course of technological progress, he tells another story:
Some years ago, a prominent American engineer bought an expensive player-piano. It became clear after a week or two that this purchase did not correspond to any particular interest in the music played by the piano. It corresponded rather to an overwhelming interest in the piano mechanism. For this gentleman, the player-piano was not a means of producing music, but a means of giving some inventor the chance of showing how skillful he was at overcoming certain difficulties in the production of music. This is an estimable attitude in a second-year high-school student. How estimable it is in one of those on whom the whole cultural future of the country depends, I leave to the reader.