Intelligence Explosion

Recently I've been asking for help battling against an overwhelming complex of science-fictiony beliefs that I'm coming to hold. I hate having these beliefs, mainly because they feel uncannily like what I'd imagine a new religion must feel like to an early believer. It's very much not compatible with my self-image to fall for a millennial cult, and I trust that feeling.

So I've asked various friends to have a look at this belief-complex, in the hope that they can point out where it's a load of rubbish.

I'd actively like to be ridiculed here.

I once started feeling all spiritual after a particularly pleasant evening in King's College Chapel listening to Bach, and a few minutes of ridicule was enough to dispel the feeling and save me from the hideous mind-virus that is Christianity.

Unfortunately, the point in this set of ideas that people are consistently choosing to attack is the idea of an intelligence explosion. The same friend who saved my soul from Christ just looked at the idea of an intelligence explosion and said 'Nah, can't happen', and then stopped thinking.

The possibility of Intelligence Explosion is something that I've been worried about for many years, and it's not the bit that feels religious. It just feels like a threat that ought to be taken seriously. A bit like 'Earth might be Hit by an Asteroid', or 'Global Warming', or 'Nanotechnology Eeek!' or 'Nuclear War', or 'Flu Epidemic', or 'Worldwide Dictatorship', or 'Surveillance State', or 'Terrorists Who are Actually Good at it Rather than Mediaeval Halfwits with Towels on they Heads'.

i.e. It's a horrible existential risk, but there's bugger all I can do about it, and it's fun to think about, and it probably won't happen in my lifetime anyway.

I don't have children, I don't plan to, and although I care about you and your children, and hope for immortality myself, I don't care/believe enough to mean that I can't think straight about these problems and look them in the face.

And the idea of an intelligence explosion is just obvious in retrospect, isn't it? Like the idea of a chain reaction, or the infinity of the prime numbers. I'd never have thought of it myself, but once someone points it out, your head would have to be way broken not to take it seriously.

And it occurs to me that if I think it's that obvious, and I think that I am good at explaining ideas, then I should be able to make it obvious to other people. And so I'm going to try.