The last invention

[timelets]

Can't wait.

The arrival of superintelligence will clearly deal a heavy blow to anthropocentric worldviews. Much more important than its philosophical implications, however, would be its practical effects. Creating superintelligence may be the last invention that humans will ever need to make, since superintelligences could themselves take care of further scientific and technological development. They would do so more effectively than humans. Biological humanity would no longer be the smartest life form on the block.

--- Nick Bostrom, 2003.

https://nickbostrom.com/views/transhumanist.pdf

I wonder how it feels to be someone's dog.

Comments

[timelets]

one starts with the task of creating an artificial software engineer, and one proceeds towards having a situation where further software engineering efforts can improve the capabilities of that artificial software engineer.

It's been invented decades ago. One of her first names is the complier ;)

[dmm]

An equivalent transformation does not count ;-) Whether it is translation or speed optimization, or anything like that ;-) Those we have indeed mastered quite well long time ago ;-)

But when one can "hire a computer system for a current junior software engineering position at an average company", that will be the threshold for this particular path. The essence of those positions also have not changed for many, many decades. (Basically, when artificial programming systems reach the level of competence in their field which today's self-driving systems possess in theirs, that will be a transition point, and that will be a major revolution (if we can reach that level, that is). From that point the path will be open, for better or for worse (equivalent transformations cannot lead to unlimited self-improvement, for obvious reasons, which is why compilers and speed optimizers are excluded).)

***

Anyway, I found it quite enlightening to re-read Vinge's essay (following a link from his Wikipedia page); as context shifts with time it does read differently each time :-) So, I certainly benefited from this conversation already, because it prompted me to re-read Vinge's essay (and also to meditate on what it was like for him to compose that essay in 1993) :-) And it particular, his emphasis on the hybrid alternatives looks more and more interesting now, and puts the whole "AI safety debate" in a somewhat different light :-) So, anyway, I did benefit from this conversation, it had certainly helped my thought process :-)

I hope you also don't find the overall experience quite empty :-)

[timelets]

It's all good – thanks for sharing your thoughts. I read the Vinge essay a long time ago and maybe it's worth rereading now. Actually, Bill Joy's essay "The future doesn't need us" still strikes me as a more relevant one in the context of the current AI discourse. As much as I can perceive, modern software engineers are already hybrids because they are joined at the hip with supercomputing systems and are paid based on their ability to leverage the immense capital, both tangible and intangible, invested into such systems.

What puzzles me a bit is that people consistently assume that the growing computational power will somehow translate into biological immortality. This smells like the good old fear of death, rather than rational thinking about Singularity, etc. And it gives me hope for the humanity in general because this fear is a consistent theme over the ages.

[dmm]

> modern software engineers are already hybrids because they are joined at the hip with supercomputing systems

Yes... the problem is, the engineers are not getting any help from those supercomputing systems.

Instead, after having considerable progress in terms of the amount of help engineers were getting from the computer systems they were using, this mandatory "going to the cloud" is associated with considerable regress in convenience and engineering productivity. It almost feels that the progress of the first decade of this century achieved in this sense (how much the computer system helps a software engineer) was erased during the second decade, because of how inconvenient those cloud systems tend to be, and how much people are forced to use them... So instead of programming becoming easier, it was getting more difficult again, mostly not for fundamental reasons, but for reasons of various social pathologies (both cloud-related, and of other kinds too).

I should probably re-read Bill Joy...

[timelets]

how inconvenient those cloud systems tend to be, and how much people are forced to use them

It's a typical business problem associated with early stages of a major technology transition. As usual, the new solution succeeds _despite_ all its deficiencies and inefficiencies. It's a sign of strength, rather than weakness.