One of my main interests within the realm of the digital revolution has been the progression in the craft of software engineering. The craft has moved to a higher abstraction level over the past, say, 70 years. Broadly speaking from machine code to assembly to C-like languages to OO-languages to the current crop of high level scripting languages. Although there have been many variations along the way, the main trend has been that the craft of programming moved to higher levels of abstraction. Instead of handwriting zeroes and ones software engineers now have languages and tools that allow them to specify for instance classes and inheritance and let the compiler turn that into machine executable instructions. As I've written before one of the questions that have been puzzling me over the past few years is why this progress seem to have stalled in the 90's. At that time the industry had quite a lot of experience with higher level languages such as COBOL (!) and there was a lively discussion surrounding 5th generation programming languages. But then the web happened and all attention was directed at writing as much software as possible to explore this vast new realm of undiscovered opportunities. Somehow that seemed to lure all attention away from fundamental progress towards the latest greatest in web development. Everybody was in a hurry to explore this new world and technologies were invented, and often reinvented, at neck-breaking speed. Today's most widely used programming language, JavaScript, was for instance developed in just 10 days at that time.
There is a problem with this stagnation, though. The demand for new software is rising so fast that there simply aren't enough humans to do the job if we stick to the labour intensive abstraction level the industry is stuck at at the moment. There is already a huge shortage of software engineers. It is actually one of the main limiters of growth in Western industries.
But there is another problem. The low level work is prone to errors while at the same time the importance of the information systems for society at large is increasing. In other words, our lives depend on software in an accelerating rate while at the same time we ar developing the software in an archaic and error prone way. The only way out of this situation is to move the job of software engineering to the next level of abstraction. This means that domain experts should be able to express exactly what they want after which a compiler generates the information system that does just that. This has been called 'semantic programming' in the past, and it is in my opinion the only idea to move the software industry forward and prevent a lot of problems. Fortunately there is light at the end of the tunnel. In a recent post I mentioned my idea that AI seems to be the surprising candidate enabling the step towards the next level of abstraction. While in traditional information systems developers tediously wrote detailed instructions how to mangle given input to generate the required output, with AI developers just specify the required output based on a certain input and let it figure out the intermediate steps itself. This is in line with the ideas of semantic programming.
Interestingly enough there is also light at the end of another tunnel that has strikingly similar properties. After diving into the world of computer graphics (CGI), game development, AR/VR, and animation I realised that this was an industry where the exact same type of transformation was taking place. Early CGI work consisted of painstakingly typing out coordinates to, for example, draw a wireframe of a doll. Subsequently tools were developed to allowed artists to draw with a mouse and automatically tween between keyframes. Over the past decades the tools became increasingly more sophisticated, especially in the area of the notoriously complex world of 3D CGI. Each step gradually freeing the artist from tedious, repetitive tasks and allowing them to focus on expressing what they actually wanted. Interestingly enough one of the biggest trends in the CGI industry is the move towards proceduralism, where things like textures, geometry and even complete worlds are generated by procedures instead of by hand. Take a look at how the 3D modelling (if you can still call it that) software Houdini has been used to procedurally generate environments for the latest episode of the Farcry series. The artists of Farcry no longer have to draw every leaf or every road by hand. they specify the properties of their required worlds at a high level after which the algorithms of software like Houdini generate it. You could say that software like Houdini is becoming the compiler, just as AI is becoming the compiler for information systems (as previously discussed). The drive towards proceduralism in the CGI industry is the wish to focus on the high level picture and not on the low level details. But also by the need to create increasingly more complex worlds for which it would be impossible to build by hand.
I find this parallel intriguing. (Somehow my brain is wired to relentlessly look for parallels between separate phenomena.) Understanding the underlying driver enables us to see where our industry, work and tools ar heading. And it gives the creative mind a glimpse into the future.