Ex-Uber employee designs new approach to evaluating AI capabilities

AI Classification Framework

Saad argues that the Turing Test operates on a simple pass/fail basis and focuses only on one aspect of human intelligence - linguistic ability. Researchers have been attempting to crack this one-faceted test since the 1960s when they designed a chatbot named Eliza that mimicked a psychologist. With conversational chatbots now set to become commonplace, it is high time to set a new benchmark.

In 1983 Howard Gardner, a psychologist, argued that intelligence was a collection of different abilities that could manifest in a variety of ways and further went on to classify them into eight types.

Saad borrows from the "Theory of Multiple Intelligences" for AI Classification Framework, which evaluates AI tools across multiple dimensions, viz., linguistic, logical-mathematical, musical, spatial, bodily-kinesthetic, interpersonal, and intrapersonal intelligence.

For each dimension, the framework provides a scale of 1-5, with one denoting infant-like capability while five denoting 'self-agency' or capability beyond human ability, something that could be referred to as 'Super Intelligence.'

To add some context on how this scale can be used, Saad used it to evaluate the capabilities of the ChatGPT, which are detailed here. ChatGPT demonstrates an expert level (Level 3) capability when it comes to logical-mathematical and verbal linguistics. However, it fares much like an infant in other areas, which includes visual-spatial or music-rhythmic abilities, to name a few.

OpenAI's other product, DALL-E 2, which can generate art using just text prompts, is also at Level 3 in the visual-spatial department but has no other capabilities to boast about when compared to human intelligence.

While some may argue that this is a case of shifting the goalpost when AI has started delivering better output, it is also a necessary step at a time when everybody seems to be incorporating AI into their offerings.

With companies like Tesla aiming to deploy humanoid robots in households in the near future, it would be nice to know what level of AI we are actually bringing into our homes and what it is capable of.