A Blog by Jonathan Low

 

Jan 24, 2018

How Fast Is Artificial Intelligence Progressing? Stanford AI Index Reveals Rising Tide



There are really two sets of issues, or tracks, in considering the measurement of AI progress: the first is activity - research, investment, startups.

The second, which is performance relative to expectations, especially versus humans and general intelligence, is harder to capture.

Either way, the growth is unmistakable. The larger question is what that actually means for the economy and for society. JL


Thomas Hornigold reports in Singularity Hub:

Measuring the performance of state-of-the-art AI systems on narrow tasks is easy to do. Where the assessment becomes difficult is in trying to map narrow performances onto general intelligence - because of a lack of understanding of our own intelligence. Progress in AI over the next few years is likely to resemble a gradual rising tide—as more and more tasks can be turned into algorithms and accomplished by software—rather than the tsunami of a sudden intelligence or general intelligence breakthrough.
When? This is probably the question that futurists, AI experts, and even people with a keen interest in technology dread the most. It has proved famously difficult to predict when new developments in AI will take place. The scientists at the Dartmouth Summer Research Project on Artificial Intelligence in 1956 thought that perhaps two months would be enough to make “significant advances” in a whole range of complex problems, including computers that can understand language, improve themselves, and even understand abstract concepts.

Sixty years later, and these problems are not yet solved. The AI Index, from Stanford, is an attempt to measure how much progress has been made in artificial intelligence.
The index adopts a unique approach, and tries to aggregate data across many regimes. It contains Volume of Activity metrics, which measure things like venture capital investment, attendance at academic conferences, published papers, and so on. The results are what you might expect: tenfold increases in academic activity since 1996, an explosive growth in startups focused around AI, and corresponding venture capital investment. The issue with this metric is that it measures AI hype as much as AI progress. The two might be correlated, but then again, they may not.
The index also scrapes data from the popular coding website Github, which hosts more source code than anyone in the world. They can track the amount of AI-related software people are creating, as well as the interest levels in popular machine learning packages like Tensorflow and Keras. The index also keeps track of the sentiment of news articles that mention AI: surprisingly, given concerns about the apocalypse and an employment crisis, those considered “positive” outweigh the “negative” by three to one.
But again, this could all just be a measure of AI enthusiasm in general.
No one would dispute the fact that we’re in an age of considerable AI hype, but the progress of AI is littered by booms and busts in hype, growth spurts that alternate with AI winters. So the AI Index attempts to track the progress of algorithms against a series of tasks. How well does computer vision perform at the Large Scale Visual Recognition challenge? (Superhuman at annotating images since 2015, but they still can’t answer questions about images very well, combining natural language processing and image recognition). Speech recognition on phone calls is almost at parity.
In other narrow fields, AIs are still catching up to humans. Translation might be good enough that you can usually get the gist of what’s being said, but still scores poorly on the BLEU metric for translation accuracy.

Measuring the performance of state-of-the-art AI systems on narrow tasks is useful and fairly easy to do. You can define a metric that’s simple to calculate, or devise a competition with a scoring system, and compare new software with old in a standardized way. Academics can always debate about the best method of assessing translation or natural language understanding. The Loebner prize, a simplified question-and-answer Turing Test, recently adopted Winograd Schema type questions, which rely on contextual understanding. AI has more difficulty with these.
Where the assessment really becomes difficult, though, is in trying to map these narrow-task performances onto general intelligence. This is hard because of a lack of understanding of our own intelligence. Computers are superhuman at chess, and now even a more complex game like Go. The braver predictors who came up with timelines thought AlphaGo’s success was faster than expected, but does this necessarily mean we’re closer to general intelligence than they thought?
Here is where it’s harder to track progress.
We can note the specialized performance of algorithms on tasks previously reserved for humans—for example, the index cites a Nature paper that shows AI can now predict skin cancer with more accuracy than dermatologists. We could even try to track one specific approach to general AI; for example, how many regions of the brain have been successfully simulated by a computer? Alternatively, we could simply keep track of the number of professions and professional tasks that can now be performed to an acceptable standard by AI.

“We are running a race, but we don’t know how to get to the endpoint, or how far we have to go.”

Progress in AI over the next few years is far more likely to resemble a gradual rising tide—as more and more tasks can be turned into algorithms and accomplished by software—rather than the tsunami of a sudden intelligence explosion or general intelligence breakthrough. Perhaps measuring the ability of an AI system to learn and adapt to the work routines of humans in office-based tasks could be possible.
The AI index doesn’t attempt to offer a timeline for general intelligence, as this is still too nebulous and confused a concept.
Michael Woodridge, head of Computer Science at the University of Oxford, notes, “The main reason general AI is not captured in the report is that neither I nor anyone else would know how to measure progress.” He is concerned about another AI winter, and overhyped “charlatans and snake-oil salesmen” exaggerating the progress that has been made.
A key concern that all the experts bring up is the ethics of artificial intelligence.
Of course, you don’t need general intelligence to have an impact on society; algorithms are already transforming our lives and the world around us. After all, why are Amazon, Google, and Facebook worth any money? The experts agree on the need for an index to measure the benefits of AI, the interactions between humans and AIs, and our ability to program values, ethics, and oversight into these systems.
Barbra Grosz of Harvard champions this view, saying, “It is important to take on the challenge of identifying success measures for AI systems by their impact on people’s lives.”
For those concerned about the AI employment apocalypse, tracking the use of AI in the fields considered most vulnerable (say, self-driving cars replacing taxi drivers) would be a good idea. Society’s flexibility for adapting to AI trends should be measured, too; are we providing people with enough educational opportunities to retrain? How about teaching them to work alongside the algorithms, treating them as tools rather than replacements? The experts also note that the data suffers from being US-centric.
We are running a race, but we don’t know how to get to the endpoint, or how far we have to go. We are judging by the scenery, and how far we’ve run already. For this reason, measuring progress is a daunting task that starts with defining progress. But the AI index, as an annual collection of relevant information, is a good start.

30 comments:

Aishu said...
This comment has been removed by the author.
DataScience Specialist said...

Thanks for your post. I’ve been thinking about writing a very comparable post over the last couple of weeks, I’ll probably keep it short and sweet and link to this instead if thats cool. Thanks.
Data Science Course in Bangalore

Tech Institute said...
This comment has been removed by the author.
Data Science Training said...

I read that Post and got it fine and informative. Please share more like that...

Data Science Training

saketh321 said...

This is a splendid website! I"m extremely content with the remarks!ExcelR Data Analytics Course

Anonymous said...

Awesome blog post,
The demand for Digital Marketing is growing big every day. With more and more users coming online, the demand for digital marketing is expected to grow even further.Digital Marketing Course with Internship

Priya Rathod said...

Thanks For sharing Your Information the Information shared Is Very Valuable Please Keep Updating Us
DevOps Training in Hyderabad
DevOps Course in Hyderabad

dhanuesh said...

SAP SD Training in Bangalore
https://intellimindz.com/sap-sd-training-in-bangalore/
IntelliMindz is the best IT Training Institute in Bangalore with placement, offering 200 and more software courses with 100% Placement Assistance.

Unknown said...

cool stuff you have and you keep overhaul every one of us cyber security course in delhi

Data Analytics Course in Dehradun said...

Extraordinary post I should state and a debt of gratitude is in order for the data. Instruction is unquestionably a clingy subject. Be that as it may, is still among the main subjects within recent memory. I value your post and anticipate more.iot course in lucknow

Unknown said...

It is the intent to provide valuable information and best practices, including an understanding of the regulatory process. top cyber security institute in delhi

tech science said...



Really, this article is truly one of the best in article history. I am a collector of old "items" and sometimes read new items if I find them interesting. And this one that I found quite fascinating and should be part of my collection. Very good work! data science course in nagpur

tech science said...


Very good message. I stumbled across your blog and wanted to say that I really enjoyed reading your articles. Anyway, I will subscribe to your feed and hope you post again soon.ai training in jaipur

tech science said...




I bookmarked your website because this site contains valuable information. I am very satisfied with the quality and the presentation of the articles. Thank you so much for saving great things. I am very grateful for this site.cyber security training in jaipur

sankar said...

Thank you for sharing this useful information. Keep sharing this kind of information.sap sd training in hyderabad

Albert Smith said...

I like the gainful knowledge you give in your articles. I'll bookmark your site and check again here habitually. You did a great job and i am very happy to see this because it is very useful for me. Thanks for sharing. The top Dubai App Development company provides good service of app development you visit here site for more info.

Aswi said...

It only takes a few minutes each day to learn and then apply your knowledge. It is highly recommended, the program in chronological order, given that sessions are held. Even the trained and experienced digital marketer will find a lot of valuable advice to create a complete and successful online strategy.
digital marketing course in Bangalore

Aswi said...

Online courses in digital marketing Manage projects in your industry
Digitally tracking each worker's work per day, i.e., managing a construction project digitally. It takes just a few minutes a day to learn this method, then you can practice what you've learned. It is advisable to arrange the program chronologically, since the sessions are held chronologically.
digital marketing training online

casinositezone.com said...

Well I truly enjoyed reading it. This subject offered by you is very helpful and accurate.
토토사이트

salome said...

interesting informations and regular updates. Python training in Chennai

chitra reddy said...

Data analyst course in chennaiThis blog consistently provides valuable content on data analytics, making it a go-to resource for professionals in the field.

crack said...

Malwarebytes Crack With Keygen Key is antivirus software that can detect potentially harmful files by simply matching certain byte patterns. Malware that goes undiscovered can ruin your network and reduce income by causing data loss and network corruption.

Post a Comment