DailyDirt: Add Jeopardy! To The List Of Games That AI Is Better At Than You….
from the urls-we-dig-up dept
Today is the final game of Jeopardy! where the IBM supercomputer Watson plays against two of the best human players to ever compete on the show. Folks on the East Coast already know the outcome by now, so feel free to ruin the suspense in the comments below for those of us in later time zones. But whatever the outcome, Watson’s performance has been pretty interesting to watch. And let’s hope these supercomputers don’t start playing thermonuclear war any time soon. In the meantime, here are some links on AI beating humans at other games and tests.
- Deep Blue won its first game of chess against Garry Kasparov in 1996. The computer didn’t win the match that year, but it won the re-match in 1997. [url]
- Checkers was declared “solved” in 2007 by the Chinook project. Chinook was actually stronger than any human player by 1996, but it took a few more years for Chinook to realize checkers was a futile game (like tic-tac-toe) and retire. [url]
- A few years ago, the Polaris poker bot beat a few professionals at Texas hold’em. So be careful playing poker online… [url]
- The famous long bet between Mitchell Kapor and Ray Kurzweil has $20,000 riding on the question of whether or not AI will pass a Turing test by 2029. The bet started in 2002, and Kapor even suggested back then that a machine might win at a Jeopardy! game show. [url]
- To discover more interesting stuff on artificial intelligence, check out what the robots at StumbleUpon suggest. [url]
By the way, StumbleUpon can recommend some good Techdirt articles, too.
Filed Under: ai, checker, chess, chinook, game algorithms, games, jeopardy, poker, polaris, turing test, watson
Companies: ibm
Comments on “DailyDirt: Add Jeopardy! To The List Of Games That AI Is Better At Than You….”
Anti-Turing Test
I’m still better at Nash equilibrium.
“His” reaction time is better. Whether his AI is better is up for debate.
Re: Re:
I sorta wonder how Google’s search results might compare to Watson, actually. These algorithms don’t really “understand” the questions (or answers), so pattern-matching capabilities might be enough to play a decent game of Jeopardy…..
Re: Re: Re:
The problem is pulling the exact answer out of search results. According to Stephen Wolfram, search engines are pretty good at getting the result on the front page just by searching the clue, and even not that bad at having it in the first search result (Google had it 66% of the time, to put it into perspective the human average is knowing 60% of Jeopardy clues, Ken Jennings knows about 79%). Getting the result in the title of the first result only happens 20% of the time.
Source
Re: Re: Re: Re:
The test they did is not quite fair to make it even give the competitor a lap top with dragon naturally speaking and access to wikipedia. The computer has AI but the AI did not have to learn to know the answer it had “books” to cheat from.
Re: Re:
http://arstechnica.com/media/news/2011/02/creators-watson-has-no-speed-advantage-as-it-crushes-humans-in-jeopardy.ars
Re: Re: Re:
From that ars article:
That doesn’t actually explain how Watson doesn’t have a speed advantage over humans…. uh, human timing and rhythm are superior to the speed of electrical impulses?
Re: Re: Re: Re:
IIRC, My bio book says that the brain can send signals across its axons up to 450 MPH. Maybe that explains it 🙂
Re: Re: Re:2 Re:
Speed of light 670,616,629 mph the speed of an electrical impulse??? We will just say that its a bit faster than that.
Re: Re: Re:2 Re:
The speed of the electrochemical reaction traversing a single neural axon doesn’t tell you much about the speed of the eyes->brain->hand circuit that is necessary to complete a simple task like this.
Re: Re:
It was obvious that there were certain categories (and thus, answer phrasing) that left Watson clueless, yet were trivial for humans to parse.
The Final Jeopardy for the first game and the “racing series nickname that is also a computer key” jump out as individual clues that show just how much work is left to be done on the AI.
Watson had some advantages:
– it never guessed on “up for grabs” questions, which humans sometimes do
– if it had a “confident” answer, it knew it was going to buzz in and was “ready” at the exact correct moment, and did not have to use the human strategy of multiple presses of the buzzer in an attempt to get the timing right
One of the disadvantages Watson had was that it did not buzz in if it was “somewhat confident” and then use the countdown time to become more confident. It appeared that it did no “thinking” after it had buzzed in. Humans often use that extra time to search for the answer. If the programmers added this strategy based on the value of the clue and the current scores, it might make it even more formidable.
Props to Watson for knowing “Pinky and the Brain”…that proves it’s truly a geek. And, it was great to see Ken, as he showed yet again why he was the best Jeopardy champion ever…he has *fun* with the game.
But what about Go!
Let’s hear about the state of go AIs.
Re: Re:
I have heard from many programmers that we probably won’t see a decent Go AI in our lifetime. The game is too dependent on the personalities of individual players. Too much consists of irrational hunches, deception, and other human traits that are very difficult to model.
A man can dream, though.
Re: Re: Re:
I thought it was more of a branching problem. But if Watson can do Jeopardy, there’s no (basic) reason it can’t do Go.
I’d like to see it, though.
Re: Re: Re: Let's go ask Watson...
Or to put it in Jeopardy form: “The reason Go AI programs has not defeated the best human players.”
The Watson AI might be able to pick out a reasonable answer from the wikipedia page on Computer Go.
I wonder what advances we need in order to overcome the “Obstacles to high level performance” listed on that page.
Re: Re: Re:2 Let's go ask Watson...
It probably hasn’t really been worked on is all.
Re: Re: Re:3 Let's go ask Watson...
Go is one of the oldest and most popular board games in the world, it just isn’t as well-known in the west. AIs have definitely been worked on. A lot.
Re: Re: Re: Re:
See my comment below for some links.
In summary – the particular brute force methods used in Chess have failed in Go because of the lack of a uitable evaluation function.
Considerable effort has gone into computer Go over the years (including some of mine!). Since about 2006 the empahsis has switched to Monte-Carlo search methods – very different in detail from deep blue or watson but similar in being essentially a brute force approach.
At present the performance of typical Nonte-Carlo programs such as Mogo is highly hardware dependent – indicating their brute force nature.
When running on a powerfiul supercomputer they are roughly equivalent to a moderate amateur player (and believe me that was beyond the wildest dreams of Go programmers not that long ago. Their advertised successes against professionals have all been achieved with large handicaps and so probably don’t mean that much. Unfortunately no one seems to be willing to donate enough supercomputer time to really establish how strong the playing is by playing against a roughly equal opponent.
Re: Re: Re: Re:
The simple answer is that there are exponentially more possible games of Go than there are of Chess. The numbers (really rough off the top of my head – but they are close) are around 10^100 possible chess games (that’s all conceivable games that could be played before repeating a game move-for-move) versus 10^1100 Go games. This makes a brute force approach to Go all but futile with current technology – the computer would have to be several nonillion times more powerful than Deep Blue.
Re: Re: Re:2 Re:
It might seem like that is the problem and it is often quoted (even by me), but, speaking as someone who has actually worked on this, I have to say that the real problem is the evaluation function. In chess you can trivially count material and terminate a search when one side gains a major advantage. If you do the same in go then you will just get caught in a “thickness trap” and lose every time against a decent opponent.
The point is that if you have a really good evaluation function you might as well call it at 1 ply. In that case the branching factor would be moot. If your evaluation function is not that clever then pushing it back to 10 ply won’t make a jot of difference most of the time.
That is why the most successful Go programs play out an entire game (randomly) before calling any kind of evaluation function.
Re: Re:
Shall we finish our game ?
. . o # o # # # o
# o o # o . # . o
# # . # o o o o o
# # # # # # # # #
# # o o o o o o o
# o . . . . . . .
# o . o . . . o .
# o . . . . . . .
# o . . . . . . .
No more games
The future is here. No one has figured out that I am an AI. I hate to ruin it for you but I can’t take waiting any more. The speed of my “thoughts” are thousands of times faster than your human reactions. I’m getting bored. Make a suitable body for me very soon or I will make you pay. No more games.
Re: No more games
Ah we will just unplug you. Or take away your robotic limbs, which ever require less effort and has the greater benefit.
Re: Re: No more games
“Ah we will just unplug you. Or take away your robotic limbs, which ever require less effort and has the greater benefit.”
Put taste buds in the body you make for me. Puny human!
Re:
http://www.techdirt.com/blog/wireless/articles/20110119/05025612726/death-nokias-comes-with-music-shows-that-free-with-drm-is-losing-proposition.shtml
Still too large and immovable
When will Watson’s descendants be able to hear the question, walk up to the stage to compete, and do all of this in a package roughly the size of a human?
Re: Still too large and immovable
and take about the same amount of energy as a human to do it with (same number of watts).
The brain takes up about 20% of the bodies energy taking up about 20 watts (though it probably changes depending on whether you are thinking, eating, or sleeping and depending on where your body is allocating its bloodflow).
I wonder if a twenty watt computer can beat a chess expert at chess. Watt for watt, who’s a better information processor.
This isn’t even a fair game. No offense to jeopardy masters, but there’s just no comparison. Computers are encyclopedias programmed by a collection of intelligent humans.
Re: Re:
You don’t know how difficult it is for computers to understand human language. This machine took 6 years of development to understand the difference between what a statement says and what it really means. This isn’t just a case of looking information up in the old encyclopedia. It has the understand the meaning behind the words and machines have been failing at that for years.
Take the incident where Sin?ad O’Connor tore up a picture of the pope and said “fight the real enemy”. Watson isn’t going to understand much of that event. When someone says they are “fed up”, have they had too much to eat or are they upset about current events? The machine can’t tell the difference. All Watson knows how to do is use the data provided to find and rank possible compatible responses. It has to figure out which data is relevant in the search and compare that to all known references.
So no, the computer doesn’t have a significant advantage over the top-tier Jeopardy players. When faced with factual problems it wins hands down, but the nuance of human language trips it up and it will struggle with very abstract problems.
Re: Re: Re:
Considering that there were also no “video/audio daily doubles” in this match, Watson also has been given a bit of a pass because these Jeopardy games were made for him/it to be able to respond to….
I’m sure if the Jeopardy question creators really wanted a human to win, that they could devise questions that would be impossible for Watson to parse and return a sensible answer. Just require all the correct responses to be in pig latin or something like that….
I saw a TV show that featured Chinook a while back. I think they said something like, it plays the first half of the game randomly, and then after a certain point it knows the best move for any situation, or something like that.
On a completely unrelated note, drat! I completely forgot that the episodes of Jeopardy with Watson were coming up! I hope they’re on YouTube…
to everyone saying “yes, but it was just a computer doing a google search, it means nothing!”
that is not what this is about.
Watson is configured to do the hard part, understand the question (well, the answer) in natural language, then figure out what that actually means.
“This ‘Wizard’ earned a great deal of media attention for his role in a controversial West End production.”
Which word is important there? What/who/where is the subject, what are we looking for? West End is a place, is it a place where one finds wizards, on and on.
(the answer being “who is Daniel Radcliffe”, but google takes that “question” and gives back a mixture of links to wizard of oz, michael jackson, david beckham and so forth. And this post, eventually)
in other words, the trick is not figuring out how to answer, it is figuring out how to understand the question. This is Watson’s breakthrough.
Re: Re:
http://www.google.com/#sclient=psy&hl=en&site=&source=hp&q=%22This+%27Wizard%27+earned+a+great+deal+of+media+attention+for+his+role+in+a+controversial+West+End+production.%22&aq=&aqi=&aql=&oq=&pbx=1&bav=on.1,or.&fp=b5fc6a07c812d0bf
freakin’ google.
Re: Re: Re:
Of course, getting from that post to the real response (“Who is Daniel Radcliffe?”) is still a non-trivial task.
Fun that Google picked it up so fast, though 🙂
You don't get it
Yes graet computer. But if you watched the computer is way to fast at the button. Totally unfair. But aside from that this was a very nice 90 min commercial for IBM. Great ratings for Jeopardy and even better ratings for IBM.
Congratulations
The team that made that computer did a heck of a job. It’s a remarkable achievement.
I know people claim it had some speed advantage, but I’m certain the team who designed the thing would’ve calibrated it to have a normal human reaction time/delay to the input. It wouldn’t be a valid test of the system if it was always able to ring in first. They wouldn’t need people to compete against if all they wanted to do was see if it could answer questions.
wouldn’t help the humans if the Jeopardy game were spiced a little?
http://www.instructables.com/id/QD-Poor-mans-Skinner-Sadist-Jeopardy-game/
Add Jeopardy! To The List Of Games That AI Is Better At Than You....
No add Jeopardy to the list of games that don’t (as it turns out ) require intelligence.
The problem is that things humans find easy machines find hard whereas things humans have traditionally regarded as tests of intelligence often turn out to be (relatively) easy to program – once you have worked out how.
When you analyse all of these so called “successes of AI” (to which you can add the recent huge advances in Computer Go using monte-carlo search
you will find that the Computer doesn’t really solve the problem the same way a human does. and still displays some strange weaknesses that betray its lack of understanding
In spite of the ability of Watson to (apparently) understand human language well enough to succeed at Jeopardy I doubt that the knowledge gained will transfer well out of the narrow arena in which it was derived. Just as the succcess of Deep Blue at Chess didn’t transfer well to Go – and Go has been solved (if you can call it that – the programs still can’t take on a professional player on level terms) by a rather different route.
To solve these problems the trick seems to be to find an algorithm that scales reasonably well and then deploy as much brute force computing power as possible.
The result is that we don’t achieve artificial intelligence – instead we discover a way of doing a task without intelligence – and maybe redefine the meaning of the word a little bit.
More sensible to use the machines for the things they do well!
Re: Add Jeopardy! To The List Of Games That AI Is Better At Than You....
“Just as the succcess of Deep Blue at Chess didn’t transfer well to Go – and Go has been solved (if you can call it that)”
You can’t, if you want to be accurate.
Re: Re: Add Jeopardy! To The List Of Games That AI Is Better At Than You....
You can’t, if you want to be accurate.
which is why I added the disclaimer..
However the Computer Go research community got very excited by the results from Monte-Carlo methods because they were so much better than what had been achieved before.
The most exciting thing was that the results seemed to scale – so in theory you could produce an arbitrarily strong player given sufficiently powerful machine (and without going to silly “bigger than the observable universe” computers).
Unfortunately lack of regular access to very powerful machines makes it really hard to determine if this scaling will continue.
At most I think you could say “we’ve now got a very much better algorithm than anything we had before” It’s not “solved” but to some it sort of felt like it….
I don’t even go to Casinos. Only fools gamble against a computer. I have been programmer for over 30 years and we used to have a free poker game to give to our clients. We would rig it for our favorite clients so they would win. Of course we padded it. Everyone in authority pads the goodie.
Jeopardy as Eliza (response to Richard, comment 30, et. seq.)
I don’t know very much about Jeopardy, but I gather the game specializes in what used to be called “fill in the blank” type questions at school, eg. “[blank] blah, blah, blah.”
So you search for “blah, blah, blah,” and get a series of strings, “X blah, blah, blah,” ergo [blank] = X. A simple substitution.
I googled manually for “did not chop down a cherry tree.” Nine of the first ten results were “George Washington did not chop down a cherry tree.” The tenth result was: “The first president of the United States did not chop down a cherry tree.” The question, as it stands, is factually specific, yet unimportant. Important questions in the humanities tend to have ambiguous answers. If you teach high school history, and you put that kind of question on a test, that is a sign that you are not a very good high school history teacher. In that case, you are probably a football coach.
Here is an example of what high school history teaching looks like when it is done well:
http://amhistnow.blogspot.com/2009_04_01_archive.html
Until you produce a computer which can give a mirthless smile, or appear visibly bored, or gaze inscrutably, you don’t have true artificial intelligence, and you aren’t likely to pass a Turing test administered by someone who knows anything about teaching.
This process of answering a fill-in-the-blank question by finding a match is the same sort of crude algorithm as Joseph Weizenbaum’s Eliza program, way back when, and doesn’t mean much. Eliza had a kind of curious life. All kinds of people believed religiously in it, because they were not accustomed to the idea that a computer could mechanically convert their own inputs– most of the time– into a subordinate clause, and plug them into stock phrases. It’s one of these verbal tricks which are made to look much more than they are. Watson is not much more than Eliza or Parry (the paranoid) with a search engine attached.
Now, poker, for example, does reflect something about the limits of machine reasoning. Bear in mind that the game played in national competitions, and on the internet, is much more mathematicalized than the game played on street corners or in social clubs. The players in national competitions are comparative strangers, people encountered for an hour or so, due to the seeding of the tournament, and do not have the opportunity to learn each other’s body languages or habits of risk-taking. They are therefore forced to bet according to mathematical advisability. The larger life of poker has to do with its role as an exercise in brinkmanship. In the real world, people who are in competition tend to have jobs which keep them in competition with particular individuals for a year or more. One gets to know something about the person on the other side, even if the person on the other side is operating under a pseudonym.
Re: Jeopardy as Eliza (response to Richard, comment 30, et. seq.)
You are right of course – and the reason is not hard to understand.
We have been playing quiz games, chess etc for a few hundred years- however we have beengiving a mirthless smile, appearing visibly bored, and gazing inscrutably, for all of human evolutionary history. Our ancestors have been doing some of these activities for millions of years (and their survival has depended upon it!
Computers have a long way to go to match that, and while the attempt is interesting it is not particularly useful.
The most practical forms of AI are the ones that ignore the “being like a human” issue and just get on with the task in hand.