Baidu disqualified from AI test

Chinese search giant Baidu has been disqualified from an artificial intelligence test in which it appeared to beat Google and Microsoft.

The test, in May, looked at how well computers recognise images.

But it has now emerged that Baidu submitted far more tests than the rules allowed, meaning its results were not comparable with others in the competition.

The firm has apologised for “this mistake”.

Its win has now been cancelled and it has been banned from similar challenges for a year.

Rule-breaking

Baidu claimed to have scored an impressive 4.58% error rate on a test which asked contestants to accurately recognise a series of images from a large picture database.

Although Google and Microsoft were not far behind – with 4.94% and 4.8% error rates respectively – it was still regarded as a significant victory especially as Baidu is a relative newcomer to AI.

But its victory turned out to be short-lived when, earlier this week, organisers of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) – run at Stanford University’s vision lab – revealed that Baidu had broken its rules.

“During the period of 28 November 2014 to 13 May 2015, there were at least 30 accounts used by a team from Baidu to the test server at least 200 times, far exceeding the specified limit of two submissions per week,” it wrote.

It included some periods of very high usage – 40 submissions in five days during March.

In a statement Baidu said that it had been informed that “we exceeded the allowable number of weekly submissions to the ImageNet servers”.

“We apologise for this mistake and are continuing to review the results,” it added.

Dog breeds

ImageNet was set up as a resource for AI researchers and contains millions of pictures.

Getting computers to accurately recognise the content and context of pictures is one of the key goals in the field of artificial intelligence and has obvious benefits for the search industry.

The competition required competitors to arrange pictures into predefined categories and recognise small differences in images, such as different breeds of dog.

For the test, Baidu used its deep learning supercomputer dubbed Minwa. In January, it said its computer vision system – Deep Image – was the world’s “most accurate”.

At the time of the competition victory, Baidu scientist Ren Wu claimed the firm was now “leading the race in computer intelligence”.

Since the issue emerged, the AI world has been buzzing, with many leading experts asking for details about what happened.

Baidu has offered no explanation beyond calling the incident a “mistake”.