"Computer: Analyse the distribution of the pieces that we have, correcting for changes in star configurations over four billion years, then extrapolate for the missing piece" (Star Trek, The Chase)

News

1/7/2018: We have discovered an issue with the partitioning of the dataset and are releasing ComplexWebQuestions version 1.1 with a new partitioning of the dataset. The Question file format remains the same, except that we added an additional field with extra supervision describing the answer of a decomposed question. Please read our report 

A dataset for answering complex questions that require reasoning over multiple web snippets.

ComplexWebQuestions is a new dataset that contains a large set of complex questions in natural language, and can be used in multiple ways:

  1. By interacting with a search engine, which is the focus of our paper (Talmor and Berant, 2018);

  2. As a reading comprehension task: we release 12,725,989 web snippets that are relevant for the questions, and were collected during the development of our model; 

  3. As a semantic parsing task: each question is paired with a SPARQL query that can be executed against Freebase to retrieve the answer.

Question: The actress that had the role of Martha Alston, plays what role in Finding Nemo?

Answer: "Dory"

Title: "Ellen DeGeneres - Wikipedia"

Web Snippet: "..She also played Martha Alston in the 1996 Touchstone Pictures film Mr. Wrong and.. "

Title: "Ellen DeGeneres | Disney Wiki | FANDOM powered by Wikia"

Web Snippet: "... provided the voice of Dory in Disney/Pixar's 2003 animated film, Finding Nemo. ..."

Leaderboard

The dataset contains 34,689 examples, each containing:

  • A complex question

  • Answers (including aliases)

  • An average of 366.8 snippets per question

  • A SPARQL query (against Freebase)

Freebase version used is freebase-rdf-2015-08-02-00-00

 

Sample Questions

  • “Which school that Sir Ernest Rutherford attended has the latest founding date?” 

  • “what movies does Leo Howard play in and that is 113.0 minutes long?"

  • “Where is the end of the river that originates in Shannon Pot?”

  • Test results may be submitted every two weeks, and scores will be computed using the official Eval Script

  • Please send predictions in JSON format (according to the Eval Script) to alontalmor@mail.tau.ac.il

  • If you find this data helpful in your work, please cite this paper:

@inproceedings{talmor18compwebq,
  author = {A. Talmor and J. Berant},
  booktitle = {North American Association for Computational Linguistics (NAACL)},
  title = {The Web as a Knowledge-base for Answering Complex Questions},
  year = {2018},
}

 2019, Alon Talmor

 Tel-Aviv University Natural Language Processing Group