QA data preparation

Before the training can start, you need to upload all the necessary ingredients to start the training job. For this example, we will focus on open book question answering on a set of Wikipedia articles, where the answer to every question is present in the corresponding reading passage. To train a model for this purpose, we will need the following:

Task description

Describes the task you expect the model to perform; you can think of it as an LLM prompt that would help the model solve your task. In practice for a question answering problem, we expect a single component: task_description  that describes the main task.

The expected format is a JSON blob, and for Wikipedia QA, we should have the following:

1{
2 "task_description": "Answer the question using information in the context"
3}

Test/Train data

We need a testing dataset that we can use to evaluate the performance of the fine-tuned model on your task. For the training stage, we need only a few dozen examples to fine-tune your model. Of course, the more diverse and bigger the dataset, the better.

The expected format is CSV or JSON-lines with the following columns

  1. question is the question the model must answer.
  2. context holds information needed to answer the question.
  3. answer is the answer to the question (based on the context).

The data for the open book Wikipedia question answering should look like this:

JSON format

1{"context":"On August 15, 1971, the United States unilaterally pulled out of the Bretton Woods Accord. The US abandoned the Gold Exchange Standard whereby the value of the dollar had been pegged to the price of gold and all other currencies were pegged to the dollar, whose value was left to \"float\" (rise and fall according to market demand). Shortly thereafter, Britain followed, floating the pound sterling. The other industrialized nations followed suit with their respective currencies. Anticipating that currency values would fluctuate unpredictably for a time, the industrialized nations increased their reserves (by expanding their money supplies) in amounts far greater than before. The result was a depreciation of the dollar and other industrialized nations' currencies. Because oil was priced in dollars, oil producers' real income decreased. In September 1971, OPEC issued a joint communiqu\u00e9 stating that, from then on, they would price oil in terms of a fixed amount of gold.","question":"What does it mean when currencies are left to \"float?\"","answer":"to \"float\" (rise and fall according to market demand)"}
2{"context":"Firstly, certain costs are difficult to avoid and are shared by everyone, such as the costs of housing, pensions, education and health care. If the state does not provide these services, then for those on lower incomes, the costs must be borrowed and often those on lower incomes are those who are worse equipped to manage their finances. Secondly, aspirational consumption describes the process of middle income earners aspiring to achieve the standards of living enjoyed by their wealthier counterparts and one method of achieving this aspiration is by taking on debt. The result leads to even greater inequality and potential economic instability.","question":"What are those with lower incomes often unable to manage?","answer":"their finances"}
3{"context":"Much of the city's tax base dissipated, leading to problems with funding education, sanitation, and traffic control within the city limits. In addition, residents in unincorporated suburbs had difficulty obtaining municipal services, such as sewage and building code enforcement. In 1958, a study recommended that the city of Jacksonville begin annexing outlying communities in order to create the needed tax base to improve services throughout the county. Voters outside the city limits rejected annexation plans in six referendums between 1960 and 1965.","question":"In what part of the city did residents suffer from a lack of city services?","answer":"unincorporated suburbs"}
4{"context":"The region is home to about 2.5 million insect species, tens of thousands of plants, and some 2,000 birds and mammals. To date, at least 40,000 plant species, 2,200 fishes, 1,294 birds, 427 mammals, 428 amphibians, and 378 reptiles have been scientifically classified in the region. One in five of all the bird species in the world live in the rainforests of the Amazon, and one in five of the fish species live in Amazonian rivers and streams. Scientists have described between 96,660 and 128,843 invertebrate species in Brazil alone.","question":"How many species of insects are known in the region?","answer":"2.5 million"}
5{"context":"In the fall quarter of 2014, the University of Chicago enrolled 5,792 students in the College, 3,468 students in its four graduate divisions, 5,984 students in its professional schools, and 15,244 students overall. In the 2012 Spring Quarter, international students comprised almost 19% of the overall study body, over 26% of students were domestic ethnic minorities, and about 44% of enrolled students were female. Admissions to the University of Chicago is highly selective. The middle 50% band of SAT scores for the undergraduate class of 2015, excluding the writing section, was 1420\u20131530, the average MCAT score for entering students in the Pritzker School of Medicine in 2011 was 36, and the median LSAT score for entering students in the Law School in 2011 was 171. In 2015, the College of the University of Chicago had an acceptance rate of 7.8% for the Class of 2019, the lowest in the college's history.","question":"In the fall quarter of 2014, how many students signed up for the university's professional schools?","answer":"5,984"}

CSV format

questioncontextanswer
What does it mean when currencies are left to “float?”On August 15, 1971, the United States unilaterally pulled out of the Bretton Woods Accord. The US abandoned the Gold Exchange Standard whereby the value of the dollar had been pegged to the price of gold and all other currencies were pegged to the dollar, whose value was left to “float” (rise and fall according to market demand). Shortly thereafter, Britain followed, floating the pound sterling. The other industrialized nations followed suit with their respective currencies. Anticipating that currency values would fluctuate unpredictably for a time, the industrialized nations increased their reserves (by expanding their money supplies) in amounts far greater than before. The result was a depreciation of the dollar and other industrialized nations’ currencies. Because oil was priced in dollars, oil producers’ real income decreased. In September 1971, OPEC issued a joint communiqué stating that, from then on, they would price oil in terms of a fixed amount of gold.to “float” (rise and fall according to market demand)
What are those with lower incomes often unable to manage?Firstly, certain costs are difficult to avoid and are shared by everyone, such as the costs of housing, pensions, education and health care. If the state does not provide these services, then for those on lower incomes, the costs must be borrowed and often those on lower incomes are those who are worse equipped to manage their finances. Secondly, aspirational consumption describes the process of middle income earners aspiring to achieve the standards of living enjoyed by their wealthier counterparts and one method of achieving this aspiration is by taking on debt. The result leads to even greater inequality and potential economic instability.their finances
In what part of the city did residents suffer from a lack of city services?Much of the city’s tax base dissipated, leading to problems with funding education, sanitation, and traffic control within the city limits. In addition, residents in unincorporated suburbs had difficulty obtaining municipal services, such as sewage and building code enforcement. In 1958, a study recommended that the city of Jacksonville begin annexing outlying communities in order to create the needed tax base to improve services throughout the county. Voters outside the city limits rejected annexation plans in six referendums between 1960 and 1965.unincorporated suburbs
How many species of insects are known in the region?The region is home to about 2.5 million insect species, tens of thousands of plants, and some 2,000 birds and mammals. To date, at least 40,000 plant species, 2,200 fishes, 1,294 birds, 427 mammals, 428 amphibians, and 378 reptiles have been scientifically classified in the region. One in five of all the bird species in the world live in the rainforests of the Amazon, and one in five of the fish species live in Amazonian rivers and streams. Scientists have described between 96,660 and 128,843 invertebrate species in Brazil alone.2.5 million
In the fall quarter of 2014, how many students signed up for the university’s professional schools?In the fall quarter of 2014, the University of Chicago enrolled 5,792 students in the College, 3,468 students in its four graduate divisions, 5,984 students in its professional schools, and 15,244 students overall. In the 2012 Spring Quarter, international students comprised almost 19% of the overall study body, over 26% of students were domestic ethnic minorities, and about 44% of enrolled students were female. Admissions to the University of Chicago is highly selective. The middle 50% band of SAT scores for the undergraduate class of 2015, excluding the writing section, was 1420–1530, the average MCAT score for entering students in the Pritzker School of Medicine in 2011 was 36, and the median LSAT score for entering students in the Law School in 2011 was 171. In 2015, the College of the University of Chicago had an acceptance rate of 7.8% for the Class of 2019, the lowest in the college’s history.5,984

Unstructured dataset

The unstructured dataset is used to guide the teacher model in generating diverse, domain-specific data. It can be documentation, unlabelled examples, or even industry literature that contains such information.

For the open book question answering problem we will use assorted passages from wikipedia. We can generate new questions based on the information in those passages and then answer them.

The expected format is CSV or JSON lines with a single column (context). For the wikipedia QA task it should look like this:

JSON format

1{"context":"For months each side had been building forward rifle pits and defensive positions, which resulted in many skirmishes. Artillery fire aiming to gain superiority over the enemy guns.:450\u2013462 September saw the final assault. On 5 September, another French bombardment (the sixth) was followed by an assault by the French Army on 8 September resulting in the capture of the Malakoff by the French, and following their failure to retake it, the collapse of the Russian defences. Meanwhile, the British captured the Great Redan, just south of the city of Sevastopol. The Russians retreated to the north, blowing up their magazines and the city fell on 9 September 1855 after a 337-day-long siege.:106","question":"Who failed to take back Malakoff?","answer":"the Russian defences"}
2{"context":"\"Funday Night at the Movies\" was replaced in 2008 by \"Essentials Jr.\", a youth-oriented version of its weekly series The Essentials (originally hosted by actors Abigail Breslin and Chris O'Donnell, then by John Lithgow from 2009 to 2011, and then by Bill Hader starting with the 2011 season), which included such family-themed films as National Velvet, Captains Courageous and Yours, Mine and Ours, as well as more eclectic selections as Sherlock, Jr., The Music Box, Harvey, Mutiny on the Bounty and The Man Who Knew Too Much.","question":"Along with Chris O'Donnell, who was the original host of Essentials Jr.?","answer":"Abigail Breslin"}
3{"context":"This view of 'diplomatic drift' as the cause of the war was first popularised by A. W, Kinglake, who portrayed the British as victims of newspaper sensationalism and duplicitous French and Ottoman diplomacy. More recently, the historians Andrew Lambert and Winfried Baumgart have argued that, first, Britain was following a geopolitical strategy in aiming to destroy a fledgling Russian Navy which might challenge the Royal Navy for control of the seas, and second that the war was a joint European response to a century of Russian expansion not just southwards but also into western Europe.","question":"Who believed Britain was following a strategy when trying to destroy Russian Navy?","answer":"Andrew Lambert and Winfried Baumgart"}
4{"context":"The original city council line-up of 14 members (nine district-based and five at-large positions) was based on a U.S. Justice Department mandate which took effect in 1979. At-large council members represent the entire city. Under the city charter, once the population in the city limits exceeded 2.1 million residents, two additional districts were to be added. The city of Houston's official 2010 census count was 600 shy of the required number; however, as the city was expected to grow beyond 2.1 million shortly thereafter, the two additional districts were added for, and the positions filled during, the August 2011 elections.","question":"What population figure needs to be passed to add at large council members?","answer":"2.1 million"}
5{"context":"The school newspaper of the University of Kansas is University Daily Kansan, which placed first in the Intercollegiate Writing Competition of the prestigious William Randolph Hearst Writing Foundation competition, often called \"The Pulitzers of College Journalism\" in 2007. In Winter 2008, a group of students created KUpedia, a wiki about all things KU. They have received student funding for operations in 2008\u201309. The KU Department of English publishes the Coal City Review, an annual literary journal of prose, poetry, reviews and illustrations. The Review typically features the work of many writers, but periodically spotlights one author, as in the case of 2006 Nelson Poetry Book Award-winner Voyeur Poems by Matthew Porubsky.","question":"What is the name of the poet who won the Nelson award in 2006?","answer":"Matthew Porubsky"}

CSV format

context
For months each side had been building forward rifle pits and defensive positions, which resulted in many skirmishes. Artillery fire aiming to gain superiority over the enemy guns.:450–462 September saw the final assault. On 5 September, another French bombardment (the sixth) was followed by an assault by the French Army on 8 September resulting in the capture of the Malakoff by the French, and following their failure to retake it, the collapse of the Russian defences. Meanwhile, the British captured the Great Redan, just south of the city of Sevastopol. The Russians retreated to the north, blowing up their magazines and the city fell on 9 September 1855 after a 337-day-long siege.:106
”Funday Night at the Movies” was replaced in 2008 by “Essentials Jr.”, a youth-oriented version of its weekly series The Essentials (originally hosted by actors Abigail Breslin and Chris O’Donnell, then by John Lithgow from 2009 to 2011, and then by Bill Hader starting with the 2011 season), which included such family-themed films as National Velvet, Captains Courageous and Yours, Mine and Ours, as well as more eclectic selections as Sherlock, Jr., The Music Box, Harvey, Mutiny on the Bounty and The Man Who Knew Too Much.
This view of ‘diplomatic drift’ as the cause of the war was first popularised by A. W, Kinglake, who portrayed the British as victims of newspaper sensationalism and duplicitous French and Ottoman diplomacy. More recently, the historians Andrew Lambert and Winfried Baumgart have argued that, first, Britain was following a geopolitical strategy in aiming to destroy a fledgling Russian Navy which might challenge the Royal Navy for control of the seas, and second that the war was a joint European response to a century of Russian expansion not just southwards but also into western Europe.
The original city council line-up of 14 members (nine district-based and five at-large positions) was based on a U.S. Justice Department mandate which took effect in 1979. At-large council members represent the entire city. Under the city charter, once the population in the city limits exceeded 2.1 million residents, two additional districts were to be added. The city of Houston’s official 2010 census count was 600 shy of the required number; however, as the city was expected to grow beyond 2.1 million shortly thereafter, the two additional districts were added for, and the positions filled during, the August 2011 elections.
The school newspaper of the University of Kansas is University Daily Kansan, which placed first in the Intercollegiate Writing Competition of the prestigious William Randolph Hearst Writing Foundation competition, often called “The Pulitzers of College Journalism” in 2007. In Winter 2008, a group of students created KUpedia, a wiki about all things KU. They have received student funding for operations in 2008–09. The KU Department of English publishes the Coal City Review, an annual literary journal of prose, poetry, reviews and illustrations. The Review typically features the work of many writers, but periodically spotlights one author, as in the case of 2006 Nelson Poetry Book Award-winner Voyeur Poems by Matthew Porubsky.