Researchers have developed an AI model to help computers work more efficiently with a wider variety of languages. 

African languages have received little attention from computer scientists, so few natural language processing capabilities have been available to large swaths of the continent. The new language model, developed by researchers at the University of Waterloo’s David R. Cheriton School of Computer Science, begins to fill that gap by enabling computers to analyze text in African languages for many useful tasks.

The new neural network model, which the researchers have dubbed AfriBERTa, uses deep-learning techniques to achieve state-of-the-art results for low-resource languages. 

The neural language model works specifically with 11 African languages, such as Amharic, Hausa, and Swahili, spoken collectively by more than 400 million people. It achieves output quality comparable to the best existing models despite learning from just one gigabyte of text, while other models require thousands of times more data.

“Pretrained language models have transformed the way computers process and analyze textual data for tasks ranging from machine translation to question answering,” said Kelechi Ogueji, a master’s student in computer science at Waterloo. “Sadly, African languages have received little attention from the research community.”

“One of the challenges is that neural networks are bewilderingly text- and computer-intensive to build. And unlike English, which has enormous quantities of available text, most of the 7,000 or so languages spoken worldwide can be characterized as low-resource, in that there is a lack of data available to feed data-hungry neural networks.”

Most of these models work using a technique known as pretraining. To accomplish this, the researcher presented the model with text where some of the words had been covered up or masked. The model then had to guess the masked words. By repeating this process, many billions of times, the model learns the statistical associations between words, which mimics human knowledge of the language.

“Being able to pretrain models that are just as accurate for certain downstream tasks, but using vastly smaller amounts of data has many advantages,” said Jimmy Lin, the Cheriton Chair in Computer Science and Ogueji’s advisor. “Needing less data to train the language model means that less computation is required and consequently lower carbon emissions associated with operating massive data centers. Smaller datasets also make data curation more practical, which is one approach to reduce the biases present in the models.”

“This work takes a small but important step to bringing natural language processing capabilities to more than 1.3 billion people on the African continent.”

Assisting Ogueji and Lin in this research is Yuxin Zhu, who recently completed an undergraduate degree in computer science at Waterloo. Together, they present their research paper, Small data? No problem! Exploring the viability of pretrained multilingual language models for low-resource languages, at the Multilingual Representation Learning Workshop at the 2021 Conference on Empirical Methods in Natural Language Processing.

Academics and data scientists from Durham University, a public research university in Durham, England, and UN Global Pulse (UNGP) have developed an agent-based model to simulate the spread of COVID-19 in the Cox’s Bazar refugee settlement in Bangladesh.

The researchers analyzed several operational interventions by modeling the interactions of over 900,000 Rohingya refugees and found that mask-wearing is highly effective to slow the spread of COVID-19. 

Researchers also established that handling of positive cases in isolation and treatment centers have little impact on the spread of COVID-19 in comparison to home isolation for individuals with mild symptoms, mainly due to the exceptionally high population density in the settlement and many facilities being communal that poses an increased risk of coronavirus transmission. 

Furthermore, at the time of the study, the simulation results indicated that the reopening of learning centers could lead to a higher infection rate in the refugee settlement, where social distancing is nearly impossible. This led the researchers to explore various mitigation strategies.

The study adapted the JUNE epidemic model to the settlement setting. The team took a scenario-based approach that focused on simulating the relative effectiveness of the above-mentioned interventions in the settlement.

The modeling followed a three-step process of (1) building a ‘digital twin’ of the Cox’s Bazar refugee settlement that (2) simulated the possible movement and interaction patterns among the residents and (3) implementation of operational interventions to simulate its effects on the spread of COVID-19 in the settlement. 

Virtual individuals were included in the model with different demographic attributes that mirrored real-world statistics. A simulation engine was designed by the researchers that captured the movement and interaction patterns of the people in the model.

Full results of the study have been published in the journal PLOS Computational Biology. 

The study findings have allowed decision-makers in the refugee settlement to set up new contingency plans for high case numbers and develop policies on the safe opening of various indoor spaces.

A mask-wearing strategy was rolled out, which included mask-making, and communication and engagement campaigns to increase correct mask usage, as the model showed how this could significantly reduce the spread of COVID-19 over time.

The model has been informed by data from UNHCR, the UN’s Refugee Agency, on geography, demographics, comorbidities, physical infrastructure, and other parameters obtained from real-world observations. 

The study results were presented in a series of reports that provided crucial insights and limitations relevant to this modeling approach to the World Health Organisation and UNHCR public health professionals operating in the settlement on the potential effectiveness of interventions to curb the spread of COVID-19.  

Chris Earney, Deputy Director of UNGP said: “The project has fulfilled its operational objectives successfully and the team is aiming to scale the model implementation further with future applications and partnerships.”

The JUNE open-source modeling framework has been developed by the researchers during the pandemic and was originally applied to simulating the spread of COVID-19 in England.

Professor Frank Krauss of Durham University said: “The work with the UN and the WHO is super-exciting and a very good example for the caliber of our Ph.D. students. It is great to see their enthusiasm, skills, and drive: this project started from zero, and within months we had a highly competitive COVID simulation for the UK, all while they also collaborated with international agencies to apply this to a completely new setting. This is nothing short of a truly excellent achievement!”               

The research conducted by UNGP has been supported by the Government of Sweden, and the William and Flora Hewlett Foundation and the Ph.D. students were supported by the UKRI-STFC grant.