This post presents how to train a sequence to sequence Transformer model for English to Chinese translation, nicely abbreviated to Transection. We adopt BART’s (Lewis, Mike, et al. 2019) architecture for this model and train it in two ways. The first way is to train it from scratch and second way is to fine-tune it from BART’s pre-trained base checkpoint that is available in 🤗transformers. The training examples consist of around 5M English-Chinese sequence pairs, along with a test set that has around 40k pairs. Later in this blog, the performance on the test set measured by sacrebleu is compared between the two ways. In addition, a popular pre-trained model in this domain, i.e., Helsinki-NLP/opus-mt-en-zh from 🤗Huggingface’s models hub is used as a baseline to the two ways. Not only in sacrebleu, their performance in other metrics such as generalisation, model size, and training cost is also discussed.

TL;DR. This link provides the code repository that contains two readily downloadable fine-tuned GPT-2 weights, a quick start guide of how to customize Autocoder, and a list of future pointers to this project. Although this blog looks like a technical introduction to Autocoder, I also by the way talk about a lot of relevant stuff, such as nice work, status quo, and future directions in NLP.

Human have walked into an uncharted territory since the outbreak of the COVID-19 coronavirus. Much effort has been made to stop the crisis. For example, in machine learning community, some people have been seeking computational techniques for extracting insights from COVID-19 literature, such as the COVID-19 Open Research Dataset Challenge (CORD-19), which was also mentioned in the news.

This post guides you to write a python script that is able to monitor the open-access repository of electronic preprints (arXiv) for automatic post on Twitter, named Feeder-bot. Its workflow is illustrated as follows.

Text classification as an important task in natural lanugage understanding (NLP) has been widely studied over the last several decades. The task describes input as a document and output as the category of which the document belongs to. In literature, both supervised and unsupervised methods have been applied for text classification.

In this post, I share the Chinese Room Thought Experiment that I played with kids in my Communication and Outreach course.

As I said, currently, I am a big fun of AI2. This began when I got knowing its work into an open-source NLP research library - AllenNLP. The more I hack the library, the more attractive it is for me. A short paragraph of praise below was written when I was hacking the tool.