What is NLG?
Natural Language Generation (NLG) includes techniques for translating information into text. NLG is a subset of Natural Language Processing (NLP). If natural language processing techniques can be separated into two, Natural Language Understanding (NLU) is the part that analyzes natural language, NLG is the part that synthesizes meaningful text for human consumption. NLP=NLU+NLG. For example, let's imagine that you are writing to a conversational AI solution. When you ask what the weather is like today, NLU techniques are used first to understand your intent. Weather data is then searched, the results are analyzed and an answer generated thanks to NLG techniques.
NLG techniques can be used to automatically summarize and explain data in a human-like manner. NLG can automate the transfer process of analyzed structured data to humans in seconds.
How does NLG work?
An NLG engine works in 6 stages:
- Content Determination: The limits of the content should be determined. The data often contains more information than necessary.
- Data interpretation: The analyzed data is interpreted. Thanks to machine learning techniques, patterns can be recognized in the processed data.
- Document planning: In this stage, the structures in the data are organized with the goal of creating a narrative structure and document plan.
- Sentence Aggregation: It is also called microplanning and this process is about choosing the expressions and words in each sentence for the end-user.
- Grammaticalization: this stage makes sure that the whole report follows correct grammatical form spelling and punctuation.
- Language Implementation: Makes sure that the document is output in the right format and according to the preferences of the user.
Why is it important now?
- Available data is increasing and NLG helps humans get quick insights from data easily.
- NLG solutions help users get insights information from various data sources. Different data sources can feed complex information in a prioritized manner to NLG solutions and NLG solutions can communicate insights to users. A Gartner report in 2019 predicts that “By 2022, 25% of enterprises will use some form of natural language generation technology.” Though most industry analyst estimates are not accurate, they tend to be directionally right. We also agree that the market is set to expand as companies want to democratize access to their data among employees.
What are the NLG application areas?
Since NLG aims to make sense of the data and create human readable insights, it can be applied to all areas dealing with reporting, content creation and content personalization. Some of these areas are listed:
- Banking & Finance: The banking industry highly relies on data and insights for performance reporting. Additionally, profit and loss reports can be automated using NLG systems. NLG techniques can be used to support fintech chatbots that interact with customers for personal financial management advice.
- Manufacturing: As IoT devices are implemented more widely in production sites, they generate a significant volume of data useful for performance improvement and maintenance. NLG can automate communication of important findings so employees can take action faster.
- Retail and wholesale: NLG solutions can provide not only product descriptions and categorization for online shopping and e-commerce but also help personalize customer communication via chatbots.
- Media: Summarization and content creation can be aided by NLG solutions. Especially sports and financial news tend to follow similar templates and text explaining such events can be easily created.
- Insurance: NLG solutions can help to improve communication of personalized plans for customers.
- Transportation: Chatbots can deliver warnings about delays and schedules. NLG tools can be used to create personalized, easy to read travel plans.
- Politics: Probably the most dangerous use case is using NLG solutions to spread personalized propaganda and misinformation. This is unfortunately has the risk of making the current flow of political disinformation even more dangerous and personalized.
What are the benefits of NLG?
NLG saves time and costs as well as contributes to scalability and consistency via automating repetitive processes involving language.
- Time and cost savings: NLG reduces time and costs as it speeds up the reporting processes.
- Reporting consistency: Reports can be precise and standardized as though they are created by one person. Accuracy of the reports can be improved by time as NLG solutions advance.
- Scalability: By automating parts of the reporting and content generation processes, it becomes possible to create large amounts of content.
How to choose NLG software?
Important factors in evaluating an NLG solution are: evaluating use cases, the existence of suitable data sources, security, language capabilities and vendor dependency.
- While evaluating NLG solutions, it is necessary to find vendors that provide solutions for a particular use case. For example, an NLG solution trained to prepare content in the field of e-commerce would not be suitable for reporting the efficiency of IoT devices.
- NLG solutions can be as good as the underlying data. The project team needs to ensure that they have the structured data which can be combined with the NLG solution to serve users.
- The audience of the reports or content is important. Publishing public content is always risky and would require a more robust quality assurance process.
- In situations where language skills are important, a robust assessment of the NLG tools' language capabilities is necessary. For example, correct use of language is far more important in media and journalism compared to a company's internal reporting.
- Large-scale enterprises can build their own NLG engines using open source solutions and minimize vendor dependency. Two aspects are important in this decision: 1) Is the use case highly specialized? If yes, it may be harder to find an NLG solution from a vendor for that case. 2)Is this a simple or difficult NLG problem? Simple problems can be easily resolved with open source packages.