One of the major components of bot design is training . Only an adequately trained bot is effective and accurate. So how can a developer know whether the bot is trained appropriately?
Let’s look at how to use the Training Validation feature and identify incorrect training - patterns during design time. The article touches upon this feature at a high level, to help the users with more details and examples.
Patterns are unique key words used mainly for Intent detection through the FM engine of the platform. They can be used to define metaphors and other idiomatic expressions for task names. Concepts can also be used as patterns.
Problem Statement:
Due to incorrect patterns present in the bot, the platform will not identify intents and entities accurately which leads to increase in FP count. This has a negative impact on overall accuracy of the bot prediction.
Solution:
The XO Platform offers a feature ‘Validate Model’ that can help the users validate and show the patterns or training that have issues.
This feature allows the NLP designers/developers to identify these issues proactively in the training phase (design time itself) through a set of recommendations, thereby improving the NLP performance. Thus, providing the best experience to end-users with more accurate conversations and reducing the cost required to rectify models after deployment. (Note – Implementation for incorrect entity patterns will be done in future)
The validation happens for both Intent identification patterns and negative patterns.
Incorrect pattern errors can be of two kinds – Errors and Warnings, for Syntactic and Semantic errors respectively
- Syntax errors – shown when there is an error in syntax
- Semantic errors – shown when there is no error in pattern syntax but if it contains undefined/unknown invalid words (Say the pattern refers to a concept but the concept is not defined yet)
Out of scope - If there are patterns that are syntactically correct but are clashing with other patterns in the bot and causing an impediment to identifying an intent, this feature will not be helpful.
Real time instance where this feature benefited the customer:
One of the customers had issues with intent identification from Patterns with many FPs. For the same bot we executed the validation phase and suggested all the improvements.
Assumptions before we continue further with the article:
- A bot is created with incorrect patterns (having both syntactic and semantic errors) added to the dialog tasks in it
- Also, negative incorrect patterns are added (having both syntactic and semantic errors)
- The bot is trained and is ready for checking the validations
How to validate Patterns using NLU validate model:
VA’s Build menu → Natural language → Training - Click on ‘Validate Model’ (No need to select K-fold/Confusion Matrix), please ensure that you click on the refresh button (even if you are using this feature for the first time)
Refreshing Recommendations:
Once the developer makes changes to the incorrect pattern, to revalidate, the developer will need to train the bot again and hit the refresh button
-
On the VA’s Build menu, click Natural language → Training → Click on ‘Validate Model’ button and click on ‘Refresh recommendations’ button.
- Model validation will get initiated and once it is completed, the timestamp of the last refresh is displayed along with a number of recommendations and all the recommendations will be listed and shown on the screen.
Under the ‘All’ tab – errors and warnings are listed - Under the ‘Error’ tab – errors of utterances and patterns are listed
- Under the ‘Warning’ tab – warnings of utterances and patterns are listed
- Model validation will get initiated and once it is completed, the timestamp of the last refresh is displayed along with a number of recommendations and all the recommendations will be listed and shown on the screen.
-
In the below pattern recommendation, ‘16 intents have Patterns with invalid syntax.’ - 16 is the count of intents with the issue i.e. patterns with invalid syntax.
Note: Model validation status along with the timestamp is updated in the status-docker after refreshing the recommendations.
Recommendations can be refreshed in the below scenarios -
- Modifying the data/pattern – When the data is modified, you can check for new recommendations
- Adding new data/pattern – When new patterns are added, you can check for new recommendations
- Resolving the validations – When validations are corrected, you can check if they are resolved and if intents are identified properly
How to view incorrect Pattern validations:
- Recommendations summary can be seen in ‘Validate model’ popup as mentioned in above section
- Set of recommendations can be seen in Training → Intents screen for the intents as shown in the attachment. The validations vary depending on the error or warning recommendations as explained above.
- Click on the error icon and observe that all recommendations(of normal and negative patterns) will be shown so that user can resolve them
- User can view the errors in detail in Patterns screen by clicking on ‘View Patterns’ button displayed below the recommendations
- Errors are shown beside the invalid patterns which has syntax errors in Patterns screen
- Clicking on it will display the error message to let the user know what the error is
- Warnings are shown beside the invalid patterns which has semantic errors in Patterns screen
- Clicking on it will display the error that needs to be rectified
- Hover on the Bulb icon to view the summary of errors and warnings
- Go to Dialog tasks page and validations are shown inside the dialog
Note:
- The Training Validation works for incorrect negative patterns also.
- This feature is not applicable for the published mode. I.e, all the validations are done based on the Development version NLP ONLY. If the user is referring to the published version of the bot (selecting the published version from the top menu), the user will not be able to see any last recommendations and training validation results.
How to validate if the change in patterns is working:
For warnings:
Incorrect pattern having warning validation
Have an incorrect pattern added to dialog intent
Talk to bot and provide the input from the incorrect pattern
The bot will not identify the intent
Resolve the warning of incorrect pattern
Talk to bot and provide the same above input
The bot will now identify the intent
For errors:
Incorrect pattern having error validation
Have an incorrect pattern added to dialog intent
Talk to bot and provide the input from the incorrect pattern
The bot will not identify the intent
Resolve the error of incorrect patterns
Talk to bot and provide the same above input
The bot will now identify the intent
Supported languages : This feature is supported for all languages
So… what do you think of this feature. Try it out and let us know 
Content credits: @sowmya.veerabattini 