The Requirement

I was recently exploring multiple models for building a cognitive search bot for open-domain question answering. The bot should be capable of returning the appropriate answer to the question posed by the user. Of course, the efficacy of the bot is limited by the content of the training dataset. However, this bot should be robust enough to handle misspelled words — especially named entities (proper nouns such as names, places, animals, things).

For my prototype, I chose the WikiQA dataset from Microsoft Research. …

The problem of incident ticket classification is one of huge impact to IT companies. When users raise a ticket, the ticket needs to be directed to the right team as quickly as possible, to ensure speedy resolution. Sending it to the wrong department leads to longer resolution times since it takes time for the ticket to be redirected to the right team. Currently, in most of the companies, ticket classification is done manually, which is prone to errors, and is tedious as the volume of tickets increases. Hence, we need a better solution to handle this problem.

Auto-ticket assignment: Problem

In order to…

While building neural networks, a lot of beginners and non-beginners alike, seem to get caught up in figuring out the input shape that needs to be fed into the neural network.

But why should we know the input shape, and why should we feed it? Can’t the Neural Network figure it out on its own?
The answer to this question lies in the basics of matrix multiplication.

Suppose we have two matrices A and B. Let the dimensions of B be m rows x n columns. Now, for the two matrices to be compatible for multiplication, the column dimension…

The overwhelming world of mammoth language models

Ever since the advent of transfer learning in Natural Language Processing, larger and larger models have been presented, in order to make more and more complex language tasks possible.

But the more complex the models, the more time and the more amount of data it needs to train. The latest GPT-3 model achieves state of the art results in most natural language tasks, but it has close to 175 billion parameters to train, and takes years to train!

So is there a way around it?

Timo Schick and Hinrich Schutze came up with an ensemble masked language model training method which has proven to be as potent…

Every Data Scientist working in the Enterprise AI domain must have, or will be dealing with smart chatbots. With the surge in NLP models such as the BERT family, the GPT family and other heavyweight models, semantic question answering has become very easy.

Add to this knowledge-base providers such as Elasticsearch, which allow for custom search functions, the bots have become efficient as well.

However, when you build a smart-bot you need to quantify its performance. This is very important in order to figure out if it is even a good idea to go ahead with the bot. …

There is now no dearth of Data Scientists all over the world. Gradually, the NLP, Computer Vision, Predictive Analytics, you name it!

With such tough competition, one needs to set themselves apart from the herd. And this edge is quickly revealing itself. There is now a huge demand for Operations Research Scientists.

What is Operations Research?

If you have watched the movie ‘The Imitation Game’, then you might have some idea of what Operations Research is, what power it wields.

Operations Research deals with using analytics for solving optimization problems under constraints. It is also referred to as discrete optimization or decision science. …

If you are a Data Scientists like me, who needs to develop solutions with the end user in mind, chances are your product/project manager will ask you to demonstrate your solution as it would work in the production environment. You need to fashion a POC (proof of concept) demonstrating the workflow from the user’s perspective.

For this, you will need to understand a thing or two about microservices. To get a fundamental understanding myself, I took this free course from Udacity. I am summarising my learning from the course for the benefit of fellow Data Scientists.

Microservice Architecture: What does it mean?

The phrase Microservice Architecture…

All the data scientists out there who have worked in the domain of automating IT Service Management must have tried their hand at the highly (in)famous IT ticket classification problem.

For those who haven’t, this is the problem statement:

Given a corpus of IT incident tickets, find the optimal number of categories the tickets can be distinctly segregated into, and then classify the incoming tickets into the identified categories

Sweet LDA!

For anyone who read this problem, your minds might immediately wander off to the Topic Modelling domain. The first thing anyone would land on is the LDA (Latent Dirichlet Allocation) algorithm.

…