Image by Author

I happened to receive many calls from friends, and friends of friends on how to break into a Data Scientist job. So I thought, for the benefit of all those who have similar questions, this would be a definitive place to find all the answers.

Data Scientist: Roles and Responsibilities

Though the exact scope of roles and responsibilities of a Data Scientist may vary a little depending on what type of company you join, the core responsibilities are pretty standard:

  1. Data Analysis and Visualisation
  2. Building AI/ML Models
  3. Cross-functional Communication Skills
  4. Basic to Intermediate Software Development Skills

Apart from these you may also be expected to…

View in a valley
View in a valley
Photo by the author.

The Requirement

I was recently exploring multiple models for building a cognitive search bot for open-domain question answering. The bot should be capable of returning the appropriate answer to the question posed by the user. Of course, the efficacy of the bot is limited by the content of the training dataset. However, this bot should be robust enough to handle misspelled words — especially named entities (proper nouns such as names, places, animals, things).

For my prototype, I chose the WikiQA dataset from Microsoft Research. …

Image by Author

The problem of incident ticket classification is one of huge impact to IT companies. When users raise a ticket, the ticket needs to be directed to the right team as quickly as possible, to ensure speedy resolution. Sending it to the wrong department leads to longer resolution times since it takes time for the ticket to be redirected to the right team. Currently, in most of the companies, ticket classification is done manually, which is prone to errors, and is tedious as the volume of tickets increases. Hence, we need a better solution to handle this problem.

Auto-ticket assignment: Problem

In order to…

While building neural networks, a lot of beginners and non-beginners alike, seem to get caught up in figuring out the input shape that needs to be fed into the neural network.

But why should we know the input shape, and why should we feed it? Can’t the Neural Network figure it out on its own?
The answer to this question lies in the basics of matrix multiplication.

Suppose we have two matrices A and B. Let the dimensions of B be m rows x n columns. Now, for the two matrices to be compatible for multiplication, the column dimension…

Beauty of Simplicity (Image by Author)

The overwhelming world of mammoth language models

Ever since the advent of transfer learning in Natural Language Processing, larger and larger models have been presented, in order to make more and more complex language tasks possible.

But the more complex the models, the more time and the more amount of data it needs to train. The latest GPT-3 model achieves state of the art results in most natural language tasks, but it has close to 175 billion parameters to train, and takes years to train!

So is there a way around it?

Timo Schick and Hinrich Schutze came up with an ensemble masked language model training method which has proven to be as potent…

Image by Author

There is now no dearth of Data Scientists all over the world. Gradually, the NLP, Computer Vision, Predictive Analytics, you name it!

With such tough competition, one needs to set themselves apart from the herd. And this edge is quickly revealing itself. There is now a huge demand for Operations Research Scientists.

What is Operations Research?

If you have watched the movie ‘The Imitation Game’, then you might have some idea of what Operations Research is, what power it wields.

Operations Research deals with using analytics for solving optimization problems under constraints. It is also referred to as discrete optimization or decision science. …

Image by Author

If you are a Data Scientists like me, who needs to develop solutions with the end user in mind, chances are your product/project manager will ask you to demonstrate your solution as it would work in the production environment. You need to fashion a POC (proof of concept) demonstrating the workflow from the user’s perspective.

For this, you will need to understand a thing or two about microservices. To get a fundamental understanding myself, I took this free course from Udacity. I am summarising my learning from the course for the benefit of fellow Data Scientists.

Microservice Architecture: What does it mean?

The phrase Microservice Architecture…

Image by Author

All the data scientists out there who have worked in the domain of automating IT Service Management must have tried their hand at the highly (in)famous IT ticket classification problem.

For those who haven’t, this is the problem statement:

Given a corpus of IT incident tickets, find the optimal number of categories the tickets can be distinctly segregated into, and then classify the incoming tickets into the identified categories

Sweet LDA!

For anyone who read this problem, your minds might immediately wander off to the Topic Modelling domain. The first thing anyone would land on is the LDA (Latent Dirichlet Allocation) algorithm.

Image by Author

Chances are, if you are working on building conversational agents (a.k.a chatbots), you have explored various platforms such as RASA, or Microsoft’s QnA Maker. Each one is composed of very similar algorithmic components but different technological choices. However, each of them comes with different features, different levels of customisation, post-analytics, etc.

And now, there is a new kid on the block: Mindmeld!! It is based on Elasticsearch, is completely open source and is offers in-built ‘blueprints’ that correspond to various domain (food orderig, HR assistant, Home assistant, etc.) …

Image by Author

Every Product Manager is faced with one important question every release cycle: which feature(s) to prioritize for this release?

Some features might look super savvy, and having them on your marketing material might be super attractive, but it may take a whole lot of development effort from multiple teams. On the other hand, there might be some low-hanging fruit that may just be some minor bug fixes. How would you choose which one to prioritise?

First Principles

If I had to choose a feature to work on first, I would pick the one with the biggest bang for the buck — the…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store