It is only recently that I have started developing a keen interest in Text Analytics and below is a summary of some of the key learning I have had till date.

At the crux of it, Text Mining helps derive high quality information from natural language text. Text mining works by constructing numeric data from unstructured text and analyzing it. During my learning, I have started off by taking some unstructured data from the web and applying different techniques on it. I used the tool R for text mining. R is a good tool for analyzing data. But what I have found is that as the data set becomes larger, R will start having memory issues. Python would be a solution for that, which have been used by friends/colleagues for their project. I am trying to learn Python as well slowly. Right now let me take you through text mining with few R codes

Steps in Basic Text Mining

1. Corpus Creation

The first step is data collection. The data could be anything including social media data, speeches, reviews, letters, website information and anything which is in a text format.

Once we have the data, we need to bring the data together into one big collection. This is called as Corpus in text analytics language. A corpus may contain texts in a single language or text data in multiple languages. Text is usually annotated as well using POS tagging (Part of speech tagging). In my example I have not annotated, but could be done

There are various types of corpora and there are various ways they are being used currently. Some corporas can also be used for speech recognition and machine learning. It could be used for translation to any language as well

The library ‘tm’ in R is very helpful in doing text mining. It has a lot of useful functions through which you could do a lot of manipulation and modeling. Below is the code for corpus creation

# Initiating the library
library(tm) 

# In the path the forward slash (/) should 
# be used. Remember to change them in the 
# code. In the path you could have a 
# collection of notepad, it will collect
# everything from the path and create a corpus
Doc <- Corpus(DirSource("Path of the file")) 

2. Data Cleaning

Once the corpus is ready, I would want to clean the data. As you know the data could be from Social media etc which is very cluttered and not clean. I will demonstrate below some of the basic data cleaning steps which could be used for cleaning of data in text analysis.

Towards cleaning, mainly two functions from tm package could be used. They are tm_map and sub. A lot of functionalities could be done using these tow functions

# Removing all the punctuation’s in the data
Doc <- tm_map(Doc, removePunctuation) 

# It helps to convert the data into either 
# upper case or lower case. Lower case is 
# usually preferred. Tools such as R is 
# case sensitive. If there is a mix of cases 
# in words (not uniform), it won’t be identify
# and group the words
Doc <- tm_map(Doc, tolower) 

# Removing all the stop words in the data 
# such as which, and , the etc. If you 
# inspect stopwords(“english”) , R will give 
# the list of stop words in English
Doc <- tm_map(Doc, removeWords, stopwords("english")) 

# This function strips all white spaces 
# in the data
Doc <- tm_map(Doc, stripWhitespace) 

# This function stems all the document 
# and creates roots. Stems ending like 
# ‘ing’, ‘ed’, ‘s’, ‘es’ etc.
Doc <- tm_map(Doc, stemDocument) 

# Using this, all the special characters 
# such as /,@,,” etc could be deleted. If 
# there is a long list of special characters 
# while looking at the data , a user defined 
# function could be written to remove all at once
Doc <- gsub("/", " ", Doc) 

# Once we do the entire cleaning, it helps 
# to convert the data back into text as 
# sometimes the above operations could 
# convert the text into character (Please 
# refer R doc to understand the different 
# data types)
Doc <- tm_map(Doc, PlainTextDocument)

3. Creation of Document Term Matrix

Corpus is in text format which we need to convert into some form of numeric data to be able to do mathematical operation and modeling on that data. For this we have to create a Document Term Matrix

A document term matrix is a mathematical matrix in which the rows are the documents (We spoke of saving different notepads in the path from which we create the corpus) and columns are different terms. I have analysed Obamas and Mccain’s speeches. So each speech is the row and the terms in the speech are the columns

We have frequency of each terms in the matrix. There are two types of frequency that we use. They are tf-idf. Tf is the term frequency and Idf is the inverse document frequency. Tf- IDf shows how important is a term in corpus. It is often used as weighing factor in information retrieval and text mining.

Term frequency is the row frequency of term in a document and Inverse Document Frequency is the measure of how much information the word provides, that is the word is common across all the documents.

# Creation of Document term matrix. The rows 
# will be documents and columns will be terms 
# and the cells will be frequencies
DTM <- DocumentTermMatrix(Doc) 

# This is called Term Document Matrix which 
# is nothing but the transpose of the above. 
# Here columns will be documents and rows 
# would be term. We could use whichever we 
# find suitable for our analysis
TDM <- TermDocumentMatrix(Doc) 

# Obtain term frequencies as a vector by
# converting the document term matrix and 
# summing up the counts
Freq <- colSums(as.matrix(dtm)) 

# Sorting based on frequencies
ord <- order(Freq) 

# We can look at most frequent items
findFreqTerms(DTM, lowfreq=50) 

# sorting and converting into a matrix
Freq <- sort(colSums(as.matrix(DTM)), decreasing=TRUE) 

# Creation of a data frame / table
DF <- data.frame(word=names(Freq), Freq=Freq)

4. Analysing the above data

The above data could be analyzed using various techniques like clustering, SVM, decision tree and the likes . I am just starting by doing a basic word cloud first

# Creation of word cloud
wordcloud(names(Freq), Feq, min.freq=10, scale=c(5, .1), colors=brewer.pal(6, "Dark2"), random.order=F) 

Attaching below a snippet of comparison word cloud between Obama’s and Mccain’s speech.

I have also attempted a clustering using KNN (nearest neighbour algorithm). Using that, if I mention any part of the speech, my model will be able to classify it correctly. It will be able to predict with a probability whose speech it is. I have not completed that as yet. Will share it in my next blog. This is my first attempt in blog writing, hope you didn’t find it very boring 🙂