Linguistic Analysis and Data Science

1. Corpus and NLP for Text Analytics
2. Crash course for R: Regular Expression

• 嘗試模擬人類語言理解與處理知識的技術。
• 在這領域其他語言的套件應該比 R 還豐富 (e.g., python 的 nltk)
• NLP-powered Text Analytics 已經是主流，看看這個例子

BUT it is possible to use libraries written in those lower-level and hence faster languages, while writing your code in R and taking advantage of its functional programming style and its many other libraries for data analysis.

• 到現在人都還不完全了解人。
• 語言的豐富性平日可以多多觀察：隱喻譬喻、諷刺幽默、情緒說謊、語用脈絡、 個體群體差異、社會文化變異、甚至討價還價、談情說愛、、。

見 nlp.Rmd

• 線上演示

• 安裝 第一線的研究團隊

• coreNLP package: Wrappers Around Stanford CoreNLP Tools (Java library). Methods provided: tokenisation, part of speech tagging, lemmatisation, named entity recognition, coreference detection and sentiment analysis

# works from CRAN !
install.packages("coreNLP")
# wget http://nlp.stanford.edu/software/stanford-corenlp-full-2015-04-20.zip
download.file("http://nlp.stanford.edu/software/stanford-corenlp-full-2015-04-20.zip")
unzip("stanford-corenlp-full-2015-04-20.zip")
library(coreNLP)
initCoreNLP("stanford-corenlp-full-2015-04-20/")
FB = c("Facebook is looking for new ways to get users to share more, 
rather than just consume content, in a push that seemingly puts it in more 
direct rivalry with Twitter.")
output = annotateString(FB)
getToken(output)[,c(1:3,6:7)]
getParse(output)
getDependency(output)
getSentiment(output)
getCoreference(output)

• 互動言談的研究就是需要標記的好例子
• 對訪談質性研究的協助 (coding similar to annotation in qualitative research)
• qdap: (Quantitative Discourse Analysis Package) is an R package designed to assist in quantitative discourse analysis.

1. Corpus and NLP
2. Crash course for R: Regular Expression

網路資料很多 請同學上台練習。

comments <- read.table("perfumes_comments.csv", header = TRUE, 
                       sep = "\t", dec = ".", quote = "\"")
summary(comments)
# random rows of the data set
x <- sample(nrow(comments), 10, replace = FALSE)
comments[x,]

strong <-grepl("strong",comments$Comment, ignore.case = TRUE)
sum(strong)/nrow(comments)
sweet <-grepl("sweet|soft",comments$Comment, ignore.case = TRUE)
sum(sweet)/nrow(comments)