In this practical, we are going to use the following packages to create document-term matrices on BBC news data set and apply LDA topic modeling.
library(tidyverse) library(tidytext) library(tm) library(e1071) library(topicmodels) library(stringi)
This data set consists of 2225 documents from the BBC news website corresponding to stories in five topical areas from 2004 to 2005. These areas are:
1. Use the code below to load the data set and inspect its first rows.
2. Find out about the name of the categories and the number of observations in each of them.
3. Convert the data set into a document-term matrix and use the
findFreqTerms function to keep the terms which their frequency is
higher than 10. It is also a good idea to apply some text preprocessing
before this conversion: e.g., remove non-UTF-8 characters, convert the
words into lowercase, remove punctuation, numbers, stopwords, and
4. Partition the original data into training and test sets with 80% for training and 20% for test.
5. Create separate document-term matrices for the training and the test sets using the previous frequent terms as the input dictionary and convert them into data frames.
6. OPTIONAL: Use the cbind function to add the categories to the train_dtm data and name the column y.
7. OPTIONAL: Fit a SVM model with a linear kernel on the training data set. Predict the categories for the training and test data.
Latent Dirichlet allocation (LDA) is a particularly popular method for fitting a topic model. It treats each document as a mixture of topics, and each topic as a mixture of words. This allows documents to “overlap” each other in terms of content, rather than being separated into discrete groups, in a way that mirrors typical use of natural language.
8. Use the LDA function from the
topicmodels package to train an LDA
model with 5 topics with the Gibbs sampling method.
tidy() method is originally from the
(Robinson 2017), for tidying model objects. The
provides this method for extracting the per-topic-per-word
probabilities, called “beta”, from the LDA model. Use this function and
check the beta probabilites for each term and topic.
10. Use the code below to plot the top 20 terms within each topic.
lda_top_terms <- lda_topics %>% group_by(topic) %>% slice_max(beta, n = 20) %>% # We use dplyr’s slice_max() to find the top 10 terms within each topic. ungroup() %>% arrange(topic, -beta) lda_top_terms %>% mutate(term = reorder_within(term, beta, topic)) %>% ggplot(aes(beta, term, fill = factor(topic))) + geom_col(show.legend = FALSE) + facet_wrap(~ topic, scales = "free") + scale_y_reordered()
11. Use the code below to save the terms and topics in a wide format.
beta_wide <- lda_topics %>% mutate(topic = paste0("topic", topic)) %>% pivot_wider(names_from = topic, values_from = beta) %>% mutate(log_ratio21 = log2(topic2 / topic1)) %>% mutate(log_ratio31 = log2(topic3 / topic1))%>% mutate(log_ratio41 = log2(topic4 / topic1))%>% mutate(log_ratio51 = log2(topic5 / topic1)) beta_wide
12. Use the log ratios to visualize the words with the greatest differences between topic 1 and other topics. Below you see this analysis for topics 1 and 2.
# topic 1 versus topic 2 lda_top_terms1 <- beta_wide %>% slice_max(log_ratio21, n = 10) %>% arrange(term, -log_ratio21) lda_top_terms2 <- beta_wide %>% slice_max(-log_ratio21, n = 10) %>% arrange(term, -log_ratio21) lda_top_terms12 <- rbind(lda_top_terms1, lda_top_terms2) # this is for ggplot to understand in which order to plot name on the x axis. lda_top_terms12$term <- factor(lda_top_terms12$term, levels = lda_top_terms12$term[order(lda_top_terms12$log_ratio21)]) # Words with the greatest difference in beta between topic 2 and topic 1 lda_top_terms12 %>% ggplot(aes(log_ratio21, term, fill = (log_ratio21 > 0))) + geom_col(show.legend = FALSE) + scale_y_reordered() + theme_minimal()
13. Besides estimating each topic as a mixture of words, LDA also
models each document as a mixture of topics. We can examine the
per-document-per-topic probabilities, called “gamma”, with the
matrix = "gamma" argument in the
tidy() function. Call this function
for your LDA model and save the probabilities in a varibale named
14. Check the topic probabilities for documents with the index number of 1, 1000, 2000, 2225.
15. Use the code below to visualise the topic probabilities for the example documents in question 14.
# reorder titles in order of topic 1, topic 2, etc before plotting lda_documents[lda_documents$document %in% c(1, 1000, 2000, 2225),] %>% mutate(document = reorder(document, gamma * topic)) %>% ggplot(aes(factor(topic), gamma)) + geom_boxplot() + facet_wrap(~ document) + labs(x = "topic", y = expression(gamma)) + theme_minimal()
The LDA() function in the topicmodels package is only one implementation of the latent Dirichlet allocation algorithm. For example, the mallet package (Mimno 2013) implements a wrapper around the MALLET Java package for text classification tools, and the tidytext package provides tidiers for this model output as well. The textmineR package has extensive functionality for topic modeling. You can fit Latent Dirichlet Allocation (LDA), Correlated Topic Models (CTM), and Latent Semantic Analysis (LSA) from within textmineR (https://cran.r-project.org/web/packages/textmineR/vignettes/c_topic_modeling.html).