What can text data on goodreads tell us? Data analytics for predictive modelling.

Could we use the readability and sentiment of reviews to understand user preferences and make better book recommendations? It seemed simple enough on the surface—analyze the text, derive some meaningful metrics, and uncover insights. What followed was a deep dive into text analysis, computational challenges, and a host of lessons learned.

Link to My original Paper and codebase: Here

The Starting Point: Why Readability and Sentiment?

Online reviews are a cornerstone of digital decision-making, from picking a book to buying a gadget. On Goodreads, book reviews are rich in personal opinions, diverse writing styles, and sentiment. My curiosity was piqued by two questions:

1. Does the readability of reviews correlate with the types of books users enjoy?
2. Could sentiment analysis uncover patterns that help connect readers with similar preferences?

1. Measuring Readability

To quantify readability, I used the Flesch Reading Ease score, a widely-used metric that calculates how easy a text is to read. The formula is:

$$\text{Flesch Reading Ease} = 206.835 - 1.015 \left( \frac{\text{total words}}{\text{total sentences}} \right) - 84.6 \left( \frac{\text{total syllables}}{\text{total words}} \right)$$

• Intuition: Higher scores indicate simpler text (e.g., children's books), while lower scores signify more complex writing (e.g., academic or literary works).
• Implementation: Using Python’s textstat library, I calculated the Flesch scores for each review. This provided a numeric representation of the complexity of the language used.

2. Sentiment Analysis

Sentiment analysis aimed to capture the emotional tone of reviews. For this, I used the VADER (Valence Aware Dictionary and sEntiment Reasoner) sentiment analyzer, which is particularly effective for social media-style text and informal language. VADER outputs:

• 1. Polarity Scores: Positive, neutral, and negative sentiment scores for each review.
• 2. Compound Score: A normalized metric (-1 to +1) summarizing the overall sentiment.

This step revealed the underlying emotions in reviews:

• - Positive Sentiment: Reviews with enthusiastic or glowing tones.
• - Negative Sentiment: Critical or disappointed reviews.
• - Neutral Sentiment: Objective, factual, or unemotional reviews.
• 
  

3. Combining Features for Clustering

Once I had the readability scores and sentiment metrics, the next step was to analyze them together. I normalized the features (to ensure scale uniformity) and applied k-means clustering to group users into clusters with shared characteristics. The features used for clustering included:

• Flesch Reading Ease.
• VADER sentiment scores.
• VADER polarity scores
• 
  

Challenges Encountered in Clustering:

• 1. Dimensionality Issues: Adding multiple features increased complexity, and feature scaling was critical to ensure no single metric dominated the clustering process.
• 2. Baseline Comparisons: Clustering was compared against random groupings to evaluate its performance. While some patterns emerged, the results often struggled to outperform random baselines, indicating room for refinement.

Results and Key Insights

1. Readability Patterns

While readability scores provided an interesting perspective, they were not strong predictors of user preferences. For example:

• - Users who wrote simpler reviews didn’t necessarily prefer light fiction, and vice versa.
• - Many reviews with high readability scores were neutral or objective, making it difficult to draw actionable insights from readability alone.

2. Cluster Analysis

Combining readability and sentiment revealed patterns that wouldn’t have been apparent otherwise. For example:

• - A cluster of users who wrote highly readable and positively toned reviews often gravitated toward bestselling novels.
• - Users with complex reviews and mixed sentiment preferred niche literary works.
• 
  
• 
  
  
  

Final Thoughts

• This research journey taught me that data is only as valuable as the story it tells. Analyzing Goodreads reviews provided a unique lens to understand how text complexity and emotional tone influence user behavior. While the findings are a small step toward more personalized recommendations, they open doors for deeper exploration. Although the results of this research were not what we hoped to be and failed to outperform random sampling it taught me important statistical data analysis techniques and experience working with a massive dataset. 

  If you’re interested in the technical details or the dataset, feel free to reach out or check out the project and the paper on Github. I’d love to hear your thoughts or collaborate on future enhancements!
  
  
  

•