Using Natural Language Processing in Behavioral Science: A How-To-Guide
As NLP becomes a popular analysis technique, it is important to use it wisely. We offer a How-to-Guide from our new paper.
Every day, millions of people communicate through text, via emails, social media posts, online reviews, and news articles. In fact, the average person scrolls through nearly 300 feet of online content daily, which would be the same as reading the New York Times three times over every day or scrolling the height of the Statue of Liberty!
These text data contain valuable information about people’s personality traits, emotions, social group membership, among other psychological variables. Natural Language Processing (NLP), enables psychologists and other scientists to analyze massive amounts of text to uncover these psychological patterns, and study human interactions at a scale never before possible.
In a review published in Nature Reviews Psychology, we lay out how NLP can be used to study human psychology and behavior, describing methods, trade-offs, and processing pipelines. For this newsletter, we summarize the ways NLP is used in psychological science, important considerations when using such methods, and our recommendations for best practices.
Applications of NLP in Behavioral Science
There are three main ways that psychologists and behavioral scientists use NLP for text analysis:
1. Exploratory Content Analysis
Exploratory content analysis uses NLP to identify patterns, themes, or insights from large text datasets without an a-priori hypothesis. Examples include identifying common topics in social media discussions, tracking shifts in public sentiments over time (such as changes in attitudes toward climate change), and uncovering discourse patterns, such as how political rhetoric evolves during election campaigns. For these analyses, it’s important to distinguish between words and concepts: words are symbols that can represent concepts, while concepts are collective understandings that are shaped by their use in society. Researchers must keep in mind that the meaning of a word can be context-dependent and may change over time. In other words, the literal meaning of a word sometimes differs from its implied meaning in context.
2. Annotating Text by Psychological Constructs
A critical goal in analyzing text data is identifying psychological constructs. For example, researchers might analyze text to determine its emotional sentiment, assess a person’s personality traits based on their language use, or detect signs of depression and anxiety in social media posts. Different approaches have different tradeoffs. For example, the key-word based approach involves counting the frequency of keywords such as “angry” and “sad”. While it is a reliable and easily interpretable method, it often fails to capture the context or semantics of the text. In contrast, Large Language Models (LLM) use machine learning methods to process and generate text. They are highly efficient and more accurate because they consider both context and semantics. However, they are often considered “black boxes”, making it difficult to interpret how they arrive at their decision.
3. Relating Psychological Constructs to Behavioral Outcomes
NLP-generated insights are often integrated into statistical models to study how language relates to behavior. Examples include analyzing how moral-emotional language in social media posts influences engagement (e.g., posts expressing anger or outrage tend to receive more shares), whether certain linguistic features are linked to specific mental health states, and how political rhetoric correlates with changes in public opinion.
Beyond explanation, NLP is also used for prediction. For instance, researchers can develop machine learning models to forecast mental health crises based on people’s online language use, to assess the risk of violent behavior by analyzing radicalized rhetoric, and to predict the effectiveness of persuasive messaging in marketing and public health campaigns. Evidently, these predictive models have important real-world applications, such as serving as an early warning system for harmful online behavior. However, they come with ethical concerns—such as the risk of misinterpreting signals and privacy violations—must be carefully addressed by researchers.
All NLP methods have their strengths and limitations. Researchers should therefore not overly rely on or interpret any single method. Common best practice is to combine multiple methods–using simple interpretable approaches alongside advanced models–to enhance the robustness of their results. Transparency and reproducibility are equally important. This includes adopting open science practices, clearly defining how psychological constructs are measured, and thoroughly reporting all aspects of the used models. Research using NLP methods should also be firmly grounded in theoretical frameworks, using theory as a benchmark to validate findings.
Challenges and Ethical Considerations
Despite the proliferation of NLP methods in all areas of psychology, NLP isn’t a magical tool that can be used indiscriminately; it has several limitations.
For example, LLMs and other NLP models inherit biases from their training data, which can lead to reinforcing stereotypes, and misinterpreting certain dialects. To mitigate these biases, researchers leverage bias detection tools and use diverse and representative datasets, but eradicating bias in NLP models entirely remains a challenge. Additionally, proprietary LLMs (like ChatGPT) change frequently and in unclear ways, rendering it difficult to perfectly replicate studies. Open-source models provide more transparency but often lag behind in performance.
The data used for NLP analysis is often sourced from online platforms like social media. However, users may be unaware that their data is being collected and analyzed. When web scraping certain websites or platforms, researchers should check the terms of data usage in user agreements, and remove personally identifiable information when handling sensitive information. Ethical frameworks for how to use online public data are still evolving. It is critical that these frameworks address the potential effects of research as well as the negative effects and how they can be minimized.
However, as NLP approaches continue to evolve, new opportunities arise to mitigate their current limitations. Open-source models are rapidly improving, with recent models like DeepSeek rivaling GPT’s performance at a significantly lower cost. Utilizing diverse data and complementing NLP’s with traditional survey or behavioral methods can further strengthen the robustness of research findings. Our full review offers several actionable recommendations to ensure the success and reproducibility of NLP-based analysis in behavioral science.
To learn more:
Citation: Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, Claire., Rathje, S., Hartmann, J., Mohammad, S., Netzer, O., Siegel, A., Plank, B., & Van Bavel. J. J. (2025). Using natural language processing to analyze text data in behavioral science. Nature Reviews Psychology. www.nature.com/articles/s44159-024-00392-z
Recent Events
Jay spoke at the University of Texas at Austin on March 7 as part of a conference on what it looks like to engage in healthy democratic debate and discussion.
Outreach
Jay was interviewed for an article in PNAS about whether science-based interventions can help tamp down the rising polarization sweeping the globe. Check out his full quote below! You can read the full article here.
Postdoc Rémi Thériault secured a € 23,500 Francisco J. Varela Research Award grant from Mind & Life Europe for the lab! This will fund Rémi's project, "Reading Books Changes Lives: Investigating the Effects of Popular Psychology Books on the Development of Moral Character and Action".
Rémi will also be presenting at NYU’s Spring 2025 Scholar speaker series; see below for more information and to RSVP! Thursday, March 20th, 12:00 PM - 2:00 PM
Join OGS at our Spring 2025 Scholar Speaker Series, an event we hold each semester to celebrate the work of our international visiting scholars! This is a great opportunity to network, collaborate, and mingle with people from all departments and fields, all while enjoying a light lunch and refreshments! Our two speakers, Dr. Eng. Andrzej Głuchowski and Rémi Thériault, will present their current research all relating to our theme: Strengthening Our Foundations: Altruism & Geotechnics, detailing ways to strengthen the literal and social foundations that make up our daily lives. Please fill out this form to RSVP!
Jay was interviewed for an article in Scientific American, where he discusses our work on how fact-checking is less effective when the issue is politically polarized. Read the full article here.
Lab alum Jesper Rasmussen recently secured a position as an Assistant Professor of Political Science at Aarhus University. Congrats, Jesper!
Jesper and colleagues Lasse Lindekilde and Michael Bang Petersen also recently published a paper in the Journal of Experimental Political Science showing that investing in citizens' communication competences through actionable communication from health authorities, can decrease Covid-19 misinformation sharing. Read the full paper here: bit.ly/3UuKZJb
Jay was recently featured in a Lavin agency clip, where he talked about giving to members of our community makes us feel good. Check it out here!
This newsletter was created from a summary of the below article and edited by Sarah Mughal, Anni Sternisko and Jay Van Bavel.