Zero-Shot Learning Capabilities of Large Language Models (LLMs) in Extracting Key Insights from Suicidal Narratives

The proliferation of social media health platforms has led to the widespread availability of data encompassing mental health discussions and personal narratives, highlighting the need for scalable and cost-effective solutions for analyzing them. Recent advancements in large language models (LLMs) have shown promise in analyzing complex textual data at scale. This study investigates the effectiveness of advanced LLMs in zero-shot settings for extracting key information and insights from emotionally charged, real-world mental health text, focusing particularly on posts related to suicidal ideation. We assess the performance of six leading LLMs - Gemini, GPT, Claude, Llama, DeepSeek, and Mistral AI - across three distinct task sets: (1) keyword extractions for identifying mental and physical health conditions, (2) binary classification for detecting references to family abuse, substance use, and prior suicide attempts, and (3) open information extraction (OIE) for identifying life events and coping mechanisms mentioned in posts. Our evaluation highlights promising performance of LLMs across the three task sets, with Gemini demonstrating the most consistent and effective performance among the LLMs. Out of the seven tasks, Gemini outperforms the other LLMs in four, although the differences are not always statistically significant when compared to the second-best LLMs, as indicated by the McNemar and Wilcoxon Signed-Rank tests. These findings underscore the potential of LLMs in the large-scale analysis of mental health data, as well as the variability in their performance, which could guide the development of effective mental health tools and interventions.