Screening Adolescent Suicidal Behavior and Risk Factors: A Machine-Learning Approach

1.1 Contextual Background

The global suicide rate has, by and large, shown a decreasing trend since 2010. However, the suicide rate in South Korea reported from 2018 to 2020 was the highest among OECD countries. The country’s suicide rate was 24.1 per 100,000 inhabitants in 2020[1]. Additionally, according to Korean Social Trends 2023, as of 2022, the percentage of suicides among the causes of death by age group was 42.3% for teenagers. Suicide ranked as the leading cause of death in this age group[2]. Beyond Korea, suicide is also recognized as a critical public health issue on a global scale[3]. Suicide is known to be the second-leading cause of death in adolescence[4], and adolescent suicide attempts are increasingly common[5], and therefore, early identification and intervention for the condition is one of the top public health priorities in the United States[6]. Suicide is of critical importance at developmental ages because the sharpest increase in the number of deaths due to suicide occurs in the transition between adolescence and early adulthood[7], and the prevalence of suicidal ideation and attempted suicide increases dramatically during adolescence[8]. Compared to adults, adolescents are more prone to impulsively contemplate or act on suicidal thoughts. Suicide attempts in this age group often occur as a means of expressing internal pain to seek help, rather than stemming from a clear intent to die by suicide. Impulsive attempts triggered by events or stressors are more prevalent in this context[9].

It is challenging to predict and prevent individual adolescents’ suicide planning or attempts in advance. Psychological distress encompasses a broad spectrum of emotional and mental health symptoms or conditions. Suicidal planning or attempts can be a significant facet of psychological distress. There is a vast volume of literature on the development and utilization of screening assessments for predicting psychological distress and/or mental health[10, 11], and in recent years quite a few studies employed novel machine learning techniques to predict psychological and/or mental states[12-15]. A recent study used a machine learning model with the random forest algorithm to predict suicide attempt risk in adolescents, reporting an AUC of 0.72, Sensitivity of 0.36, and Specificity of 0.91[16]. Another study employed an artificial neural network to predict suicidal behavior type with a final accuracy of 0.87 in a single-center retrospective observational cohort study involving 237 patients[8]. Additionally, a study comparing various algorithms found logistic regression, lasso, ridge, and random forest equally effective in distinguishing future suicide attempt cases from non-attempt cases, with AUCs ranging from 0.82 to 0.83[17].

1.2 Prediction for Korean Students at Suicide Risk

Since 2004, the Korean government has implemented Suicide Prevention Basic Plans, including the fifth plan (2023-2027)[18]. To prevent youth suicides, the Adolescent Personality and Mental Health Problems Screening Questionnaire, Third Version (AMPQ-Ⅲ) is conducted annually for 1st and 4th graders in elementary school, 1st graders in middle school, and 1st graders in high school to diagnose emotional and behavioral issues. If students are identified as at-risk based on 3 Likert-scale items inquiring about suicidal thoughts, planning, and attempts, they will be referred to specialized institutions for professional counseling and medical treatment[19, 20]. In recent years, an increasing number of studies utilizing novel techniques have emerged in the literature, offering various solutions to address pressing issues associated with suicide among adolescents and young adults in Korea. One study employed latent class growth modeling and machine learning algorithms, achieving accuracies ranging from 0.71 to 0.89[21]. Another study utilized a balanced bag of histogram gradient boosting algorithm to predict adolescents’ suicide attempts. Using data from the 2020 Korean Youth Risk Behavior Web-based Survey (KYRBWS) for training and testing, and the 2021 KYRBWS for external validation, the model achieved an AUC of 0.89 and an accuracy of 0.80 on the test dataset. Sensitivity and specificity on the external validation set were 0.77 and 0.84, respectively[22].

A big data study compared models using the KYRBWS dataset from 2014 to 2018 to predict adolescents’ suicidality, including ideation, planning, and attempts. The discriminant analysis consistently outperformed other models in accuracy, sensitivity, and specificity, with values ranging between .80 and .82 for accuracy and specificity, and .80 and .81 for sensitivity[23]. Additionally, a study utilizing the random forests machine learning algorithm on a large survey dataset comprising 46 features, with suicidal ideation as the outcome variable, repor ted an accuracy of .81/.74, sensitivity of .34/.60, and specificity of .95/.83 for male and female adolescents, respectively[24].

1.3 Objectives of the Study

From a standpoint of policy and governance, accurately predicting individual adolescents at risk of suicidal planning or attempts is imperative to deliver timely and tailored care, treatment, and educational assistance. Education from Kindergarten to 12th grade in Korea is overseen not only by the central government but also managed by 17 individual education jurisdictions. Among these, the Seoul Metropolitan Office of Education is the largest. Furthermore, the policies of each education jurisdiction can vary within their respective capacities while aligning with nationally mandated policies, allowing for flexibility and autonomy[25]. Hence, it is essential to develop high-quality predictive tools tailored for operation at a specific school level within a particular jurisdiction to fully leverage the practical relevance of research findings within educational contexts.

The primary objective of this study was to create practical screening tools utilizing machine-learning algorithms tailored for predicting suicidal behavior among middle school students in Seoul Metropolitan City. Secondly, the aim was to identify and describe the risk factors for suicidal behavior. To the best of our knowledge, there were no automated tools available to predict suicidal behavior at a specific school level tailored for a local office of education. The middle school level was selected for the study because students at this stage have been reported to experience higher susceptibility to suicidal ideation[26] compared to high school students. Moreover, early identification and intervention, if possible, would be beneficial for students at risk. Suicidal behavior in this study was defined as either planning or attempting suicide. To ensure an ample and high-quality dataset for training the machine and yielding efficient tools to be used in school-based sites, we sourced the latest data from a rigorous national survey program, the 2022 KYRBWS data, that assesses various health-related behaviors among secondary school students.

2.1 Predictive Machine Learning Models

Five predictive models including three multi-layer perceptrons were fitted to the same data to predict individual adolescent’s suicidal behavior. The first model was the logistic regression model equipped with a penalized maximum likelihood estimation method. The logistic regression is frequently used to test the association between explanatory variables and a binary outcome in an attempt to obtain a better understanding of factors that affect the outcome[27]. It can also be used for prediction purposes. The first model was fitted using the FIRTH and PEVENT options in SAS version 9.4, with the specific cut point set at 0.06. These options are typically employed to address the challenge of unbalanced data, where the distribution of classes is uneven. The chosen cut point of 0.06 served as a threshold for making classification decisions.

The second model was a logistic regression model utilizing the logistic regression algorithm from the scikit-learn machine learning library in Python[28]. Its optimization algorithm is known to approximate the Broyden–Fletcher–Goldfarb–Shanno algorithm[29, 30], which belongs to quasi-Newton methods. The last three models, known as multi-layer perceptrons, were sequential neural network models with different structures frequently used for non-image data classification. They were fitted using the Tensorflow and Keras libraries in Python. It is common practice to experiment with adding hidden layers or dropout layers in order to enhance the performance of deep learning models[31].

The three deep-learning models shared similarities in terms of the output layer configuration, loss function, and optimizer. They all used binary crossentropy as the loss function, and accuracy as the evaluation metric, and shared the same optimizer, adam for training. However, they differed in the architecture of the hidden layers, with variations in the number of neurons and the inclusion of dropout layers. The third model had two hidden layers with 8 neurons each while the fourth model had three hidden layers with 64, 32, and 16 neurons, respectively. The fifth model, which was the most complex model, had three hidden layers with 64, 32, and 16 neurons, but it included dropout layers between each dense layer. These five models were referred to as Model_1 to Model_5 hereafter.

The four models, excluding Model_1, utilized machine learning algorithms provided by Python 3.11.5. These models were fitted to the data using a class weight option each to address the issue of unbalanced classification. Model_2 employed the compute_class_weight() function from the scikit-learn library in Python to calculate weights based on the inverse of the class frequencies in the training data. In contrast, Model_3 to Model_5 utilized class weights, where the reference group had a weight of 1.0, and the event group had a weight of 14.6, respectively.

2.2 Data Preprocessing and Feature Selection

The study utilized data from the 18th KYRBWS database in 2022, an annual survey since 2005 assessing health behaviors among South Korean secondary school students. The database is managed by the Korea Disease Control and Prevention Agency (KDCA). Sampling for the 2022 survey ensured population representativeness, comprising middle and high school students nationwide as of April 2022. A total of 114 items were investigated, covering smoking, drinking, physical activity, dietary habits, obesity and weight control, mental health, oral health, personal hygiene, injury prevention, violence, sexual behavior, substance abuse, atopy and asthma, internet addiction, and general characteristics[32]. Items with high missing values were excluded. Smoking and drinking items were recategorized. A new variable, ‘Drink/smoke,’ was created with the following codes: 0 for no smoking or drinking, 1 for having tried drinking but not smoking, 2 for having tried smoking (whether combustible or electronic) but not drinking, and 3 for having tried both smoking and drinking. Another new variable, ‘Suicide,’ was created to represent whether a student planned or attempted suicide in the past 12 months, coded as 1 for yes and 0 for no. It served as the outcome variable or target in the model training.

To ensure effective screening tools, providing machines with high-quality and sufficient data is crucial[33]. In the machine learning process, it’s common to optimize and select variables before training to improve predictions and address issues like overfitting. Logistic regression analysis with forward selection was employed to identify the optimal predictor variables for model training. Features for Model_2 to Model_5, trained using Python, were normalized using the Min-Max Scaling method for consistent scales during data preprocessing.

2.3 Model Evaluation

Regardless of the specific algorithm used, the aim of machine learning techniques in prediction is to optimize predictive capacity autonomously. Model performance is evaluated based on its ability to predict outcomes in new cases not used during training. This study focused on the following metrics to evaluate prediction capacity since it was crucial to identify at-risk students at the expense of false alarms. Accuracy measures the proportion of correct predictions in an unseen test dataset. Sensitivity represents the proportion of correctly identified positive cases, while specificity refers to correctly identified negative cases. AUC, the area under the ROC curve, summarizes classifier performance across different thresholds. These metrics collectively gauge the effectiveness of a binary classification tool.

2.4 Importance of Features

SHAP values measure the contribution of each feature to a model’s prediction[12, 34]. Average absolute SHAP values for categorical variables provide insights into their impact, though interpreting their sign may be less straightforward. In this study, average absolute SHAP values were calculated for Models_2 to Model_5, while Model_1 compared standardized regression coefficients (beta) across predictor variables to assess importance.

2.5 Cross-Validation

For each model, the cross-validation process occurred in two stages. Initially, the data was split into training and held-out test sets at an 80:20 ratio. Subsequently, a 10-fold cross-validation was conducted during training using the StratifiedKFold function from scikit-learn. This involved partitioning the training data into 10 folds, using each fold as a validation sample while training the machine learning models on the remaining data. The two best-fitted estimation results across the 10-fold iterations for each predictive model were then applied to the unseen held-out test dataset. The better results from this testing phase were reported as the final outcomes for the model. The k-fold validation approach is widely recognized as a common practice[35, 21].

2.6 Institutional Review Board Approval

The research was exempted from the need for review by the Korea Disease Control and Prevention Agency and our institution because open-access datasets were utilized.

3.1 Baseline Characteristics of Participants

Middle school students (n = 3,812) from Seoul Metropolitan City were selected from the 2022 KYRBWS database. Both males and females were approximately evenly distributed. The grade distribution also appeared similar, ranging between 32% and 34% (Table 1). Suicidal behavior (suicidal planning or attempts) was the outcome variable. Binary logistic regression with forward selection utilized 45 predictor variables, excluding some due to high missing rates (close to or exceeding 20%) or substantial overlap with already included variables. Five predictor variables had missing responses between 0.05% to 2%. Hence, no imputation methods were used. The best-performing model had nine predictors. No missing data were found in these predictors or the outcome variable. The dataset was split into training and test sets (80:20 ratio), followed by 10-fold cross-validation during training. Python normalized features for the last four machine-learning models. Only 6.43% of cases reported that they had planned or attempted suicide in the past 12 months.

Table 1.

Baseline Characteristics of Study Participants, Model Features and Target Variable

3.2 Predictability of Screening Tools

Each of the five models addressed the dataset’s imbalance, with 94% of students without suicidal behavior compared to 6% with such tendencies. Moel_1 used FIRTH and PEVENT=0.064 options in SAS PROC LOGISTIC, while others used built-in class weight options in Python. Evaluation metrics for all models are presented in Table 2, prioritizing sensitivity over accuracy and specificity due to data imbalance. Model_5 emerged as the top performer, with Models 1 and 4 also performing well. These models performed overall better in AUC, accuracy, sensitivity, and specificity as compared to previously reported ones which had aimed to identify suicidal tendencies among Korean adolescents[21-24]. Figure 2 shows ROC curves for the four Python machine-learning models.

Table 2.

Performance of Five Machine-Learning Models Predicting Adolescent Suicidal Behavior

Figure 1.

Average Absolute SAHP Values Illustrating the Importance of FeaturesNote. Shapley additive explanations (SHAP) values were computed for the four models using algorithms available in Python.

Figure 2.

ROC Charts for Machine Learning Models Predicting Youth Suicidal BehaviorNote. (A) Model_2. (B) Model_3. (C) Model_4. (D) Model_5. The ROC charts display the performance of the four models, each utilizing algorithms available in Python.

3.3 Risk Factors for Suicidal Behavior

The average absolute SHAP values were computed for Model_2 to Model_5 to measure each feature’s impact, considering its interaction with others, shown in Figure 1. Notably, Suicidal thought had the most significant impact on all four Python models. Examining Model_5, it was followed by the features Feel sad and Feel lonely, and then by the features Drink/smoke and Dental health. These features also appeared to exert relatively stronger impacts on the predictability of Model_4, although the magnitude and order of impact varied slightly between the two models. On the other hand, for Model_1 fitted using SAS PROC LOGISTIC, the best-performing model associated with the dataset in the third fold exhibited the largest absolute standardized regression coefficients in the predictors, Suicidal thoughts, Dental health, Feel lonely, and Feel sad, in descending order, all of which were statistically significant at the 0.05 significance level. It is worth noting that the features were normalized in the last four models to improve predictability, while Model_1 treated the categorical predictor variables as such.

Examining odds ratios from Model_1 revealed insights that adolescents considering suicide were approximately 12.7 times more likely to make plans or attempt suicide than their peers without such ideation (OR = 12.7; 95% CI = [8.0-20.1]). Those perceiving dental health as ‘Very bad’ were 4.4 times more prone to suicidal behavior than those with ‘Average’ ratings (OR = 4.4; 95% CI = [1.4-13.5]). Similarly, those always feeling lonely were 2.8 times more likely to engage in suicidal behavior (OR = 2.8; 95% CI = [1.1-7.1]), while persistent feelings of sadness increased the likelihood by 1.8 times (OR = 1.8; 95% CI = [1.2-2.8]).

4.1 Highlighting Study Outcomes

Youth suicide is a significant healthcare and educational concern in many countries. Predicting high-risk adolescents for suicide planning and attempts and providing timely interventions is a policy priority. Efforts are ongoing to develop screening tools for schools to enhance policy effectiveness. However, the literature suggests current prediction accuracy remains low, often no better than chance[36], or requires improvement[16, 17, 21-24].

The study aimed to develop effective screening tools for middle school students in Seoul, predicting suicide risk. Features were included based on potential performance enhancement, resulting in a set of nine features selected through forward selection. The machine learning models under investigation showed higher sensitivity and AUC while maintaining comparable accuracy and specificity to previously reported models. Logistic regression, using SAS, performed similarly to Python-based models, but appropriate options like FIRTH and PREVENT were crucial due to data imbalance[37]. However, logistic regression’s performance with more than two outcome classes remains uncertain.

The SHAP values for interpreting black-box machine learning models revealed that factors such as suicidal ideation and affective variables like Feel lonely and Feel sad had significant impacts on predicting students at risk of planning or attempting suicide. These features also emerged as top risk factors for suicidal behavior, confirmed by the betas of the statistical logistic regression model, Model_1, with Dental health as the second strongest risk factor.

The suicide of adolescents is reported to be attributed to not a single individual factor but to multidimensional and complex factors related to personal, family, and school contexts[38]. It is valuable to touch on the nine features to deepen our understanding of their relationships with suicidal behavior and ultimately aid in devising educational and health policies for suicide prevention. While not all individuals who contemplate suicide proceed to plan or attempt it, those who engage in such contemplation or planning face an increased risk of eventual suicide.

Persistent suicidal thoughts are acknowledged as a contributing factor, particularly in impulsive suicide attempts[3, 39, 40]. In the realm of mental health, emotions such as feeling lonely, sad, and distressed are frequently encompassed within the broader concept of psychological distress[41]. These emotional states are considered factors directly influencing suicidal behavior, with origins in both genetic predisposition and environmental influences. This dichotomy underscores their complex nature as contributors to the risk of adolescent suicide.

On the other hand, factors such as dental health, substance use (drink/smoke), fast food consumption, and sexual experiences could be viewed more as indirect indicators connected to the hazardous environmental circumstances that adolescents may find themselves in, such as lacking adequate care. Consequently, they are considered indirect proxies linking to the aforementioned emotional states (feel lonely, feel sad, feel distressed), which have a direct impact on suicidal behavior.

Elaborating on dental health further, adolescence is often considered the healthiest period throughout one’s lifetime, and common health issues experienced during school years include dental conditions such as dental caries. Poor dental health or a negative perception of dental health is reported to be associated with suicidality[42, 43]. Several studies have indicated that the dental health of adolescents is related to mental health, health behaviors, and the socioeconomic factors of parents, and that experiencing more dental symptoms is associated with an increased risk of suicide-related factors[44,45,46]. Highlighting the prevalence of asthma as a chronic condition, particularly among younger individuals, a potential association with suicide mortality, ideation, and attempts has been reported across age groups[47]. Asthma’s impact on the quality of life, manifesting in symptoms detrimental to emotional well-being, is seen as contributing to or being linked with the emotional states of feeling lonely, sad, and distressed, thus connecting to adolescent suicide behavior.

Adolescence is a crucial period where the foundation for lifelong health management is established physically, mentally, and emotionally. Consequently, rather than addressing suicide prevention, mental health, and dental health as separate issues, it would be advisable to develop integrated educational programs aimed at recognizing students’ health risk factors and enhancing their health capabilities.

4.2 Strengths and Limitations

The strengths of this study lie in several key aspects. Firstly, it developed suicide behavior screening tools tailored to a specific school level within a local educational jurisdiction to enhance their suitability for school settings. According to prior literature, while prediction models developed using nationwide data have been reported, no locally customized models like those developed in this study have been documented. The tools developed in this study exhibited improved predictive performance compared to previously published models. Suicide is not simply an individual problem but a social issue, resulting from the complex interaction of various factors, that should be collectively addressed by the entire nation and local communities. The significant regional differences in suicide rates indicate the importance of developing tailored screening tools for evidence-based policies at the local level. Secondly, leveraging the national database established through rigorous sampling practices minimized bias in the development of the screening tools, consequently improving the generalizability of the study’s outcomes. Thirdly, the developed tools do not require creating new survey items for screening to identify or predict students at risk of suicidal planning or attempts. They solely rely on students’ responses to twelve items from an existing survey questionnaire. Fourthly, an elaboration on risk factors, including individuals’ perceptions of dental health and mental health characteristics, was provided to inform the development of educational and health policies aimed at suicide prevention. Fifthly, the developed tools can address the limitations of the national screening test AMPQ-Ⅲ, which only consists of three suicidality items requiring a follow-up human interview and is used only for freshmen in middle schools. Lastly, if one aims to develop similar screening tools for binary targets with severe imbalance, this study provided guidance by presenting the candidate algorithms and proper options.

Regarding limitations, the developed prediction tools need external validation in future studies. The study performed 10-fold cross-validation during the training phase and conducted additional validation using an unseen held-out test dataset. Despite the cross-validation, they need to be applied in an independent study with new students. Additionally, the factors associated with suicidal behavior that were included in the analytical models did not cover various variables at multiple levels, as shown in some suicide research. This was because the study primarily focused on developing a screening tool to identify students at risk, and the final set of variables was chosen through a data-driven method to maximize the predictability.