Development and Validation of a Task-Strategy Scale for Profiling Generative AI Use Among Undergraduates at an Engineering-Oriented University

2.1 Generative AI Use in Engineering Education

Research on generative AI use in higher education initially focused on how learners used generative AI for academic purposes. These studies found that students used generative AI for a range of activities, including assignment completion, information retrieval, and idea generation [1,9]. At the same time, learners were found to recognize both the convenience and usefulness of generative AI while also expressing concerns about output reliability and ethical use. Issues related to verification have also remained prominent in the literature. These include concerns about the accuracy of AI-generated outputs, the opacity of information sources, and the possibility that learners may accept AI-generated content uncritically [1,4,7]. Taken together, these findings suggest that a more precise understanding of generative AI use in this learning environment requires going beyond frequency-based indicators to capture both task-related use patterns and strategic regulatory behaviors.

Much of the existing research on generative AI use has adopted a technology acceptance perspective, focusing primarily on intention to use and frequency of use [10,11]. However, this perspective is limited in explaining the specific academic tasks for which learners use generative AI and how they regulate their interactions with it during task performance.

The need for more precise measurement becomes particularly evident in engineering education. Compared with students in non-engineering disciplines, engineering undergraduates use generative AI more frequently for producing technical outputs [2-3], especially in tasks that require high accuracy, such as code generation and data analysis. Given the comparatively high proportion of accuracy-critical tasks in engineering curricula, it is necessary to consider not only the diversity of task types but also the importance of verifying AI-generated outputs.

2.2 Task Types in Generative AI Use

A task-type perspective on generative AI use offers a multidimensional way of understanding the academic tasks for which learners use generative AI. It is based on the premise that generative AI use may take different forms depending on the nature of the task and the learning context [2,3]. Prior studies have repeatedly shown that generative AI use varies according to task characteristics. Specifically, Ravšelj et al. [1] and Oh [9] found that university students’ AI use tends to concentrate on text-based tasks such as information retrieval, drafting, and summarization. Singer-Freeman et al. [2] also found that generative AI use varies across academic tasks in both frequency and pattern, with relatively frequent use for study preparation tasks and comparatively lower use for writing and data-related tasks. In addition, Stöhr et al. [3] derived usage profiles by measuring the degree of students’ reliance on AI across multiple tasks, showing that task type can serve as a key criterion for distinguishing patterns of AI use. Taken together, these findings suggest that task type should be understood not as a simple frequency variable, but as a multidimensional construct.

However, most previous studies, even when providing lists of tasks, have not systematically reflected them in a measurement structure [1-2,9]. In particular, non-text tasks frequently performed by engineering learners, such as data analysis, visual material generation, and structural formatting of documents, have not been sufficiently addressed. Thus, prior task classifications have focused mainly on general text-based writing and have therefore been limited in covering the non-text tasks distinctive to engineering education.

In the context of engineering education, generative AI use can be broadly understood in terms of two task types. Prior research has shown that university students often use generative AI for text-based academic activities, such as information retrieval, drafting, summarization, and learning support [1,2,9], which in this study are conceptualized as text-based task use. At the same time, engineering learners also use generative AI for more structured forms of output production, including code generation, data analysis, visual or video material generation, and document formatting, which are conceptualized here as media and data task use. The latter category is particularly relevant to engineering education and also includes document formatting, as it involves rule-based structural processing rather than simple text generation.

2.3 Advanced Use of Generative AI

The qualitative differences in how learners engage with generative AI cannot be adequately explained by usage volume or task categories alone. Even when performing identical tasks, the quality of AI use and the reliability of its outputs vary considerably depending on how learners construct and refine their prompts, how they select among multiple generated outputs, and how they verify the accuracy and sourcing of AI-generated content [4-8,11-14]. Recent research on AI literacy has increasingly emphasized critical judgment and self-regulatory behaviors as core dimensions of AI use competence, approaching these capabilities from a multidimensional perspective [12-14]. However, few studies have examined these capacities together with the concrete interaction strategies learners use during actual academic task performance [2,3].

Research on prompting behavior has directly addressed the strategic aspects of interaction with generative AI. Gibreel and Arpaci [6] proposed prompt construction and adjustment ability as a measurable competence dimension, while Lee and Palmer [5] showed that prompt design and revision are not merely input techniques but strategic acts involving goal specification, constraint setting, and the comparison and selection of outputs. These findings suggest that the ability to regulate the generative phase of AI interaction represents a distinct and assessable strategic construct.

Alongside the forms of control during the generative phase, verification and critical evaluation of outputs also need to be treated as a separate strategic dimension. Yan et al. [8] identified information verification behavior as a key outcome variable and demonstrated its importance in the context of generative AI use. Domestic studies have likewise highlighted the risks of uncritical acceptance of AI-generated content and the need for source checking and reliability assessment [4,7]. Overall, previous studies suggest that strategic behavior in generative AI use should not be treated as a single dimension, but should instead be conceptualized in terms of prompting and selection during generation and critical verification after generation.

2.4 Synthesis of Previous Studies

Previous research on generative AI use can be grouped into four main lines of inquiry. First, AI literacy-centered studies have examined AI use competence from cognitive, behavioral, ethical, and evaluative perspectives [12-14]. Second, AI use behavior studies have explained intention to use and frequency of use from a technology acceptance perspective [10,11]. Third, task type-centered studies have analyzed the specific academic tasks for which learners use generative AI [1-3,9]. Fourth, studies on prompting strategies, verification, and evaluative behavior have highlighted the importance of prompt refinement [5,6] and the need for critical evaluation and information verification of AI-generated content [4,7,8]. Table 1 presents a thematic synthesis of these strands of research, their major findings, and their implications for item development and measurement structure design in this study.

Table 1.

Thematic Synthesis of Key Studies on GenAI

3.1 Research Procedure

To develop and validate the Task–Strategy Scale for Profiling Generative AI Use, this study proceeded in five phases.

First, generative AI use in academic learning contexts was organized into two broad domains, task-related use and strategy-related use, and an initial item pool was constructed on that basis. Items for the task-related domain were developed with reference to Singer-Freeman et al. [2] and Ravšelj et al. [1], yielding 15 candidate items across four subfactors. Items for the strategy-related domain were developed with reference to Lee and Palmer [5], Gibreel and Arpaci [6], and Yan et al. [8], yielding 10 candidate items across three subfactors. Second, to establish content validity, a panel of five experts in education, communication, and educational measurement reviewed the initial items. Based on the quantitative results and qualitative feedback, abstract terms that were not appropriate for the student level were revised, and the items were reorganized to reflect actual generative AI use contexts in engineering education. Third, the items for the main survey were finalized, and data were collected from 201 undergraduates enrolled at an engineering-oriented university. Fourth, item analysis, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA) were conducted to examine the integrated factor structure. Fifth, reliability and both convergent and discriminant validity were assessed for the finalized factor structure.

3.2 Scale Development

3.2.1 Preliminary Item Generation

Preliminary item generation began with a broad initial pool of candidate items designed to cover the conceptual components of both task-related use and strategy-related use as comprehensively as possible. The initial item pool was then refined through expert review and empirical analysis (see Table 2).

Table 2.

Preliminary Item Pool and Construct Definitions

The task-related domain was conceptualized as a categorization of specific academic activities in which learners integrate generative AI into disciplinary learning. The initial pool for this domain consisted of 15 items across four preliminary subfactors: Writing and Editing, Idea Generation, Learning Support, and Technical and Creation.

The strategy-related domain was conceptualized as a pattern of active human–AI collaborative behavior through which learners optimize learning outcomes by refining prompts iteratively and verifying outputs. The initial pool for this domain consisted of 10 items across three preliminary subfactors: Prompt Engineering, Verification Behavior, and Critical Choice.

3.2.2 Content Validity Review

A content validity review was conducted to verify whether the survey items developed by the research team adequately reflected the purpose and target population of the study. The composition of the expert panel is presented in Table 3. The panel was asked to evaluate the content validity of all 25 items, including 15 items from the task-related domain and 10 items from the strategy-related domain.

Table 3.

Characteristics of the Expert Panel(N=5)

To support the experts’ evaluation, the review materials included both the original English source items and a preliminary Korean translation prepared by the research team, as several of the constructs represented in these items have not yet been clearly established in the Korean context. Experts rated each item on a five-point Likert scale ranging from 1 (not appropriate at all) to 5 (very appropriate). For any item rated as not appropriate at all, not appropriate, or neutral, experts were asked to provide the reason for the rating and suggest directions for revision or improvement. In addition, they were invited to record any recommendations regarding item addition, integration, separation, deletion, or modification in an open-ended comment section.

3.2.3 Survey Administration

Data were collected via an online survey administered from February 4 to February 12, 2026, targeting currently enrolled students at T University, a specialized four-year technology institution located in the Seoul metropolitan area. Prior to participation, all respondents were informed of the study's purpose, the anonymous nature of their responses, the scope of personal data collection, and the intended use of the data. A total of 206 responses were received. Of these, five were excluded due to either insincere response patterns or non-consent to personal data collection, yielding a final analytical sample of 201 participants.

While the majority of the sample (86.1%) belonged to traditional engineering colleges, it also included students from the Business School (n = 12) and the Design Engineering School (n = 16). Importantly, these specific schools offer STEM-integrated majors such as IT Management, Data Science Management, and Industrial Design Engineering. Therefore, rather than being strictly limited to traditional engineering disciplines, the sample is more accurately defined as undergraduates sharing a cohesive, engineering-oriented educational context. The demographic characteristics of the study sample are presented in Table 4.

Table 4.

Participant Characteristics(N=201)

3.3 Data Analysis

All analyses were conducted in R (version 4.3.3) using the psych and lavaan packages. The data were analyzed in four steps. First, descriptive statistics (means and standard deviations) were computed for all items, and skewness and kurtosis were examined to assess univariate normality.

Second, exploratory factor analysis (EFA) was conducted to examine the factor structure of the scale. Given the ordinal nature of the five-point Likert-type responses, a polychoric correlation matrix was used. Factors were extracted using Principal Axis Factoring (PAF), and an oblique rotation method (Oblimin) was applied. The factorability of the data was assessed using the Kaiser–Meyer–Olkin (KMO) measure and Bartlett’s test of sphericity.

Third, confirmatory factor analysis (CFA) was conducted to evaluate the validity of the four-factor structure derived from the EFA. The weighted least squares mean- and variance-adjusted (WLSMV) estimator was used because it is robust to multivariate non-normality in ordinal data. Model fit was evaluated using the Comparative Fit Index (CFI), Tucker–Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMR).

Fourth, the reliability and validity of the final scale were examined. Internal consistency reliability was assessed using Cronbach’s alpha. Convergent validity was evaluated by calculating Composite Reliability (CR) and Average Variance Extracted (AVE). Discriminant validity was assessed primarily using the Fornell–Larcker criterion [15]. When inter-factor correlations were high or the Fornell–Larcker criterion was borderline, the Heterotrait–Monotrait ratio (HTMT) was additionally calculated. An HTMT value below .85 was used as the criterion for discriminant validity [16].

4.1 Content Validity Results

A single round of ratings was completed by five experts for the 25 preliminary items, including 15 items from the task-related domain and 10 items from the strategy-related domain. Content validity was evaluated primarily using the Item-level Content Validity Index (I-CVI) and Aiken’s V, summarized at the construct level. The level of expert agreement was additionally examined using the median IQR and the proportion of items with IQR ≤ 1 (see Table 5).

Table 5.

Summary of Content Validity Indices by Construct

For the task-related domain, the mean I-CVI (S-CVI/Ave) was .875, the S-CVI/UA was .562, the mean Aiken’s V was .875, and the proportion of items with IQR ≤ 1 was .875. For the strategy-related domain, the mean I-CVI (S-CVI/Ave) was .920, the S-CVI/UA was .800, the mean Aiken’s V was .870, and the proportion of items with IQR ≤ 1 was .800.

The S-CVI/UA reflects the proportion of items on which all experts reached agreement. The relatively low S-CVI/UA of .562 for the task-related domain suggests that some items elicited differing judgments across experts. Accordingly, five items were revised based on expert feedback: TA22, TA32, TA46, AU23, and AU33. These items showed either lower than average Aiken’s V values at the construct level or relatively large IQR values. Revisions focused primarily on clarifying terminology, removing redundant expressions, and improving item comprehensibility for respondents.

Overall, both domains demonstrated acceptable content validity. The S-CVI/Ave ranged from .875 to .920, mean Aiken’s V ranged from .870 to .875, and the proportion of items with IQR values of 1 or lower ranged from .800 to .875. Accordingly, the preliminary item pool was considered to demonstrate acceptable content validity [17].

4.2 Descriptive Statistics and Item Normality

Descriptive statistics and normality checks were conducted for the 25 preliminary items retained after the content validity review (see Table 6). Item means ranged from 2.60 to 4.46, and standard deviations ranged from 0.57 to 1.53. To assess item-level normality, skewness and kurtosis were examined. The absolute values of skewness ranged from 0.25 to 1.58, and the absolute values of kurtosis ranged from 0.04 to 2.77. Based on the commonly used criteria of absolute skewness below 3.0 and absolute kurtosis below 10.0, no extreme non-normality was observed at the item level.

Table 6.

Descriptive Statistics and Item Analysis for the Initial Item Pool

Corrected item-total correlations (ITC) were then computed within each initial theoretical subfactor. ITC values ranged from .241 to .751. Following the criterion suggested by Nunnally and Bernstein [18], according to which an ITC of .30 or higher indicates an acceptable item, AU12 (“I can write effective prompts to get desired results.”) was removed because it did not meet this threshold.

4.3 Exploratory Factor Analysis (EFA) Results

After AU12 was removed based on the ITC criterion in Section 4.2, the remaining 24 items were examined for factorability. The KMO measure of sampling adequacy was .800, and Bartlett’s test of sphericity was significant, (276) = 1709.122, p < .001, supporting the suitability of the data for factor analysis.

EFA was conducted on the 24 candidate items. The number of factors to retain was determined on the basis of theoretical interpretability, the scree plot, parallel analysis, and eigenvalues. Items were removed iteratively based primarily on low primary loadings (< .40) and salient cross-loadings (loading differences < .20), in conjunction with theoretical interpretability. Communalities were used as a secondary diagnostic indicator to flag items for further review rather than as a strict deletion rule. Accordingly, a small number of items with lower communalities were retained to preserve content coverage and maintain an identifiable factor with an adequate number of indicators. Using these criteria, nine items were removed, resulting in a final four-factor solution with 15 items (see Table 7).

Table 7.

Deleted Items and Reasons for Deletion

The EFA yielded a four-factor solution with eigenvalues of 1.211, 2.717, 1.140, and 1.845 (see Table 8). The four factors accounted for 8.1%, 18.1%, 7.6%, and 12.3% of the total variance, respectively, with a cumulative explained variance of 46.1%. Item communalities ranged from .219 to .729, and all retained items had factor loadings of .40 or higher. Based on the content of the retained items and their loading patterns, the four factors were labeled as follows: Text-based Task Use (3 items), Media and Data Task Use (5 items), Prompting and Selection Strategy (4 items), and Critical Verification Strategy (3 items).

Table 8.

EFA Pattern Matrix, Communalities, and Variance Explained for Retained Items

4.4 Confirmatory Factor Analysis (CFA) Results

CFA was conducted to evaluate the validity of the four-factor measurement model derived from the EFA. Because the scale used an ordinal five-point Likert-type responses format, the WLSMV estimator was applied. Before finalizing the model, item pairs with theoretically justifiable residual covariance were identified on the basis of item content and wording. Residual covariances were allowed for two theoretically justified item pairs (TA42–TA43 and AU22–AU23).

Model fit was evaluated using X², CFI, TLI, RMSEA, and SRMR. The final model yielded the following fit indices: X²(82) = 158.782, p < .001, CFI = .923, TLI = .901, RMSEA = .068, and SRMR = .065. Both CFI and TLI exceeded .90, and both RMSEA and SRMR were below .08, indicating that the four-factor model showed acceptable fit to the data.

Standardized factor loadings were also examined to determine whether each item loaded significantly on its intended factor. All items had standardized loadings of .40 or higher, and all loadings were statistically significant (p < .001). Detailed CFA results are presented in Table 9.

Table 9.

Standardized Factor Loadings, Residual Variances, and R2 from the CFA

4.5 Reliability and Validity Results

To evaluate the reliability and validity of the measurement model, Cronbach’s alpha (α), Composite Reliability (CR), Average Variance Extracted (AVE), and inter-factor correlations were computed. Discriminant validity was examined using the Fornell–Larcker criterion by comparing the square root of each factor’s AVE (diagonal elements) with the inter-factor correlations (off-diagonal elements) (see Table 10).

Table 10.

Reliability, Convergent Validity, and Discriminant Validity of the Four Factors

In terms of reliability, media and data task use showed good internal consistency (α= .853), and critical verification strategy also demonstrated acceptable reliability (α= .828). In contrast, text-based task use (α= .654) and prompting and selection strategy (α= .623) showed relatively lower values. These lower values indicate that the two subscales should be interpreted with caution; their substantive and theoretical implications are examined in the Discussion.

With respect to convergent validity, media and data task use (AVE = .520) and critical verification strategy (AVE = .569) met the conventional .50 criterion. In contrast, text-based task use (α= .654, AVE = .397) and prompting and selection strategy (α= .623, AVE = .297) showed comparatively lower internal consistency and convergent validity indices, and these results should be interpreted with caution (see Table 10). However, all four factors showed CR values above .60, ranging from .626 to .844. Following the criterion suggested by Fornell and Larcker [15], convergent validity was therefore considered acceptable at the overall construct level.

Inter-factor correlations ranged from .294 to .582. Within the task domain, media and data task use and text-based task use showed the highest correlation (r = .582), which is theoretically interpretable given that both involve academic task performance using generative AI. Within the strategy domain, critical verification strategy and prompting and selection strategy were also moderately correlated (r = .560), suggesting that prompt refinement and strategic selection may co-occur with verification behavior in actual use.

Regarding discriminant validity, the square root of each factor’s AVE exceeded or approximated the corresponding inter-factor correlations for most factor pairs, indicating general support for discriminant validity. One borderline case was observed: the correlation between critical verification strategy and prompting and selection strategy (r = .560) slightly exceeded the square root of the AVE for prompting and selection strategy (.545). To examine this case further, the Heterotrait–Monotrait ratio (HTMT) was additionally calculated. Using the criterion of HTMT ≤ .85 [16], the HTMT value for this factor pair was .535, supporting adequate discriminant validity between critical verification strategy and prompting and selection strategy.

4.6 Final Scale

Following the expert content validity review and the EFA and CFA of the main survey data, the final version of the Task–Strategy Scale for Profiling Generative AI Use was established as a 15-item scale with a four-factor structure (see Table 11). The final factor structure consists of two task-related factors, Media and Data task use (5 items) and Text-based task use (3 items), and two strategy-related factors, Prompting and selection strategy (4 items) and Critical verification strategy (3 items).

Table 11.

Final Structure of the Task–Strategy Scale for Profiling Generative AI Use

The overall internal consistency of the 15-item scale was acceptable (Cronbach’s α = .836). However, the final scale was designed to be interpreted not through a single total score, but through combinations of scores across the four subscales. In this respect, the scale supports multidimensional profiling of learners’ generative AI use.