A Study on the Development of Proficiency Test Contents for Evaluating Digital Forensics Technical Competency

2.3.1 Difficulty

Item difficulty is an index indicating the degree of easiness or difficulty, representing the proportion of respondents who answered correctly. Generally, it can be interpreted and classified as follows: a correct answer rate of less than 20% indicates a very difficult item, 20% to less than 40% is a difficult item, 40% to less than 60% is an item of medium difficulty, 60% to less than 80% is an easy item, and 80% or more indicates a very easy item [31]. Alternatively, an item difficulty of 0.25 or below is classified as a difficult item, 0.25 to less than 0.75 as an appropriate item, and 0.75 or above as an easy item [32].

N : Total Number of Participants
R : Number of Correct Respondents by item

2.3.2 Discrimination

Discrimination is an index indicating the extent to which each item can discriminate the participants' ability levels. It is distributed from -1 to +1, and the closer it is to 1, the higher the discriminatory power. If the item discrimination is 0.3 or higher, it can be evaluated as a good item [33]. Methods for calculating discrimination include the "upper and lower group differentiation method, " which compares the difficulty difference between the high-scoring and low-scoring groups, and the "item-total correlation method, " which calculates the correlation coefficient between an item and the total score [34]. The upper and lower group differentiation method divides respondents into upper and lower groups based on their scores and measures discrimination by analyzing the difference in item responses within each group. The item-total correlation coefficient measures discrimination by analyzing the correlation between the score of each item and the total score.

x : Item Score, y : Total Score, n : Number of Respondents

2.3.3 Validity

Validity refers to the extent to which an evaluation tool or test reflects its intended evaluation goals [35]. Types of validity are classified based on evidence based on test content, internal structure, relations to other variables, response processes, and testing consequences. Evidence based on test content (content validity) is defined as the degree to which items, questions, and purposes represent the content intended to be measured. It is verified by the subjective judgment of content experts with professional expertise. Evidence based on internal structure (construct validity) analyzes abstract human psychological characteristics or traits as psychological constructs and, after operational definition, verifies whether they were measured properly [31].

2.3.4 Reliability

For reliability, Cronbach's alpha (α), which is an internal consistency reliability, is utilized. A Cronbach's α coefficient closer to 1 implies higher reliability [31].

N : Number of items
$$ {\sigma }_i^2$$ : item Variances
$$ {\sigma }_T^2$$ : Variance of Total Scores

3.2.1. Research Procedure

The production of the proficiency test contents was largely divided into five stages. First, a proficiency test item classification framework was established. Subsequently, a digital forensic scenario was created, and corresponding actions were performed within a virtual environment to generate digital evidence. At this stage, proficiency test item was drafted according to the item classification framework established previously. Furthermore, proficiency test item was produced through the imaging of the virtual environment where the digital evidence was stored. The overall five-stage development process—from establishing the classification framework, through scenario creation and evidence generation, to item drafting and imaging—is summarized in Figure 1, which serves as the roadmap for the procedures detailed in the following subsections (3.2.2–3.2.7).

Figure 1 presents the five stages in their logical dependency order rather than as a strict chronological sequence. In practice the stages overlapped: refinement of the classification framework (Section 3.2.2), scenario-based evidence generation and imaging (Section 3.2.4), item drafting (Section 3.2.5), and portal development (Section 3.2.6) proceeded concurrently in late September 2025, which is why the reported date ranges overlap. The proficiency test in Section 3.2.7 was administered on the finalized image, item set, and portal.

3.2.2. Establishment of Proficiency Test Item Classification Framework

The classification framework consists of three hierarchical levels. The top level was organized into eight major categories that group artifacts by the activities that can occur in a Windows environment: (1) System, (2) User behavior and processes, (3) File-based, (4) Network-based, (5) Security policy-based, (6) Web browser, (7) Data concealment and manipulation, and (8) Purpose-specific behavior. These eight major categories are further refined into 52 sub-categories and, at the most granular level, into approximately 170 specific artifacts. For example, the 'File-based' major category comprises sub-categories such as File Metadata, File System, Volume & boot information, Deleted Data, Cache & preview data, and External Storage device connection. Within File Metadata, the specific artifacts include File/Folder Metadata, File Creation/Modification/Deletion Logs, and NTFS Object ID. This serves as the basis for deriving the individual assessment items. Table 4 summarizes the major and sub-categories.

The framework was developed and validated in two stages. First, a structure-appropriateness survey was administered by email over one week, from September 26 to October 2, 2025, to 15 digital-forensics experts recruited through purposive sampling. The selection criteria were (i) affiliation with an investigative agency (police, prosecution, or special judicial police) or a digital-forensics/investigation institution, (ii) current practice in the relevant field, and (iii) prior experience analyzing OS-based artifacts. The 15 participants comprised nine from the police (including one from the Korea Coast Guard), one from the prosecution, one from a military investigative body, and four from special judicial police and administrative-investigation agencies (the Financial Supervisory Service, the Korea Fair Trade Commission, the National Election Commission, and the Gyeonggi Provincial special judicial police), spanning the breadth of investigative and inspection practice. Their experience ranged from 3 to 25 years (mean ≈ 9.5, median 9), with 13 having five or more years. The survey primarily addressed the appropriateness of the major-, sub-, and specific-artifact divisions of Windows OS artifacts, along with queries about missing artifacts.

Appropriateness was determined by aggregating expert opinion rather than by a numerical rating. An item was revised when two or more experts raised the same change (relocation, consolidation, or removal) and was otherwise retained as drafted, and a missing artifact was added only when proposed in common by two or more experts. The items thus derived were then subjected to a second-stage difficulty-and-frequency rating survey using a fivepoint Likert scale, administered to 12 practitioners— a subset of the first-stage panel comprising seven from the police, one from a military investigative body, and four from special judicial police and administrative-investigation agencies, whose experience ranged from 4 to 25 years (mean ≈ 10.3, median 9). Difficulty ranged from 1 (immediately verifiable in the UI) to 5 (requiring advanced skills such as kernel/cryptographic analysis or customparser development), and frequency from 1 (never performed) to 5 (performed in virtually every case, ≥ 80%). For each item, the mean, mode, and sample standard deviation were computed and used to prioritize item development: items with a high frequency mean and a low standard deviation—indicating convergent expert judgment—were prioritized for inclusion, while the difficulty mean was used to keep the difficulty distribution across items balanced.

3.2.3. Scenario Development

In this study, a scenario was developed based on the theme of industrial secret leakage. The scenarios are as follows:

“Company X faced management difficulties after a competitor released a similar product to a new product scheduled for release in the second half of the year. Internally, suspicions grew about a leak of industrial secrets, and an internal audit was conducted. The audit revealed that employee Kim Ma-ba had been using a personal USB storage device and copying certain documents to external media. Kim Ma-ba, who was part of the business operations team, had access to confidential industrial documents, and after suspicions were raised that these documents were shared with competitors in external markets, Kim Ma-ba was identified as a key suspect in the leak. The investigative agency seized Kim Ma-ba's work PC in accordance with digital forensics procedures and copied the entire disk image to begin analysis. The main issue is whether Kim Ma-ba actually leaked confidential documents via USB and whether he tried to cover up the leak.”

3.2.4. Creation of Proficiency Test Item

The proficiency-test items were developed for personal computers (PCs). First, based on the classification framework defined earlier, digital evidence was generated experimentally in a virtual machine (VM) environment according to the scenarios. To configure an environment similar to an actual work computer, productivity tools such as word processors were installed, and snapshots were taken at each action to verify whether the intended artifacts were properly generated. When artifacts were generated contrary to the intent, the system was rolled back to the previous snapshot and the action was re-performed; this process was repeated to preverify whether each scenario action in the Windows environment produced artifacts at the intended time and form. Based on the scenarios finalized through this pre-verification, an actual PC environment was set up to generate digital evidence and perform imaging. Windows 11 Pro was installed on a physical PC equipped with a 512 GB SSD and 32 GB RAM, and a local account "Kim Ma-ba" was created. The scenario actions were then performed sequentially. After evidence generation was complete, the system was powered off and the Logicube Falcon imaging device was connected via the SATA interface to perform a bit-by-bit copy. The integrity of the acquired image was then verified by computing its SHA-1 hash value. Two hash algorithms are used in this study for different stages. At the acquisition stage, image integrity was verified with SHA-1, and this SHA-1 value was also provided to participants on the portal so they could confirm the integrity of the downloaded image. At the analysis stage, assessment Item 1 instead asks participants to compute the SHA-256 value of the disk image as a scored task. The two hashes therefore serve distinct roles—download-integrity verification versus an analysis competency check—and are not interchangeable. The Digital Forensic Toolkit (DFT) was used to analyze the generated artifacts. The technical specifications of this examination environment are summarized in Table 5.

Whether the final image conformed to the scenario design intent was confirmed through two stages of post-verification. First, the researchers who participated in the scenario design repeatedly checked whether the evidence generated at each action step matched the artifacts required by the assessment items, and when inconsistencies were found, the image or the items were revised iteratively. Second, an internal researcher who had not participated in the scenario design directly attempted the test on the completed image, and the final image was confirmed by incorporating the feedback submitted as a result. This verification serves as an internal content-validity check ensuring that every assessment item can be answered solely from the distributed image.

3.2.5. Development of PT Assessment Items

Figure 2 shows the resulting assessment-item structure, which links each of the 20 items to its corresponding category in the classification framework in Table 4 and to the scenario actions. A total of 20 assessment items were constructed on the basis of the proficiency test item developed for this study. These items were intended to assess the identification and analysis of different forms of digital evidence, such as OS information, application execution records, file metadata, file systems, and traces left by the use of external storage devices. Instead of listing the full range of the classification framework established earlier, we focused on those categories that matched the scenario and developed them into specific assessment items. Table 6 summarizes the principal user actions in the scenario and the artifacts analyzed for each.

In developing the items, we relied on two pathways already established in the manuscript to support their validity. First, every item was anchored to a specific sub-category of the artifact classification framework presented in Section 3.2.2, which had itself been validated through expert review (a structure-appropriateness survey with 15 practitioners and a difficulty-and-frequency rating survey with 12 practitioners). Second, the items were verified through the iterative scenario-and-image cycle in Section 3.2.4, which confirmed that each item could be answered from the artifacts produced by the scenario in the released image. In this study, the term proficiency test item refers to the digital forensic test object provided to participants (e.g., an image file), whereas assessment items refer to the 20 scored prompts used to evaluate participants' responses. These were supply-type short-answer items in which participants extracted a specific value from the image and entered it as free text (e.g., a file path, device name, serial number, timestamp, or execution count), rather than selection-type items such as multiple choice. Item 1 (disk-image hash) is an integrity-verification step common to all digital-forensic examinations and is therefore shown separately, not mapped to a specific artifact category. A structured content-validation procedure, such as expert Delphi review or CVR analysis, was not conducted in this study and remains a task for future research.

3.2.6. Development of Proficiency Test Guide/ Distribution Website

After producing the items, an information and distribution portal was developed to conduct the proficiency test. A user login function was implemented to allow only authorized personnel to access the proficiency test page, and tabs such as scenario, test items, and feedback registration were implemented within the portal. In the 'Scenario Tab', participants can review the scenario for problem-solving and download the proficiency test item (image). The 'Test Items Tab' provides the assessment items and allows the submission of image analysis results. Furthermore, the 'Register your opinion Tab' was implemented so participants could raise objections during the problem-solving process or suggest improvements (Figure 3).

3.2.7. Execution of the Proficiency Test

The proficiency test was conducted targeting 17 practitioners in digital forensics-related fields over approximately three weeks, from September 27 to October 17. The 17 participants were recruited from practitioners with digital forensics-related experience. Of these, nine were selected from the private sector (such as companies specializing in digital forensics, internal audit consulting, and information security) and eight from the public sector (such as the military, the police, and intelligence agencies). They were individually contacted by email or phone and took part after confirming their willingness to participate. This recruitment was conducted separately from the expert-validation procedure described in Section 3.2.2, so the proficiency test was administered to a participant pool independent of the expert panel. When each participant performs the proficiency test and submits their results to the portal, the results are stored in the database. This study primarily focused on evaluating the appropriateness of the designed assessment items and identifying issues that emerged from the actual proficiency test results in the domestic context.

4.2.1. Assessment Item Difficulty

Under CTT the average proportion-correct across items came to 0.600, which puts the test just on the easy side of moderate difficulty. When those proportion-correct values were rescaled to logits, the mean difficulty estimate (b*) was -0.482—again suggesting that participants, on the whole, did not find the items especially demanding. This means that participants did not perceive the assessment items as particularly difficult. It represents an appropriate difficulty distribution for measuring general competency or basic conceptual understanding, which is the purpose of the PT. At the same time, some revision will be necessary if the test is to distinguish more effectively between higher- and lower-performing groups. In particular, the very difficult items (Items 6 and 13), whose near-floor success rates limit their measurement value, are the primary candidates for revision or replacement. The very easy items (≥80% correct: Items 4, 12, 14, 16, 19, and 20) are less of a concern: although they leave little headroom at the top of the ability range, most still discriminate adequately (rpb = 0.48 to 0.83) and were therefore retained, flagged only for possible difficulty rebalancing in future iterations.

4.2.2. Assessment Item Discrimination

Under classical test theory, item–total correlations ranged from -0.28 to 0.83, with an average of about 0.45. Taken together, this suggests that the items were doing a reasonably good job of distinguishing higher-performing participants from lower-performing ones. Decisions about whether to keep, revise, or remove items were therefore based mainly on rpb and UL27. Items 2 and 6 yielded very low discrimination and were identified as items with insufficient discriminatory power. In particular, Item 6 had only one correct answerer, and its item-total correlation was close to 0, indicating that the item's function practically did not operate properly. In addition, Item 13 showed a clearly negative item–total correlation (rpb = −0.280), while Item 9 showed a near-zero correlation (rpb = −0.003) together with a negative upper–lower index (UL27 = −0.40); in neither case did the higher-scoring group outperform the lower-scoring group. For Item 9, this signal is attributable to its structural dependence on Item 7 (see Section 4.3) rather than to a defect in the item itself. Because the item conditions or wording may be ambiguous, the wording should be revised and the correct answers re-examined; a detailed cause analysis is provided in Section 4.3.

4.2.3. Exploratory Evidence Based on Response Processes (Comprehensibility, Readability, Ambiguity)

This section examines exploratory evidence based on response processes by analyzing participants’ ratings of comprehensibility, readability, and ambiguity for each assessment item. To this end, participants rated each assessment item in terms of comprehensibility, readability, and ambiguity. Most items received scores of 3 or above for comprehensibility and readability, indicating that participants generally understood the assessment items without much difficulty. On the whole, the wording appears to have been reasonably clear and the language easy to follow. At the same time, several items received lower ratings on one or more of these dimensions, suggesting that their wording should be improved before the test is used again. Nevertheless, Items 10, 11, and 13 showed relatively low comprehensibility and readability, along with comparatively high ambiguity, suggesting that participants had difficulty grasping the meaning of these items. In addition, Items 7 and 8 demonstrated a medium level of comprehensibility and readability, but their ambiguity was slightly high. Therefore, it is necessary to improve these items by simplifying their sentence expressions and clearly adjusting the semantic differences in the item stems. Next, the relationships between item comprehensibility, readability, ambiguity, and the correct answer rate were examined. Overall, a positive correlation was observed where higher comprehensibility and readability were associated with a higher correct answer rate. The average correct answer rate of all items was approximately 60.0% (consistent with the mean proportion-correct of 0.600 reported in Section 4.2.1), the average comprehensibility was 3.150 points, the average readability was 3.212 points, and the average ambiguity was 1.241 points. Item-level correlations among proportion correct, comprehensibility, readability, and ambiguity were examined. The correlation analysis results illustrated that item comprehensibility (r = .608**, p < .01) and readability (r = .546*, p < .05) exhibited a significant positive correlation with the correct answer rate. Conversely, ambiguity (r = -.364) had a negative correlation with the correct answer rate and was not statistically significant (p = .112), as shown in Table 8.

This suggests a descriptive pattern in which more ambiguous items tended to be accompanied by lower comprehensibility and readability, rather than directly affecting the correct answer rate. In fact, ambiguity demonstrated a very strong negative correlation with comprehensibility (r = -.922, p < .001) and readability (r = -.904, p < .001), indicating a tendency where more ambiguous items simultaneously led to lower comprehensibility and readability. Accordingly, the items should be revised to improve clarity, reduce potential misunderstanding, and enhance comprehensibility and readability.