Using Vosaic as a Framework for Research

A study conducted by Richard M. Kubina Jr., Madeline Halkowski, Kirsten K. L. Yurich, Kimberly Ghorm, and Nora M. Healy compared the detection accuracy of operational definitions and pinpoints. The researchers aimed to discover professionals' ability to accurately detect instances of a target behavior using two different methods. They used Vosaic, a video analysis platform, as a framework for the study.

Comparing the Detection Accuracy of Operational Definitions and Pinpoints

This study compared the detection accuracy of an operational definition for self-injurious behavior, with a corresponding pinpoint, across professionals who work in schools that serve students with an autism spectrum disorder.

Operational definitions exist to ensure clarity in the data collected, especially when visual checks are made by multiple people, leaving room for confusion. They have a significant history in applied behavior analysis and play a role in detecting an event, human thought, or action. Recently in psychology literature, there have been concerns with the translation validity and detection accuracy of target behaviors and operational definitions.

Pinpoints represent an alternative for describing target behaviors. A pinpoint uses a formula to construct a sentence using an action verb, an object or an event that receives the action, and a defined context where the observation of the action occurs.

The study’s results found lower accuracy scores for the operational definition compared to the pinpoint. The consistency of scores varied more for the operational definition than the pinpoint.

Operational Definitions

There is no operational definition framework that is universal and actionable. Several earlier studies of operational definitions varied their structure along the following dimensions:

• Categories of specific behaviors

• Abbreviated list of expected behaviors

• Examples that demonstrate the presence and absence of the behavior

• Positive and negative examples

• Addition of a time element

This study, in contrast, examined the participants’ accuracy in detecting a student’s target behavior when identifying the behavior using two different forms of a target behavioral definition (TBD1 & TBD2). The first definition, TBD1, followed an operational definition used by the student’s school. TBD1 included more descriptive words, such as “swings forcefully.” The second definition, TBD2, featured a shorter phrase with only an object term and present tense active verbs describing movements associated with the target behavior.

The average accuracy for the participants came to 35% for TBD1 and 68% for TBD2. The researchers concluded that this difference in participant accuracy stemmed from adding additional descriptive words in TBD1, which often appear in operational definitions to form an overall picture of the behavior.

If you don't have a Vosaic account for video analysis, automated transcription and research, you can start with a free trial today.

Precision Teaching and Pinpointing

Precision teaching (PT) helps parents and teachers measure behavior, analyze data, and make decisions about a child or student. PT has four steps: pinpoint, record, change, and try again. As PT has advanced, its movement cycle changed to “pinpoint” to reflect the addition of context. Pinpoint enhances the detection of the target behavior, directly represents the target selected for observation, and improves communication among relevant stakeholders.

Two interesting contrasts emerge when comparing pinpoints with operational definitions;

1. An adequately formed pinpoint won't have translation validity issues due to its clear construction

2. Pinpoints provide better detection accuracy than operational definitions

The Method of the Study

The study was held in a school for students with an autism spectrum disorder. Staff members were tasked with detecting a targeted behavior and accurately counting its occurrence.

Two video presentations served as the stimuli for the study. Each video presentation had 30 10-second video clips separated by a 5-second black clip. The black screen displayed a 3-second countdown to the next video segment. The videos showed a young man engaged in self-injurious behavior across multiple settings.

The participants viewed the stimuli at cafeteria-style tables, about 2-3 feet from the other participants. The videos had no sound, and cardboard dividers created a border around each seat to help minimize distractions.

At the start of each viewing session, participants received a black pen, a data collection sheet, and a behavior definition card. The scoring sheet had a 2-by-30 grid, with the first column listing each video segment and column two providing blank spaces for the participants to mark their responses.

The independent variable consisted of two target behaviors. The first followed an operational definition format. The second definition matched the pinpoint formula, which consisted of a movement cycle.

The dependent variable represented the count of target behaviors recorded by the participants under each behavior condition (operational definition, pinpoint). The participants watched the videos during the experimental sessions, marking each perceived instance of the target behavior on a data collection sheet after the recorded segment. The responses were analyzed using total count and segment-by-segment calculations.

Gold Standard

A “gold standard” refers to the best available measure of the presence or absence of a condition. Because participants can’t stop, pause, or rewind when watching target behaviors human errors or inaccuracies can occur. Therefore, researchers used Vosaic to analyze and annotate the recorded target behaviors using frame-by-frame viewing. Vosaic allowed multiple people to annotate a single video while keeping their marks private from their peers.

Five researchers independently viewed and marked each instance of self-injurious behavior, as defined by the behavior descriptors, in the videos using Vosaic. After each researcher completed annotating the videos, they came together to compare results and any areas of disagreement. Once they reached 100% agreement on all the video segments, the gold standard was created.

Researchers used the gold standard to analyze the participants’ data to determine if they correctly identified the target behavior in each video segment.

Data Analysis

The researchers calculated the percent agreement by comparing participants’ total count of target behaviors under each condition to the count of target behaviors identified in the gold standard. Only the segments that contained the target behavior per the gold standard or that a participant indicated an instance of the behavior occurring were factored into the percent agreement calculation.

An agreement happened when the participant within the video clip identified the target behavior. A disagreement occurred when the gold standard found an instance of the target behavior, but the participant did not report any behavior.

Results

Overall, there was a 75% average accuracy when the operational definition method was used. When the pinpoint method was used, there was an average accuracy of 93%. You can read the full study here.

About Vosaic

Vosaic is a video analysis platform for researchers. The likelihood of missing a crucial moment during in-person research sessions is high, and Vosaic eradicates lost data. Researchers can record or upload videos from anywhere, then mark essential moments on a video timeline. Vosaic helps ensure that every moment is captured, making a more effective study.