Project One: Threats to Validity
For each of the scenarios below indicate which threat to validity is indicated in the scenario. These threats are found in chapter 2 and are summarized in tables 2.2 — 2.5.
For each of the scenarios below fill in the blank with one of these threats to validity:
Major threats to internal validity
History
Maturation
Instrumentation
Testing
Statistical regression
Diffusion of treatment
Major of threats to external validity
Generality across subjects
Generality across responses are measures
Generalities across settings
Generality across times
Generalities across behavior change agents
Reactive experimental arrangements
Reactive assessment
Multiple treatment interference
Major threats to construct validity
Attention in contact according to the client
Special stimulus conditions, settings, and contexts
Major threats to data evaluation validity
Excessive variability in the data
Unreliability of the measures
Trends in the data
Insufficient data
Mixed data patterns
EXAMPLE:
The Springfield school system wanted to test a new physical education program for middle school students. When they entered middle school, they were administered a battery of physical education tests for speed, strength, and endurance. Over the next three years, the physical education teachers implemented the new physical education program for the students. When they graduated middle school, they were tested again and improvements were noted on speed, strength, and endurance. The school district concluded that the improvements and performance were due to the new physical education program. However, there was a major threat to internal validity in this program that is the performance improvement could have been due to maturation.
1. Eastfield State College decided to test a new method of education called personalized systematic instruction. Data were collected on student measures such as attendance course participation and grades on quizzes. Over the last year, the students' measured behavior started to decrease. The President of the college, therefore, questioned the effectiveness of the personalized system of instruction. The researcher argued that the decrease in performance might have been because the students had to suddenly shift to remote instruction due to the COVID-19 pandemic. Therefore, the major threat to internal validity in this program that led to the decrease in performance could have been due to the threat of ______________________.
2. The researchers were experimenting on the effects of choice of reinforcers on severe problem behavior. One problem behavior that was being measured was self-injury in the form of the client hitting their elbows on the table. When the principal investigator examined the data from two independent observers there were high levels of disagreement. Upon questioning the observers, the principal investigator learned that one observer would score self-injury only when the child hit the table with their elbow with excessive force from a distance of over 6 inches away. The other observer would score self-injury also if the child lightly tapped the table with their elbow. The major threat to data evaluation validity in this program would be called__________________________________________.
3. The Clinical Director of the school decided to conduct an experiment to see if a performance management system would improve the data collection done by the paraeducators in the class. The Clinical Director chose to work in Classroom 3 because the teacher was highly cooperative and easy to work with. The results of the study show that the performance management system was effective in improving the data collection done by the paraeducators. The major threat to construct validity in this study was _____________________________________.
4. Two doctoral students from the University of Sepulveda wanted to see if an intervention to improve social behavior was effective with students with ADHD. They conducted the intervention with three students with ADHD that were selected from the gifted and talented children program. The intervention was highly effective with these three students, and therefore the doctoral students concluded that the social behavior intervention was effective with children with ADHD. There is a problem in the study due to external validity. This problem was _______________________________.
5. Dr. Joseph Smith was doing a study on the effects of visual schedules on problem behavior with a student with severe behavior problems. When examining the data, Dr. Smith noticed that the student had especially high levels of problem behavior. He decided that this was an urgent situation and he should intervene by having the teacher use the visual schedule the next day. As soon as the visual schedule was put into place the data showed that the problem behavior decreased dramatically. One threat to internal validity that could explain this decreasing problem behavior is ___________________________________________________.
6. Susie Researcher was conducting a parent training study. She wanted to see the effects of praise on compliance. She trained one parent to deliver praise contingent upon the children coming to the dinner table when called. She decided not to train the other parent to see if the use of praise would result in different levels of compliance across both parents. When examining the data, she found that compliance increased with both parents. When she went to the house, she observed that both parents were delivering praise. This threat to internal validity is called _______________________________________.
7. Dr. Handan conducted an intervention where a child was taught to ask for a break rather than engaging in problem behavior to escape from the task. The intervention was highly effective such that the child engaged in low rates of problem behavior and high rates of requesting. When Dr. Handan instructed the parent to do the same intervention the child's problem behavior persisted at high levels. The problem in this scenario might be explained by the threat to external validity called__________________________________________.
8. At Bright Start ABA school the staff were trained to implement naturalistic language training techniques to increase spontaneous communication exhibited by the students. For training, the researcher observed the staff member and provided them with feedback on their use of the naturalistic language training techniques while they worked with the children. When the researcher left the room, the staff members no longer used the naturalistic training techniques. This threat to external validity is called __________________________________________.
9. Debbie was an Occupational Therapist and also a BCBA. She had heard of a new treatment called TAG treatment. This was also known as teaching with acoustic guidance. The treatment involved providing the student with an auditory signal each time they made a correct response. She wondered if this would help one of her students to improve their dressing performance on a Task Analysis. She had a classroom meeting and demonstrated the procedure to all the staff. After two weeks of implementing the program, she noticed that's some days the data showed high levels of correct responses approaching 80% – 90% correct, and other days the data showed very low levels of responses approaching 10 – 20% correct. She decided to watch the staff to see what was going on. She found out some of the staff we are implementing the treatment correctly but others were making errors in their implementation. This threat to data evaluation validity is described as ____________________________________________________.
10. The staff in Classroom 4 were wondering if visual schedules would decrease levels of problem behavior for one of their students. They all decided that they would start the treatment on January 15. The week before January 15th, the child's problem behavior started to decrease. When they implemented the visual schedule, it decreased even more. It was hard to conclude that the visual schedule what is effective in improving behavior because of the threat to data-evaluation validity due to __________________________________________
