Online version of Exact: Equivalence Study
The current version of Exact runs online, rather than from a CD. There have been some updates to the illustrations, audio and animations. A study was carried out between September and December 2018 to evaluate the equivalence of the old and new versions.
A total of 146 students from six schools (n=5)/colleges (n=1) in the UK completed both the old (CD) and new (online) versions of Exact, with an interval of four to six weeks between test sessions. Each school/college collected data on just one form of the test (Form A or Form B – these were randomly allocated to schools), and test order (either the old or new version being delivered first) was randomly allocated to account for order effects. Students were selected based on their birth day of month (with each school randomly allocated a day of the month) to avoid any selection bias within the student sample.
Pupil population data for the school sample (based on the 2016/17 figures, which were the latest available at the time of the study) were compared to the national averages (for state- funded secondary schools, not including special schools, where available; for FSM and absence, the national figures include special schools as these are the only figures available). The school sample was found to be not significantly different from the national average on any of the pupil population measures: number on roll (t=0.36, df=4, p>0.05, NS); percentage of female students (t=0.29, df=4, p>0.05, NS); percentage of students eligible for free school meals at any time during the past six years (t=1.37, df=4, p>0.05, NS); percentage of pupils whose first language is not English (t=1.99, df=4, p>0.05, NS); percentage of pupils with a statement/EHCP (t=0.05, df=4, p>0.05, NS); and overall absence (t=0.41, df=4, p>0.05, NS).
Of the 146 students in the sample, 8.2% were eligible (at the start of the study) for free school meals. Twenty students in the sample (13.7%) were recorded as having a special education need/ disability (SEND). In half of these cases, the specific type of SEND was not reported (6.8%) but, where it was stated, these included Specific Learning Difficulties (SpLD: 2.7%), Speech, Language and Communication Needs (SLCN: 2.7%) and Moderate Learning Difficulties (MLD: 1.4%). The majority of the sample (91.1%) were White, 6.8% were Asian and 2.1% were Black. With regard to language, 8.9% of the sample’s first language was not English.
The resulting correlations between the old and new versions of Exact are given in Table 4 (note that ‘n’ varies between subtests as not all students completed both versions of each subtest). For all subtests, the correlations meet the required standard of 0.70, to adequately demonstrate equivalence (according to the European Federation of Psychologists’ Associations’ test review model). In all cases, the correlations are very highly significant (p<0.001).
Table 4. Correlations between old (CD) and new (online) versions of Exact
a Data meet the assumptions required for parametric testing; correlational analysis utilises Pearson’s ‘r’.
b Data don’t meet the assumptions required for parametric testing; correlational analysis utilises Spearman’s ‘rho’.
c Exact reading comprehension results include an indication of whether the student may have rushed the test. In cases where retest was definitely required or recommended, and this was supported by clear differences between the student’s speed and accuracy scores, their reading comprehension data were not included in the correlational analysis.