Validation of the tests in LADS Plus

The three dyslexia-sensitive tests in LADS Plus were validated in three separate studies.

Study A. This study (Singleton and Horne, 2001) involved 8 centres catering for adults with dyslexia; 2 were in universities, 3 were in colleges of further education, and 3 were in basic skills centres. A total of 140 adults participated in initial trials of the system; 71 of these were known to be dyslexic on the basis of conventional psychological assessments, and the remaining 69 were not dyslexic (as far as could be determined).

Study B. This study involved 48 randomly selected university students, none of whom was known to have dyslexia.

Study C. This study involved 38 university students (19 dyslexic and 19 non-dyslexic) who had been closely matched for intelligence. A research paper describing these studies is in press (Singleton and Horne, in press).

Validation Study A

The participants were administered the three dyslexia-sensitive tests in LADS Plus, with the full (not adaptive) forms of Word Recognition (120 items) and Word Construction (50 items) being used. Scores are number of items correct in all cases. Descriptive statistics of the sample are shown in Table 2.

Table 2. LADS Plus results from three different types of institution.7

As a check on the validity of the separation into ‘Dyslexic’ and ‘Non Dyslexic’ groups, all participants in Study A were administered the Adult Dyslexia Checklist (Vinegrad, 1994), which is a list of 20 yes/no questions relating to difficulties commonly experienced by adults with dyslexia, e.g. ‘Do you have difficulties when writing cheques?’ The dyslexic group obtained a mean (average) of 12.03 positive dyslexia indicators on the checklist (SD 3.87) while the non-dyslexic group obtained a mean of 4.47 positive dyslexia indicators (SD 3.32). Scores of 8 or higher are usually regarded indicating a strong likelihood of dyslexia. Statistical analysis (ANOVA) revealed that all three tests in LADS Plus distinguished significantly between the dyslexic and non-dyslexic groups (see Table 3).8

Table 3. Comparison of dyslexic and non-dyslexic adults on the LADS Plus tests used in Validation Study A.

Internal consistency statistics were also computed and this enabled unreliable items to be eliminated. The overall internal reliability (alpha) scores for the final version of LADS Plus were: Word Recognition: α = 0.95; Word Construction: α = 0.96, which are very high.

The results of comparing dyslexics with non-dyslexics within the three types of institutions were broadly similar, although results for Working Memory were of a lower significance level, especially in the university group. For this reason, further research was carried out to see if a more sensitive measure for the backwards digit span test could be found. Statistical analysis showed that the most sensitive measure was a combined score created by adding the total number of items correct to the total number of digits in their correct positions. Using this combined score showed a highly significant difference was found between the dyslexic and non-dyslexic groups (see Table 4) and the statistical significance for the three types of institution were greatly improved. It was therefore decided to use this combined score in the developmental version of LADS Plus.

Table 4. Comparison of dyslexic and non-dyslexic adults on the combined score obtained from the LADS Plus Working Memory test.

7 SD stands for ‘standard deviation’, a statistical term that represents the amount of variability of the scores obtained by the members of the group; the higher the SD, the greater the variance amongst the scores in the group. It is not necessary to understand this concept in order to follow the statistical results in this section.

8 The level of statistical significant is shown as a probability value (p); e.g. p < 0.01 means that the result obtained would be expected to occur by chance less than once in every hundred times that these data were collected. In other words, it is highly unlikely that this result is simply a chance event and therefore highly likely that the outcome represents a real difference. Similarly, p < 0.001 means that the result obtained would be expected to occur by chance less than once in every thousand times. Hence the smaller the p value the greater degree of confidence one can have in the finding.

Creation and validation of the adaptive version of LADS Plus

The scores obtained by the participants in Study A were used to calculate difficulty levels for every item in the test and this enabled items to be selected for the adaptive forms of the Word Recognition and Word Construction tests. The data for these tests were then run through the fractionation algorithm to calibrate the adaptive forms of the tests so that their results approximated as closely as possible to those obtained from the full forms. For the Word Recognition test, a correlation9 of –0.95 was obtained between the full form and the adaptive form, while for the Word Construction test, the correlation was –0.96. Both these correlations are exceptionally high and are statistically significant (p < 0.001). These results indicate that a high degree of confidence can be placed in the fractionation algorithm as the mathematical basis for the adaptive forms of these tests. A similar calibration exercise was carried out on data from the Working Memory test to create outputs that were on the same scale as that of the Word Recognition and Word Construction tests (i.e. ranging from 1 to 9). To check this, data for Working Memory test from Study B were analysed and the correlation between the recalibrated scores and the original raw scores was found to be –0.85, which is also statistically significant (p < 0.001).

The data from the adaptive forms of the three tests were then subjected to statistical analysis similar to that carried out on the original data. The results are shown in Table 5.

Table 5. Comparison of dyslexic and non-dyslexic adults on the adaptive forms of the LADS Plus tests (score range 1 – 9 for each test).

As explained in Section 2.1, the value of any screening test depends on having low frequencies of false positives and false negatives. In general, levels of less than 25% for each of these are advocated for effective screening (see Jansky, 1977, Singleton, 1997a). The following tables show the discriminant function analysis carried out on each of the three LADS Plus tests in their adaptive versions, in order to determine percentages of false positives and false negatives. See Table 6, Table 7 and Table 8. The results indicate that the LADS Plus tests come well within the required limits for false negatives (see Table 10), and that with the exception of Working Memory, also come well within the required limits for false positives. The Working Memory test comes somewhat over the expected limits with a false positive rate of 30.6%. However, on investigation it was found that a substantial proportion of these cases comprised individuals who scored poorly on the test because the instructions were unclear. In the Developmental Version of LADS Plus, the instructions for the test have been improved in order to correct this.

Table 6. Discriminant function analysis of the LADS Plus Word Recognition Test.

Table 7. Discriminant function analysis of the LADS Plus Word Construction Test.

Table 8. Discriminant function analysis of the LADS Plus Working Memory Test.10

When the individual scores for each of the three tests is amalgamated to create a composite LADS Plus score (minimum 3, maximum 27), the incidence of false positives was 3.7%, and of false negatives was 4.5% (see Table 9 and Table 10). Overall, the percentage of cases correctly classified by the LADS Plus composite score was almost 92%, which is extremely high for a screening instrument of this type. This demonstrates the power of a screening system in which the classification is based on a composite score derived from number of strong components, each of which has clear validity and strong predictive accuracy. In addition to using the quantitatively derived classification, Administrators can use their own judgement when examining an individual’s LADS Plus profile on a qualitative basis and in making recommendations for action (see Chapter 5 for further discussion of this).

Table 9. Discriminant function analysis of the LADS Plus Composite Score.

Table 10. Percentages of false positives and false negatives obtained in the LADS Plus validation trials.

9 Correlation is a statistical measure of relatedness between scores obtained on two different measures by the same individuals. The correlation coefficient (r) varies between 1.0 (absolute correlation) and 0 (zero correlation. A positive r indicates that the scores on the two measures are both in the same direction, while a negative r indicates that scores are in opposite directions. In the case of LADS Plus, a negative r is to be expected as the output of the fractionation algorithm is in the opposite direction, i.e. low scores indicating not dyslexic, and high scores indicating dyslexia.

10 Note that in the Working Memory test, data was only obtained from a total of 134 participants.

Validation Study B

In Study B, which employed the adaptive forms of the LADS Plus tests, three additional conventional tests were also administered: Woodcock-Johnson Word Attack Test (a test of phonological decoding skills using non-words), WRAT 3 Spelling Test, and WAIS-III Digit Span Test (digits forwards and backwards). Table 11 shows the intercorrelations between these measures and the scores obtained on the LADS Plus tests.

Table 11. Intercorrelations between scores on the adaptive forms of the LADS Plus tests and three conventional tests (n = 48).

Key: * p < 0.05; ** p < 0.01; *** p < 0.001.

From these results it can be seen that although each of the three LADS Plus tests contributes significantly to the composite LADS Plus score, each are independent measures, since the intercorrelations between the three LADS Plus tests are all rather low and not statistically significant. The LADS Plus Working Memory test (digits backwards) correlates significantly with WAIS-III Digit Span Test (digits forwards and backwards): r = –0.58, p < 0.001, thus providing further validation for the former. The results also suggest that the LADS Plus Word Construction test draws upon skills involved in phonological coding, spelling of real words and short-term memory, since the correlations between LADS Plus Word Construction and the three conventional tests were all statistically significant with r in the region of –0.3 to – 0.4. This is consistent with the argument put forward in Section 2.1 and the skills needed for the LADS Plus tests shown in Table 1. By contrast, the LADS Plus Word Recognition test depends much more on lexical access skills, which were not tested separately in this study. However, in Study A, the participants were also tested using WRAT 3 Reading test, which assesses single word recognition and which therefore depends more on lexical access skills. The correlation between WRAT 3 Reading and LADS Plus Word Recognition (full form) was found to be 0.89, which is very high and statistically significant (p < 0.001), thus providing additional validation for the LADS Plus Word Recognition test.

Inspection of the centile scores for the three conventional tests (see Table 12) indicates that the participants in Study A comprise a group that is above average in phonic skills (Woodcock-Johnson Word Attack) and spelling (as might be expected in university students), but not above average in working memory (and WAIS-III Digit Span). Moreover, it should be noted that the standard deviations (SDs) for both WAIS-III Digit Span and LADS Plus Working Memory are both relatively high. This indicates that there is much greater variance in scores for short-term (working) memory. One important implication of this is that amongst non-dyslexic adults there will be quite a few who have rather weak working memory (at least, as assessed by these types of test). Such individuals are likely to show up as false positives on a dyslexia screening test, and it will be remembered that in section 2.4.2 it was pointed out that the LADS Plus Working Memory test was found to have a somewhat higher incidence of false positives. This means that extra caution should be exercised when interpreting the results of this particular test, a point that is explored further in section 4.2.4

Gender differences

In Study A, in which the full form of the LADS Plus tests were used, there were 58 males and 82 females. No significant gender differences were found, except in the Word Recognition test, in which the females (mean score 94.45; SD 20.34) were found to score significantly higher than the males (mean score 83.81; SD 22.70), p < 0.01).

In Study B, in which the adaptive form of the LADS Plus tests were used, there were 19 males and 29 females. No significant gender differences were found in scores obtained from any of the three LADS Plus tests, nor in the LADS Plus composite score (see Table 12). As a check against this, the same students were administered three conventional tests that assessed comparable skills: Woodcock-Johnson Word Attack Test, WRAT 3 Spelling Test, and WAISIII Digit Span Test. Although the mean scores suggested some slight differences, none of these were found to be statistically significant (t test). It may therefore be safely concluded that the tests in the Developmental Version of LADS Plus do not bias either males or females in an unselected sample.

Table 12. Gender differences in scores from LADS Plus and three conventional tests.

Validation Study C

38 university students (19 dyslexic and 19 non-dyslexic) were tested with LADS Plus.11 These students had been selected so that the two groups were matched on intelligence using the Wechsler Adult Intelligence Scale (WAIS–III UK). Diagnosis, in the case of the dyslexic students, was made on the basis of extensive psychological testing that followed the assessment criteria laid down in the report of the National Working Party on Dyslexia in Higher Education (Singleton, 1999). The mean IQ for the dyslexic group was 112.16 (SD 11.3), and for the nondyslexic group was 112.29 (SD 10.08). Analysis of variance indicated that there were no significant differences between the groups in intelligence. The LADS Plus scores on the adaptive form of the test for the two groups are shown in Table 13.

Table 13. Mean LADS Plus scores for the dyslexic and non-dyslexic groups in Validation Study C (standard deviations in brackets).

Analysis of variance indicated that there was a significant difference between the groups in performance on both the Word Recognition test [F(1,38) = 10.39; p<0.05] and Word Construction [F(1,38) = 9.92; p<0.05], but the difference between the groups in performance on the Working Memory test was not significant. However, the LADS Plus composite score (created by addition of the scores on all three dyslexia-sensitive tests) showed a significant difference between the groups [F(1,38) = 9.59; p<0.05]. These results indicate that overall, LADS Plus can discriminate at the group level between dyslexic and non-dyslexic adults even when intelligence is controlled for. This finding also holds for the individual tests of Word Recognition and Word Construction, but not for the Working Memory test. Inspection of the standard deviations in Table 13 shows that the variance in scores was much larger in the Working Memory test compared with the other two tests. In fact, the score distributions for the two groups on the Working Memory test overlap quite a bit, indicating that many bright dyslexics have probably developed compensatory strategies that enable them to cope fairly well with working memory tasks, while some bright non-dyslexics have surprisingly poor working memory skills (at least, on this particular test)

11 The authors are grateful to Fiona Simmons, who supervised and collected most of the data for Validation Study C.

Development and validation of the reasoning modules

The nonverbal reasoning test in LADS Plus has been adapted from the Reasoning module in the computerised assessment suite LASS Secondary (Horne, Singleton and Thomas, 1999). This has been validated and shown to be reliable and free of gender bias in a number of separate studies (for further information see Horne, 2002). In a study with 75 students (47 males and 28 females) from five different secondary schools in England and Wales, the nonverbal reasoning module was validated against the Matrix Analogies Test – Short Form (Naglieri, 1985). This is a conventional pencil-and-paper test of matrix reasoning. The correlation coefficient between the two measures was 0.52 (p <0.001). In a separate reliability study involving seven other secondary schools in England and Wales, the nonverbal reasoning module was administered to 101 students on two occasions, separated by four-week interval. The results showed no significant differences in the results on the two testing occasions (t test) and the test-retest reliability coefficient was 0.51 (p <0.001). Finally, in another study involving 341 male and 389 female students, no significant gender differences were found on the nonverbal reasoning test [male mean 35.45 (SD 9.76); female mean 34.53 (SD 10.29)].

The verbal reasoning test in LADS Plus has been adapted from the verbal reasoning module in the computerised assessment suite Lucid Ability (Singleton, Horne and Thomas, 2005). This was validated against the NFER Nelson Verbal Reasoning Test, a conventional pencil-and-paper test of verbal reasoning, using a sample of 124 students. The correlation coefficient between the two measures was 0.65 (p <0.001). Gender differences were investigated in a separate study involving 57 females and 53 males. No significant differences were found between males and females in the verbal reasoning test.

The LADS Plus studies 2004-05

The development of LAS Plus involved several further research studies, including a major project carried out during 2004-05 at HM Youth Offender Institution, Wetherby, in collaboration with the University of Hull and the British Dyslexia Association. The aim was to determine ways in which LADS might be modified to increase is screening accuracy when used with individuals who may have non-standard educational backgrounds. The project was carried out in three phases. Phase 1 was conducted during the Spring of 2004 and involved collecting data from an unselected sample of 116 male juvenile offenders aged 15–17 years at Wetherby YOI using LADS and conventional tests of reading and spelling. The results indicated that as a screening tool LADS was acceptable to this population and straightforward for staff to administer but that in its standard form it produced an unacceptably high incidence of false positives (i.e. cases that may have been wrongly classified as having a high probability of dyslexia).

Phase 2 was conducted during the summer and autumn of 2004 and involved assessing cognitive and literacy skills in a selected sample of 36 male juvenile offenders aged 15-17 years, 18 of whom had high probability of dyslexia and 18 of whom had low probability of dyslexia. The results showed that a very large proportion of the young offenders had low verbal ability, which is consistent with findings from similar studies. To a large extent, this is likely to be due to educational and social disadvantage as well as to lack of reading experience, which contributes significantly to vocabulary growth, especially after the primary stage of education. By contrast, the average non-verbal ability of participants in all phases of the study was found to be within the average range. These results indicate that in this population it is critical to have an instrument for identifying dyslexia which will allow for low levels of reading ability, low verbal intellectual skills and lack of educational opportunities. It was concluded that by incorporating a test of verbal ability into LADS and modifying the classification rules used by the program it should be possible to identify those young offenders who have dyslexia with a satisfactory degree of accuracy while reducing the number of false positives to more acceptable levels.

Phase 3 was carried out during early 2005 and involved administering a modified version of the computer-based screening test that incorporates a measure of verbal reasoning (LADS Plus) to a new unselected sample of 62 juvenile offenders aged 15-17 years. The results indicated that about 31% of young offenders at Wetherby YOI have dyslexia, a figure that is reasonably consistent with comparable studies that have used cognitive tests (as opposed to check lists and rating scales). A further 32% showed borderline symptoms.

The overall conclusions of this project were that computerised screening using the modified program LADS Plus is a practical and efficient solution for identifying dyslexia in juvenile offenders. The report of this project, entitled Practical Solutions for Identifying Dyslexia in Juvenile Offenders, was published in 2005 by the British Dyslexia Association.