The design of the isiZulu PRR Test closely follows the structure and procedures of the English PRR Test developed by the first author,¹¹ with the primary difference being the use of words sourced from isiZulu textbooks commonly used in South African primary schools. The test also shares similarities with the Wilkins Reading Rate Test,⁴ but differs in several key aspects such as: the number of words included in each paragraph, the choice of words in terms of language and the inclusion of versions to accommodate different levels of visual acuity. While the overall test format remains consistent with the English version, these adaptations ensure that the isiZulu PRR Test is linguistically and visually appropriate for isiZulu-speaking learners. Considering the intent of the optometric isiZulu PRR Test, the selection of the words, their placement in each version of the test, visual acuity letter size and print scaling, font type, contrast and the design of the pre-test chart and recording scoresheet will be discussed.

Grade one isiZulu prescribed textbooks in Portable Document Format (PDF) were imported into the NVIVO qualitative data analysis software (version 11), and by using the Word Frequency Query in the program, a list of frequently used words was generated. Words with only three to four letters were selected to ensure an average level of word difficulty, as in the case of the English PRR Test.¹¹ Numbers, English words, punctuations, special characters, abbreviations and common nouns were excluded. From the generated list of 12 words, 10 words were randomly chosen, arranged left to right, consistent with the reading direction of isiZulu and assigned to 10 rows per paragraph. A total of six paragraphs were produced with each paragraph corresponding to one of six test versions (A–F). These paragraphs were positioned centrally on each version, without any outlining borders. Although no physical borders were drawn, consistent white space was maintained around the text, equal in thickness on the top, bottom, left and right to standardise the visual presentation across all test versions. Furthermore, consistent spacing between texts and lines in all versions was maintained using computer-generated print to avoid the impact of the crowding effect on reading as suggested by previous studies.^16,17

The placement of words within each paragraph was determined using a randomisation method: each of the 12 words was assigned a number and a list of random numbers was used to guide 10-word placements per line. This ensured that each line and paragraph was unique in its word order. Consideration of the foregoing in the design was to mimic eye movement during reading, standardise reading difficulty, minimise the influence of linguistic factors and standardise the length and level of difficulty per line and version.

Each of the six versions (A–F) of the chart included text to measure near visual acuities of 1.0M, 1.2M, 1.6M, 2M, 3.2M and 4M, respectively, with notations in Snellen and Logarithm of Minimum Angle of Resolution (LogMAR) also included. Print smaller than 1.0M is not commonly found in books of primary school children, hence it was not included in the design, while relative distance magnification could be applied if print larger than 4M is required.¹¹ To ensure the appropriate scaling of the print for each test version, a graphic designer was enlisted and the formula x = d tanq was applied, where x is the letter height (in mm), d is the viewing distance (in mm) and θ is the angle (5 minutes of arc) that each letter subtends at the eye.¹⁸ The print size was maintained uniformly within each paragraph or test version and did not vary between lines within a paragraph. The scaling excluded the ascenders and descenders of letters such as b, d, g and h. Du Toit¹⁷ recommended that the typeface chosen for children’s reading tests should reflect what they are commonly exposed to, hence the Quick Sand font typeface was considered to replicate prescribed textbooks and the teaching of writing skills of South African primary school children. Versions A to E were printed on individual 29.7 centimetre by 19.5 cm (A4) matte cardboard, while Version F was on 36.0 cm by 19.5 cm cardboard, considering the large print size. Complying with the contrast specifications in research, all text was printed in black on a white background.¹⁹ For each version of the test (A–F), corresponding scoresheets were designed to provide the examiner a simultaneous opportunity to follow the reader while documenting errors and the time necessary for calculating the reading rate.

A pre-test chart was designed to ensure the child is familiar with the words before administering the test irrespective of the version used. The selected 12 words were presented on a matte white A4 cardboard, organised in two columns comprising six rows each. The words were printed in large font (5.82 mm) corresponding to an acuity level of 4.0M, in black ink with a white background to ensure legibility for children with a range of visual acuity levels.

From a list of public mainstream primary schools, with isiZulu being offered as the HL and FAL, two schools were selected by simple random sampling. Learners from Grades one to seven (ideally corresponding to ages 6–12 years) were recruited, as the isiZulu PRR Test was specifically designed for this primary school age range. The total number of normally-sighted participants in the study was determined to be 162 with 102 from isiZulu HL and 60 from FAL schools and was based on an expected power of 0.8, an effect size of 0.5 and a significance level of 0.05,²¹ with the participants being selected through convenience sampling. The first 162 isiZulu-speaking children who agreed and received consent to participate in the study, passed the pre-test, pre-visual screening and had no systemic illness or special needs according to the school records, were included.

The study was conducted in a three-metre by four-metre classroom, with consistent lighting provided by overhead fluorescent lights and natural light entering through windows located away from the testing area. Although the exact lux level was not measured, all fluorescent fixtures were operational to maintain uniform illumination throughout data collection. Only the two researchers and one participant were present in the room at any point in time, aiming to minimise distractions and create a comfortable environment for the participant to read without external pressures.

The researchers, one English-speaking and the other isiZulu-speaking, followed all protocols within the scope of optometric practice. To enhance standardisation, a detailed step-by-step guide was accessible to the researchers throughout data collection, and each examiner focused on a specific aspect of the research to ensure reliable and valid results.

The learners underwent the optometric isiZulu PRR Test pre-test visual screening and pre-screening as the first steps and those who passed both these initial procedures proceeded to the reading rate evaluation for the reliability of the optometric isiZulu PRR Test.

The necessary testing materials included the isiZulu PRR Test and scoresheet, a stopwatch, a 40 cm string and pens which were available throughout the data collection. Test instructions to each participant were provided in isiZulu and English, and participants were advised not to use their fingers to track words while reading. To ensure consistency with the learners’ schooling materials, version D (2.0M) of the isiZulu PRR test was considered in measuring the reliability of the isiZulu PRR Test. Furthermore, none of the participants had visual acuities incompatible with the chosen version, which was less than 2.0M.

The participant was instructed to read the test chart placed on the desk positioned 40 cm away with the distance maintained with a 40 cm string connected to the chart. Participants were directed to view the chart with both eyes open, adopting a natural downward reading gaze and to read the words aloud as quickly and accurately as possible. Upon being signalled to begin, the participants commenced reading, and the stopwatch was started. Upon finishing the 100 words, the examiners ended the timing. The examiners were able to follow along with the participant using the corresponding version’s scoresheet, with each examiner recording on an individual scoresheet. Errors, including additions, omissions, transpositions or substitutions were noted.^11,23 Based on the following formula (Equation 1),^4,11,24 reading rate was determined and recorded for each participant on their respective scoresheet.

The second reading rate evaluation on the same 162 participants was determined one week later (re-test or session two) using the same testing procedure as discussed. The result for each participant was documented on the same scoresheet per examiner but with a different coloured ink pen to identify test and re-test recordings. A one-week interval between the test and re-test, consistent with previous studies,^11,25 was used to minimise potential improvements in reading rate because of learning effects from prior exposure to the text. While improvements in reading performance may occur over time, this interval is considered sufficissent to mitigate immediate learning effects that could artificially enhance reliability estimates.

When multiple examiners are involved in administering and recording the results of a tool, the consistency between them is referred to as inter-rater reliability.³³ This study evaluated inter-rater reliability by comparing the results of total errors, time, and reading rate of the isiZulu (Z) and English-speaking (E) examiners on the same participants, aiming to determine if the isiZulu PRR Test could be used reliably by different examiners in the future who are not necessarily isiZulu-speaking. As the test was administered on two days (one week apart), inter-rater reliability was established on day one or session one (Z1:E1) and day seven or session two (Z2:E2) for both the isiZulu HL and FAL school children and will be discussed accordingly based on the Bland and Altman, ICC and one-sample t-test statistical analysis.

Some level of inconsistency was observed in the recording of total errors between the Z and E examiners reflected in the Bland-Altman plots (94%, 96/102 and 93%, 56/60) and one-sample t-test (P = 0.027) for session 1 at the FAL school. This variability could be related to the first-time exposure to the recording of errors and the level of experience between the examiners, with the E examiner having more exposure and experience than the Z examiner, hence contributing to the weaker inter-rater agreement for total errors. This inconsistency between examiners was similarly reported in a previous study using the MNREAD Chart, whereby more discrepancy was identified with the novice examiner.³⁴ This ‘interpersonal dynamics’ between examiners may negatively affect objectivity, which can be reduced by including examiners of equal expertise in reliability testing.³⁵ Furthermore, the considerable improvement in inter-rater reliability observed in session two at the HL school suggests that practice and experience make the results of the examiners more consistent and add value to the reliability of the test. A previous study reported that inter-rater reliability may be affected by the ‘learning effect’ or ‘fatigue effect’, with the former showing improvement in recordings between examiners in session two, while the latter reflects a decline.³⁵ It is therefore evident in the current study that the improvement can be attributed to the ‘learning effect’ between examiners. While this may reflect procedural familiarity between examiners, it is also possible that a learning effect occurred among the participants themselves, not only in terms of exposure to the reading text, but also with the testing format and expectations, potentially contributing to more consistent performances. However, as the design of the isiZulu PRR test consisted of random placement of words minimising memorisation of the test content, and that each participant was assessed by both examiners at the same time in the same session, keeping the testing procedure consistent, the impact of participant learning would least likely affect inter-rater reliability and have a greater influence on inter-session reliability or the overall validity of the test with the latter being included in a subsequent article. Following intense scrutiny of the Z and E examiners’ scoresheets, variations in the recording of the type and number of errors between both examiners were observed, such as the recording of transposition as two substitutions, while a substitution may be recorded as one omission and one addition. In both instances, two errors may be recorded instead of one, which can also add to this discrepancy. Pre-test discussion and agreement on recording conventions such as the classification of errors between examiners may help eliminate such discrepancies. Finally, the subtle difference in the words ‘yimi’ for ‘yini’, and vice versa, may go unnoticed or be recorded as a substitution error, thereby affecting the number of errors recorded. A study performed on an isiZulu cohort revealed that subtle differences in spoken words are easily identifiable to a listener from the same linguistic background, in this case,³⁶ the Z examiner. A recommendation to future examiners is to ensure that the environment is free of distractions so that subtle differences are detected. Concepts of semantics, vocabulary, words, and language structure influence the reception of spoken words. While this may have influenced the recording of total errors in this study, the findings do not significantly affect the inter-rater reliability of the isiZulu PRR test. Further, as this study includes primary school children, a 95% CI may prove too stringent for this cohort hence requiring adjustments and clinical judgement.

The Bland-Altman plots, ICCs and one-sample t-test analysis for total time and reading rate display good inter-rater reliability, excellent linear relationship (ICC ≥ 0.986) and no statistical differences between the Z and E examiners at the HL and FAL schools (P > 0.05), except for reading rate (P = 0.007) in session one at the FAL school. As the reading rate is inversely proportional to total error and total time,^11,24 the recording of the number of errors and time will affect the reading rate. With the recording of time being consistent between the examiners, the discrepancy then lies in the recording of errors. A careful inspection of the mean difference for total errors at the FAL school reflected that either the Z examiner recorded more errors, or the E examiner recorded fewer errors. This, in turn, may have contributed to a decrease in reading rate determined by the Z examiner or an increase as determined by the E examiner. Together, these recordings may have had an additive effect on the reading rate results. As stated in a previous study, inter-rater reliability and agreement may be affected by the level of examiner training and hence should be factored into the process prior to data collection or usage of the test.³⁷

Finally, the inter-rater results indicate that the reading measures are consistent, as the majority of them fall on and within the 95% LoA with P-values showing no statistically significant difference between the means of the examiners (P > 0.05), while the ICCs for all variables range from 0.848 to 1.000, suggesting good to excellent inter-rater reliability between the Z and E examiners when using the isiZulu PRR Test on isiZulu HL and FAL school participants.

Inter-session reliability involves evaluating the same participant multiple times over days, weeks, or months by the same examiner.^31,38,39 In this study, measurement was performed by the same examiner (Z or E) testing the same participant on day one (Z1 and E1) and one week later (Z2 and E2), using version D of the isiZulu PRR Test to assess total errors, time, and reading rate. Reliability is presented for each examiner as Z1:Z2 and E1:E2 and analysed using Bland-Altman plots, ICC, and one-sample t-test at isiZulu HL and FAL schools.

The inter-session averages for total errors and time obtained by both the examiners, at the HL and FAL schools, are higher in session one compared to session two (reflected by a positive value). This indicates that the participants made more errors, and took longer to read during session one, with performance improving in session two. This improvement is likely attributable to the participants’ initial exposure to the test stimuli, alongside an adaptive learning effect characterised by increased familiarity with both the assessment items and the testing protocol. Such learning effects may have influenced the outcomes and should be acknowledged as a potential confounding factor when interpreting reading performance improvements across sessions. By incorporating a practice or familiarisation trial before formal testing, participants can become accustomed to the testing format, procedures and expectations, thereby reducing performance improvements attributable to procedural learning.

Furthermore, the Bland-Altman plots for the Z examiner only reflect that 94% and 93% of the inter-session results for total error lie within the LoA in the isiZulu HL and FAL schools, respectively, indicating inconsistencies with the recording done by the Z examiner as opposed to the E examiner. This too could be attributed to the first-time exposure of the Z examiner to the testing conditions, recording of results as well as clinical experience, whereby the E examiner had more years of experience in the administration of similar assessments and clinical practice. Such differences in expertise as well as communication style or procedural familiarity may therefore negatively affect objectivity.³⁵ As suggested for inter-rater reliability, stronger inter-session reliability can be expected with proper training of examiners before administration of the test.³⁵

The inter-session Bland and Altman plots for time recorded by the Z and E examiners at the HL school, fell within the 95% LoA, but did not at the FAL school. The variation in total time could result from incorrect timing by the examiners either starting the timer too soon or stopping it too late, negatively impacting the inter-session recording of total time. Coincidentally with this trend observed for total time by both the Z and E examiners, this discrepancy in sessional results could be attributed to variation in participants’ responses. Such variation influenced by inherent or environmental consequences, such as the development of growth during the period between measurements and an increase in vocabulary, has a considerable influence on the reliability of the results.^32,40 Furthermore, reading speed (time) has shown considerable variability over time impacting on reliability results.⁴¹

Finally, the mean difference for reading rate indicates a higher result for session two (negative value), supported by the lower number of errors and time in session two. Considering the inverse relationship between total error and time to reading rate,^11,24 with its compounding variation of both total errors and time in this session, this could have contributed to the increase in reading rate in session two. This slight inconsistency in the inter-session reading rate results is evident in the Bland-Altman plots at the HL school for both examiners (93% and 91% within the LoA), respectively. A possible explanation could be more exposure of the participants to the isiZulu language in the HL school than the FAL school during that week, which resulted in the improved reading rate observed in session two. In addition, participants’ learning effect, following initial exposure, may have contributed to the improved performance and should be considered when interpreting future results. With the impact of developmental growth during the period between measurements, and the increase in vocabulary of a school child acquiring about seven new words a day,³² hence impacting inter-session reliability, the application of the 95% LoA may not be ideal. Taffé et al. confirm that the width of the LoA for the Bland-Altman plots will apply only if the bias of the test and the measurement is constant with no variability during the entire testing procedure.⁴² This therefore verifies that a 95% LoA may prove too stringent in the case of reading rate assessment in children hence the acceptability of the inter-session reliability of the isiZulu PRR test for reading rate measurement at the isiZulu HL school being interpreted as good.

While the P-values for all reading measures were statistically significant, reading rate values of less than 10 are considered to be clinically insignificant,^43,44 hence clinical relevance outweighs the statistical conclusion.⁴⁵ Finally, the ICCs for total errors, time, and reading rate revealed moderate, moderate to excellent, and excellent inter-session reliability respectively at HL school, while good to excellent inter-session reliability for total errors, time, and reading rate for both examiners at the FAL school were observed. With the isiZulu PRR test showing inter-session reliability, it is suggested that before any conclusion could be drawn, repeated measurements are encouraged with a smaller interval rather than one week for true reading performance evaluation to reflect stronger reliability.

As noted, the potential for a learning effect among participants must be considered, particularly in repeated testing scenarios. While changing the interval between test and re-test is a common strategy, studies have shown that learning effects can still persist, especially in school-aged children who are rapidly developing cognitive and linguistic skills.³² To mitigate this, standardised instructions and examiner behaviour, can help minimise procedural familiarity. Brief practice trials prior to the first test may also help stabilise early performance variability by allowing pupils to adjust to the testing process without contaminating the actual test data. Furthermore, limiting the visibility of scoring or feedback during testing may reduce behavioural adaptations. These strategies, combined with ongoing examiner training, can support more accurate and generalisable measures of reading performance in paediatric populations.