Scholarship and Research

Scholarship Related to Teaching and Learning

[1]  Alamyar, M., & Bush, D. H. (2018, March 27–30). Writing for broader digital audiences [Electronic Village Event]. Teaching English to Speakers of Other Languages (TESOL) International Convention & Expo, Chicago, IL.

For this project, students work in pairs to take a position on one of the given broader topics and, then create a website to place their argument paper in it to showcase their writing to broader audiences. For demonstration, in 10 minutes, I will introduce the overall project along with criteria for both writing an argumentative paper and creating a website through a website called Weebly. Then, I will take 15 minutes to demonstrate how to create the website, place the content in it, and use some apps and websites to edit website, content, images, charts, graphs, etc.

[2]  Alamyar, M., & Bush, D. H. (2019, March 12–15). Kahoot: The ultimate engaging and power tool for ELs [Paper Presentation]. Teaching English to Speakers of Other Languages (TESOL) International Convention & Expo, Atlanta, GA.

Integrating effective educational technology can be challenging and time-consuming for TESOL educators. This presentation demonstrates how educators can utilize Kahoot! for English language learning in higher education. The attendees learn how to use Kahoot! for assessment, review of materials, comprehension checks, warm-up activities, surveys, and class discussions. 

Scholarship Related to Testing, Evaluation, & Assessment (TEA)

[1] Allen, M. C. (2016, October 1). Measuring ESL reading fluency development with assisted repeated reading [Poster Presentation]. 18^th Midwest Association of Language Testers (MwALT) Conference, West Lafayette, IN.

Much remains unknown about how to define, measure, and develop reading fluency for ESL students at different proficiency levels (Anderson,1999; Grabe, 2009; 2014; Lems, 2012; Taguchi & Gorsuch, 2012). These practical and theoretical issues are addressed in this presentation of findings from a single-case design (ABAB) study with a small number of participants (n=12 university students).  Two research questions are addressed: (1) What are participants’ silent and oral reading rates? (2) Does use of an audio-assisted repeated reading program contribute to increased reading rates? In this study, reading fluency is operationalized as the number of words per minute read by participants, and is supported by several other measures to indicate participants’ accuracy and comprehension.  Empirical data for selected participants will be displayed in graphs showing their (1) recognition vocabulary; (2) baseline measurements of silent and oral reading rates at multiple points in time; and (3) progress in the repeated reading intervention. The graphs will show how silent and oral reading rates compare within and across participants, and the extent to which the reading intervention (IV) increased participants’ reading fluency rates (i.e., led to a stable change in the DVs). The use of graphic displays to visualize quantitative data is a hallmark of single-case designs, and the method of careful visual analysis of data to identify trends translates well to a poster session. This study advances our understanding of L2 reading fluency, with implications for assessment, curriculum and instruction, and student motivation.

[2] Cheng, L. (2018, September 21–22). Extending the validity argument for an in-house ESL proficiency test through test score gains [Paper Presentation]. 20^th Midwest Association of Language Testers (MwALT) Conference, Madison, WI.

Longitudinal tracking of ESL test performances not only contributes to program evaluation, but it also extends the validity argument for the language test used. A validity argument is supported when test scores reflect appropriate change as a function of construct-related teaching or learning (Cronbach, 1971; Messick, 1989). The Assessment of College English–International (ACE-In), a locally developed, computer-mediated, semi-direct English proficiency test has been used as embedded assessment in an English for Academic Purposes (EAP) program at a large public university for purposes such as providing diagnostic information to language teachers, tracking students’ language development, and gathering information for program effectiveness. This EAP program provides language and culture support to matriculated international undergraduate students with relatively lower TOEFL iBT or IELTS scores. This study focused on a machine-scored cloze-elide section and a human-rated elicited imitation section on the ACE-In. Test items in both sections were developed based on well-defined test specifications and pilot-tested before operationalization. Cloze-elide assesses vocabulary, grammar, and silent reading whereas elicited imitation assesses listening comprehension, information retention, and grammatical accuracy in oral production. Both sections have high internal consistency estimates and small standard errors of measurement. The cloze-elide score has a significant, moderate correlation with the TOEFL iBT writing score (r = .42, p = .04) while the elicited imitation section score has a significant, strong correlation with the TOEFL iBT speaking score (r = .74, p < .0001). Test data gathered at the beginning of a two-semester sequence, at the end of Semester 1, and at the end of Semester 2 indicates that the international students in this EAP program made gains in each semester (Cloze-elide: χ² = 217.07, p < .001; Elicited imitation: F = 41.28, p < .001, η_ρ² = .40). This evidence of score gains helps to extend the validity argument for ACE-In.

[3] Cheng, L. (2019). Diagnostic assessments. In J. Liontas (Ed.), The TESOL Encyclopedia of English Language Teaching. https://doi.org/10.1002/9781118784235.eelt0493

Diagnostic assessments are characteristic of providing detailed feedback about the strengths and weaknesses—especially weaknesses—in learner performances to monitor learning, guide the evaluation of learner progress, and direct the learner to resources for future practice, in addition to informing instructional design and revision. While tailored diagnostic assessments are rare due to their narrow focus on specific language elements, a few issues need to be considered when retrofitting a diagnostic function to an existing assessment developed for other purposes. In the recent surge of interest in diagnostic assessments, researchers are called on to investigate more into the transitions from diagnosis to intervention; language educators are encouraged to achieve a balance in assessment procedures both inside and outside of the classroom and to try to complement the weakness-based diagnostic information with the formative feedback on strengths provided by other assessments.

[4] Cheng, L. (2023, November 18). Brightspace for training ESL test raters [Poster Presentation]. 2023 Indiana Teachers of English to Speakers of Other Languages (INTESOL) Conference, Indianapolis, IN.

In behavioral sciences, regular rater training is essential whereby raters follow self-training instructions or engage in group discussions to ensure reliable and fair evaluation of learner performances. This poster showcases a project where a Brightspace course is being developed and implemented for training teacher-raters of our in-house English proficiency test.

[5] Cheng, L. (2024, March 16–19). Exploring the relationship between prompt repetition in timed independent writing tasks and second language proficiency [Poster Presentation]. 2024 American Association for Applied Linguistics (AAAL) Conference, Houston, TX.

Despite the utility of independent writing addressing a prompt, there is limited research on second language (L2) learners’ utilization of the prompt language with apparent effects such as altering expressions and framing the writing. This study explored the relationship between prompt repetition and L2 proficiency (operationalized as TOEFL iBT scores) through analyzing 50 English as a Second Language (ESL) timed essays written by L1-Mandarin first-year international undergraduate students at a large public university in the Midwestern US. A coding and ranking scheme was developed, based on Shi (2004) and Voloshinov (1929; 1986). Independent codings by two coders achieved inter-coder reliability of .85 for the agreement on the quality of prompt repetition and .94 for the agreement on the lengths. After coding disagreements were resolved, two central dependent variables characterizing prompt repetition were derived: 1) the average rating of prompt repetition, a quality-focused variable; 2) the normalized length of prompt repetition, a quantity-focused variable. The average rating was a composite score based on codings for accuracy, syntactic and morphological reformulation, and slight modification of prompt language. The normalized length of prompt repetition was calculated by norming the total word count of repeated prompt to a common base of 100 words. A strong, positive correlation was found between the average rating of the quality of prompt repetition and the learner’s TOEFL iBT total score (r = 0.84, p = 0.019); no significant correlation was found between the TOEFL iBT total score and the normalized length of prompt repetition. These findings indicate that L1-Mandarin ESL learners of higher proficiency are more likely to accurately produce sophisticated reformulations of the prompt language, but there is no decisive association between the extent of prompt repetition and L2 proficiency. This research sheds new light on how prompt repetition in independent L2 writing relates to L2 proficiency.

[6] Cheng, L., & Allen, M. C. (2016, March 11–13). Timed oral reading: A useful method for L2 reading fluency assessment and intervention [Paper Presentation]. Georgetown University Round Table (GURT) on Languages and Linguistics 2016 Conference, Washington, DC. 

Timed oral reading is a widely used method to assess reading fluency in L1 instructional contexts; fluency is typically operationalized as the ability to read aloud accurately with appropriate expression (prosody, phrasing), at a speed, such that the oral reading aligns with the meaning of the text (Rasinski, 2004). Fluency is the key because it provides evidence that the reader understands what is being read (National Reading Report). By contrast, in L2 instructional contexts, the development of reading fluency and the use of timed oral reading for assessment (or instruction) is almost nonexistent (Anderson, 1991; 1994; Grabe, 2014; Chang, 2010, 2012; Taguchi, Gorsuch, Takayasu-Maass & Snipp, 2012). This study investigates the efficacy of timed oral reading as both an assessment and intervention method for evaluating and improving L2 reading fluency. A pre- and post-test was given to 77 international graduate and undergraduate students enrolled in an English for Academic Purposes (EAP) course at a large public Research I university in the Midwestern United States. Read Naturally®, a web-based program for improving L1 English reading fluency was used as the intervention tool with our ESL group; and eight texts at Grade 6 L1 English level were implemented as homework assignments and materials or topics for teacher-student conferences. At the end of the thirteen-week intervention, this group of ESL learners was found to have a statistically significant improvement, with a very large effect size, in reading speed: an increase from 131 words per minute (wpm) to 144 wpm. Our presentation explains why we are interested in oral reading fluency and why we felt it would be a useful tool for assessing and providing intervention to matriculated EAP students.

[7] Cheng, L., & Allen, M. C. (2016, May 19–21). Timed oral reading: A useful method for second-language reading fluency assessment and intervention [Paper Presentation]. 3^rd Conference of Asian Association for Language Assessment (AALA), Sanur, Bali, Indonesia.

Timed oral reading is a widely used method to assess reading fluency in first-language (L1) instructional contexts. Reading fluency is typically operationalized as the ability to read aloud accurately, with appropriate expression (prosody, phrasing), and at a speed, such that the oral reading aligns with the meaning of the text (Rasinski, 2004). Fluency is the key because it provides evidence that the reader understands what is being read (National Reading Report). By contrast, in second-language (L2) instructional contexts, the development of reading fluency and the use of timed oral reading for assessment (or instruction) is almost nonexistent (Anderson, 1991; 1994; Grabe, 2014; Chang, 2010, 2012; Taguchi, Gorsuch, Takayasu-Maass & Snipp, 2012). This study investigated the efficacy of timed oral reading as a learning-oriented assessment method for evaluating and improving L2 reading fluency. A pre- and post-test was given to 77 international graduate and undergraduate students enrolled in an English for Academic Purposes (EAP) course at a large public Research I university in the Midwestern United States. Read Naturally®, a web-based program for improving L1 English reading fluency was used as the intervention tool with our ESL group; and 8 texts at Grade 6 L1 English level were implemented as homework assignments and materials or topics for teacher-student conferences. At the end of the thirteen-week intervention, this group of English as a Second Language (ESL) learners was found to have a statistically significant improvement, with a very large effect size, in reading speed: an increase from 131 correct words per minute (cwpm) to 144 cwpm. When the study design was replicated the following semester with a different group of 92 adult ESL learners, using 8 texts at Grade 8 L1 English level, the average pre-test oral reading rate of 106 correct words per minute increased to a post-test of 121 cwpm, as compared to L1 adult speakers of English whose rates range from 150-200 words per minute. In addition to a report of this study and its replica over two semesters, this paper presentation will also be devoted to a discussion of why we are interested in oral reading fluency and why we felt it would be a useful, learning-oriented assessment tool for matriculated EAP students at this U.S. university.

[8] Cheng, L., & Crouch, D. (2018, August 2–4). The relationship between text borrowing patterns and second language proficiency [Paper Presentation]. 2018 Symposium on Second Language Writing (SSLW), Vancouver, Canada.

This study examined the relationship between text borrowing and second language proficiency, based on analysis of 50 L1 Chinese timed essays. Results showed a strong, positive correlation (r = 0.84, p = 0.019) between the extent of accurate, syntactic/morphological reformulation, and slight modification of text from the prompt and writers’ TOEFL iBT total score.

[9] Cheng, L., Ginther, A., & Allen, M. C. (2015, October 4–5). The development of an essay rating scale for a post-entry English proficiency test [Paper Presentation]. 17^th Midwest Association of Language Testers (MwALT) Conference, Iowa City, IA.

The dramatic increase in enrollment of international undergraduate students at U.S. universities not only reflects a national trend of shifting undergraduate demographics but also highlights the need for effective evaluation of newly admitted international students’ English language proficiency.  To better inform language instruction in an English for Academic Purposes program at a large public university, an internet-based post-entry English proficiency test, the Assessment of College English–International (ACE-In) was developed.  This presentation focuses on the development of an empirically derived rating scale for the writing assessment included in the ACE-In. Drawing on the literature of L2 rating scale development (e.g., Fulcher, Davidson, & Kemp, 2011; Upshur & Turner, 1995), we began by analyzing a sample (n=42) of first-semester international students’ ACE-In essays to identify the categories and elements (i.e., constructs and variables) present and emerging levels of performance.  A series of rating and discussion sessions were iteratively conducted with 33 additional essay samples until agreed-upon descriptors were established and an acceptable level of inter-rater reliability reached.  These rater norming sessions not only served the purpose of developing and refining an essay rating scale, but also helped to build a community of practice by providing a venue for raters to share what they value as writing instructors (Kauper, 2013). This presentation provides a practical example of developing an empirically derived rating scale for a timed writing assessment.  With the emphasis on instructor values, we provide a model for creating effective communities of practice through rating scale development. 

[10] Cheng, L., & Li, X. (2023). Pre-Post elicited imitation: Documenting student gains in the Purdue Language and Cultural Exchange. In Yan, X., Dimova, S., & Ginther, A., (Eds.), Local Language Testing: Practice across Contexts (pp. 103–125). New York, NY: Springer. https://doi.org/10.1007/978-3-031-33541-9_6

This chapter provides an account of a locally-developed Elicited Imitation (EI) test used to gauge international undergraduate students’ entry-level English proficiency and the use of the instrument to document end-of-year gains. Significant gain scores, with a strong associated effect, contributed to evidence in support of program effectiveness. A brief history of the Purdue Language and Cultural Exchange, an English for Academic Purposes program, is followed by a report on a three-stage quantitative analysis of the EI test, which has played an instrumental role in securing institutional support. Stage 1 examined the test’s measurement properties through the application of Classical Test Theory. After the verification of technical qualities in Stage 1, Stage 2 focused on pre-post test score gains and the effect size. In Stage 3, pre-post score changes were evaluated at the item level through the computation of Hedge’s g, an instructional sensitivity index for each item. Besides the pre-post score gains for program evaluation in Stage 2, this chapter also demonstrates the use of an instructional sensitivity index to complement the standard item-level analyses on difficulty and discrimination. Findings informed test revision and provided insights on program integration of assessment and instruction.

[11] Cheng, L., & Li, X. (2023, September 8–9). Pre-post elicited imitation: Documenting proficiency development in a two-semester EAP program [Paper Presentation]. 24^th Midwest Association of Language Testers (MwALT) Conference, Urbana-Champaign, IL.

Elicited Imitation (EI), a psycholinguistic task eliciting oral production of language chunks, is believed to measure L2 linguistic competence, through tapping into language learners’ implicit grammar and language processing capabilities (e.g., Davis & Norris, 2021; Van Moere, 2012). Our study examined a locally developed and rated, internet-based EI test used in an English for Academic Purposes (EAP) program at a large public U.S. university to document international freshmen’s English proficiency development and provide empirical evidence for program evaluation. We conducted total population sampling on the FA 2017–SP 2018 cohort of students enrolled in the two-semester EAP course sequence. A total of 261 students’ EI item scores and section-total score were analyzed in a three-stage quantitative study: 1) Classical Test Theory (CTT) analyses of measurement properties (e.g., interrater reliability, test form comparability, item difficulty, item discrimination, and test reliability); 2) Pre-post comparisons of section-total scores and a subsequent calculation of the effect size for score changes; 3) Computation of an Instructional Sensitivity index (Hedge’s g) per item. This EI test demonstrates excellent measurement properties; in addition, the statistical testing in Stage 2 indicates that the 261 first-year students made significant gains, with a large effect size, in their EI post-test as compared to the pretest seven months earlier. While the 12 EI items in each test form were sensitive to student improvement as a function of the ESL support received, only 17 (35.4%) of the total 48 EI items were found to be instructionally sensitive. We then discussed why the other EI items had a less than desirable Hedge’s g and which linguistic features contribute to high instructional sensitivity. This detailed item-level information (e.g., difficulty, discrimination, and instructional sensitivity) has practical value in informing test revision, especially for better capturing learner development and reflecting the integration of assessment and instruction.

[12] Cheng, L., & Mutti, M. (2024, October 4–5). Practicality of utilizing ChatGPT to develop items for a local language test [Poster Presentation]. 25^th Midwest Association of Language Testers (MwALT) Conference, Pittsburgh, PA.

As the best-known Large Language Model (LLM), ChatGPT holds great promise for expedited creation of language test items, including stimulus materials and answer options (Chun & Barley, 2023; Xi, 2023). This poster shows our work-in-progress in leveraging ChatGPT’s technological affordances to facilitate item writing for a locally developed and rated, internet-based, academic English proficiency test used at a large public university in the Midwestern United States. This local test constitutes an integral part of an English for Academic Purposes (EAP) program which promotes the linguistic and intercultural development of matriculated international undergraduate students with qualifying proficiency test scores (i.e., TOEFL iBT Total < 101, IELTS Overall < 8.0, or DET < 135). The primary use of this local test has been to inform high-stakes decisions on course placement/exemption and document proficiency development via pre-post designs implemented in a two-semester sequence of foundational courses. Efficient development of replacement forms has been necessitated by increased at-home administration of our test with digital proctoring by ProctorU® since COVID-19. The reasons for increased at-home administration include: 1) the students enrolled in our program in March 2020–July 2021 were e-learning at home; 2) the test-optional policy led to increased percentages of international undergraduate students admitted without scores on the three widely-recognized tests. We utilized ChatGPT to develop various item types, e.g., Elicited Imitation Task (EIT), Cloze Elide (CE), and Listen-to-Summarize Speaking Task (LSST), the latter two with fairly to very extended stimulus material. Preliminary results indicate that 1) the more complex and demanding the task specifications are, the more challenging it is to prompt-engineer efficiently and effectively for ChatGPT to generate a workable draft of the stimulus; 2) among the three types, LSST is where ChatGPT demonstrated the greatest advantage over human item-writers because of the least complexity and rigorousness of its task specifications.

[13] Crouch, D. (2018, March 24–27). Longitudinal development of second language fluency in writing and speaking [Paper Presentation]. 2018 American Association for Applied Linguistics (AAAL) Conference, Chicago, IL.

Little is known about how complexity and fluency develop together within individual L2 learners. This study analyzed the longitudinal development of oral fluency and global written syntactic complexity in the test responses of 60 first year L1-Chinese first year undergraduate students over two semesters. The author collected responses to a post-entry computer-administered language proficiency test required of all first year international students with TOEFL scores at 100 or below at a large university in the US. The students took the test at the beginning of the first semester and again at the end of the second semester of a required two course ESL sequence. For both the written and the spoken task, each student responded in support or opposition to a statement of opinion. The author analyzed the written responses automatically to calculate mean length of sentence using Lu's (2010) L2 Syntactic Complexity Analyzer and the oral responses for speech rate (syllables/second) and mean syllables per run (syllables/run) using a proprietary system specially designed to measure oral fluency. Results showed that the test-takers increased their oral fluency significantly but not their global written syntactic complexity. When the mean syllables per run of the oral pre-tests (M=7.27, SD=2.24) was compared to that of the oral post-tests (M=7.88, SD=2.39), there was a statistically significant difference, t(59)=3.58, p=.001. In a paired sample t-test the speech rate of the oral pretests (M=2.90, SD=0.43) was compared to that of the oral post-tests (M=3.00, SD=0.51), and there was also a significant difference, t(59)=2.55, p=.014.   Finally, a paired sample t-test compared the mean length of sentence of the written pre-tests (M=19.44, SD=4.43) to that of the written post-tests (M=20.37, SD=4.80), showing no significant difference, t(59)=1.57, p=.123. The findings provide evidence that oral fluency and written syntactic complexity develop at different rates in college level L1 Chinese L2 learners.

[14] Crouch, D. (2019, October 4–5). Pre-post change in L2 oral fluency: The lexico-syntax of large fluency gainers [Paper Presentation]. 21st Midwest Association of Language Testers (MwALT) Conference, Bloomington, IN.

The theory underlying L2 oral fluency has focused on cognitive processes, particularly proceduralization (Anderson, 1983; Levelt, 1989, 1999) and linguistic constructs, especially vocabulary and grammar (Segalowitz, 2010). Towell et al. (1996) argued that development of formulaic language enables automatic speech production. However, no research has studied the longitudinal development of L2 oral fluency concurrently with any of the following lexical variables: lexical frequency profile, formulaic language use, and MTLD (a measure of lexical diversity). The purpose of the present study is to clarify the process by which L2 oral fluency, syntax, and vocabulary develop concurrently. Data analysis involved three sequential phases: oral fluency analysis, lexico-syntactic analysis, and discourse analysis. Oral fluency measures were calculated using the transcribed oral test responses of 100 L1-Chinese EAP learners at the beginning and end of a required two-course EAP language and culture sequence at Purdue University. The task completed was a computer-administered, two-minute argumentative speaking task. This study included eight oral fluency measures: speech rate, mean length of speech run, articulation rate, phonation time ratio, mean length of silent pause, mean length of filled pause, silent pause frequency, and filled pause frequency. For the ten participants who made the largest percentage-wise oral fluency gains (in terms of the oral fluency variable associated with the largest effect size of gains), oral transcripts were analyzed to compute descriptive statistics for the three lexical variables mentioned above and three syntactic variables:  coordinate clause ratio, dependent clause ratio, and words per T-unit. Results indicated significant change in all oral fluency measures, except mean length of silent pause and mean length of filled pause. The largest gains were made in mean length of speech run. Of the linguistic variables, the largest longitudinal change was associated with coordinate clause ratio. Discourse analysis of the transcripts of large fluency gainers' pre-post responses suggested that large fluency gainers used coordinate clauses to build more sophisticated discourse models in the post-test response than they did in the pre-test response. Implications for L2 oral fluency theory, EAP pedagogy, and L2 oral assessment are discussed.

[15] Crouch, D. (2020). Pre-post change in L2 oral fluency: The lexico-syntax of large fluency gainers. [Unpublished PhD dissertation]. Purdue University.

The theory underlying L2 oral fluency has focused on cognitive processes, particularly proceduralization (Anderson, 1983; Levelt, 1989, 1999) and linguistic constructs, especially vocabulary and grammar (Segalowitz, 2010). Towell et al. (1996) argued that development of formulaic language enables automatic speech production. However, no research has studied the longitudinal development of L2 oral fluency concurrently with any of the following lexical variables: lexical frequency profile, formulaic language use, and MTLD (a measure of lexical diversity). The purpose of the present study is to clarify the process by which L2 oral fluency, syntax, and vocabulary develop concurrently. Data analysis involved three sequential phases: oral fluency analysis, lexico-syntactic analysis, and discourse analysis. Oral fluency measures were calculated using the transcribed oral test responses of 100 L1-Chinese EAP learners at the beginning and end of a required two-course EAP language and culture sequence at Purdue University. The task completed was a computer-administered, two-minute argumentative speaking task. This study included eight oral fluency measures: speech rate, mean length of speech run, articulation rate, phonation time ratio, mean length of silent pause, mean length of filled pause, silent pause frequency, and filled pause frequency. For the ten participants who made the largest percentage-wise oral fluency gains (in terms of the oral fluency variable associated with the largest effect size of gains), oral transcripts were analyzed to compute descriptive statistics for the three lexical variables mentioned above and three syntactic variables:  coordinate clause ratio, dependent clause ratio, and words per T-unit. Results indicated significant change in all oral fluency measures, except mean length of silent pause and mean length of filled pause. The largest gains were made in mean length of speech run. Of the linguistic variables, the largest longitudinal change was associated with coordinate clause ratio. Discourse analysis of the transcripts of large fluency gainers' pre-post responses suggested that large fluency gainers used coordinate clauses to build more sophisticated discourse models in the post-test response than they did in the pre-test response. Implications for L2 oral fluency theory, EAP pedagogy, and L2 oral assessment are discussed.

[16] Gao, J., Crouch, D., & Cheng, L.  (2019, October 4–5). Concept mapping for guiding rater training in an ESL elicited imitation assessment [Paper Presentation]. 21st annual Midwest Association of Language Testers (MwALT) Conference, Bloomington, IN.

Elicited Imitation (EI) has been integrated into second language (L2) assessments measuring examinees’ overall language proficiency (Tracy-Ventura et al., 2014) or examining L2 learners’ language development (Ellis et al., 2006). Our study focused on rater behavior when judging L2 learners’ EI performances on a local English proficiency test. Implementation of a 5-point holistic rating scale from 0 to 4, with rater training, has rendered high rater agreement (above .90 for R1/R2 correlation) at the section level. Raters, however, seem to operate with different priorities when making decisions at the lower end of the scale. We investigated 1/2 rater splits regarding the same item response. Two trained raters rated 56 examinee responses. Of the total 672 EI sentences, 125 1/2 splits were identified. Based on transcriptions and detailed error analyses, a Performance Decision Tree (PDT) was developed with the purpose of fine-tuning the decision-making process at the lower levels of the scale and helping raters align better with each other and with the rating scale. This PDT guides raters to make grammaticality judgements of each item response and then identify semantic deviations at the word level. While the grammaticality judgements cover grammatical accuracy, the semantic comparisons between the examinee’s version and the prompt include minor or major meaning deviations resulting from word substitution, addition, omission, or distortion (using a completely different word). Preliminary results show that 59 of the 125 sentences (47.2%) have grammatical errors. Semantic deviations appear in 98 sentences (78.4%), 50% of which result from word omission. Word addition, substitution, and complete distortion contribute 2%, 15.3%, and 21.4% respectively. The remaining 7% of semantic deviations result from combinations of the aforementioned categories. This study has contributed to our ongoing rater training, with the construction of this PDT to help raters navigate through the lower end of the rating scale.

[17] Levy, M., & Rodriguez-Fuentes, R. (2015). Analysis of C-test items from the ACE-In Test: A preliminary study of a Colombian population at Purdue University [Unpublished English 618 course paper]. Purdue University.

Language placement tests are a common method to assess students upon arrival at American universities. ACE-In, a test developed at Purdue for incoming international undergraduate students, was administered in this study. Scores for C-items of this test (Module I) were analyzed for their reliability in measuring language performance in a population of 22 Colombian students at Purdue. Variation in item difficulty and item discrimination was also analyzed. A second instrument, a questionnaire regarding educational background information, including specific English language instruction details, was also applied to the test population.  Item difficulty index for the ACE-In ranged from 0.50 and 0.95, making this moderately difficult module a useful test to assess a wide range of proficiency levels while maintaining consistency and item interdependence. Analysis of the item discrimination index showed a range between -0.10 (poor) to 0.60 (excellent). Cronbach’s alpha values varied little when items were removed one at a time, suggesting that sections are reliably constructed in a cohesive manner. Both instruments appeared to be promising in detecting variability in proficiency levels across a diverse population of Colombian participants. Overall performance for the ACE-In was good, with 14 participants scoring above 80%. There was a moderate positive correlation between ACE-In module 1 scores and time spent at Purdue. The degree of correlation between self-assessed language abilities and the scores was also positive but weak. On average, participants perceived that their language instruction prepared them adequately for reading and for listening, marginally for writing and poorly for speaking.  Although more research is needed, this study provided a good foundation for refining the instruments used to characterize Colombian students interested in pursuing either advanced degrees or research visits at Purdue.  This research also served as a supporting element in the design of an intervention ESP course for this target audience.

[18] Li, X. (2018, September 21–22). An evaluation of elicited imitation task: An item analysis using Classical Test Theory [Paper Presentation]. 20^th Midwest Association of Language Testers (MwALT) Conference, Madison, WI.

Elicited imitation (EI) has been widely used to assess second language (L2) proficiency. In an EI task, examinees are provided with a series of sentence stimuli and are expected to repeat the sentences as accurately as possible (Larsen-Freeman, 1991). EI task design has been frequently investigated because the appropriate design of the task is essential to the validity of the instrument (Erlam, 2006). The present study analyzes the performance on the EI task of the Assessment of College English-International (ACE-In) by performing thorough item analysis on pre-test and post-test exams using Classical Test Theory (CTT). One hundred undergraduate students who speak English as a Second Language took a pre-test at the beginning of the semester and a post-test at the end of the semester. The present EI task has four forms, and each form comprises 12 items/sentence stimuli. The item analysis results indicate that all four forms have shown similar item difficulty levels. Although a few items in each form have close item difficulty, the overall item difficulty and the spread of item difficulty are suitable for the current assessment purpose. When comparing pre-test and post-test exams, the item difficulty of most items has decreased. As for item discrimination, a few items that have low discrimination index may need modification or deletion, but the majority of the items have discrimination index above 0.6. Item discrimination index has mostly remained stable in pre-test and post-test. This study offers insights into the features of well-designed EI items and suggestions of future modification of the current EI task. In addition, the comparison between pre-test and post-test item analysis provides information regarding items that are suitable for different stages of assessment.

[19] Li, X. (2019, October 4–5). Providing reliability evidence for EI using G Theory [Paper Presentation] 21^st Midwest Association of Language Testers (MwALT) Conference, Bloomington, IN.

Elicited imitation (EI) is a widely used approach to assess second language (L2) proficiency. In an EI task, examinees are provided with a series of sentence stimuli with target language structures embedded, and examinees are asked to repeat the sentences as accurately as possible (Larsen-Freeman, 1991). In the late 1970s, EI received a series of critiques regarding its validity (e.g. Hood & Lightbown, 1978; Hood & Schieffelin, 1978; McDade, Simpson & Lamb, 1982). The major criticism is that examinees may complete EI tasks using mere rote repetition instead of L2 knowledge. The aim of the present study is to provide reliability evidence for EI via using Generalizability Theory (GT) and offer suggestions for the future improvement of the test administration. The EI task used in the present study was developed locally at Purdue University as a part of the Assessment of College English-International (ACE-In) to screen the English language proficiency of undergraduate students whose first language (L1) is not English. The test scores of one hundred fifty-nine freshmen were analyzed in this study. The result from the Generalizability study (G study) shows that examinee effect is accounted for 71.41% of the EI score variability. As the majority of the variance is contributed by the examinee effect, the study result suggests that the current EI test is a reliable measure of L2 proficiency. Meanwhile, more rater training sessions is also desirable as the rater effect (12.52%) claims the second main source of score variability. Although there are four forms of the current EI task, the form effect as well as the form-by-rater interaction effect are very small. A Decision study (D study) with 3 raters and 2 forms is also performed to explore further options of test administration. The generalizability coefficient of the given D study is 0.96.

[20] Li, X. (2020). The Analysis of Performance on the Elicited Imitation Task of the ACE-In [Unpublished doctoral dissertation]. Purdue University.

Elicited imitation (EI) was originally designed for first language (L1) development research, but since the 1970s, it also has been widely used in the SLA field. In the late 1970s, EI underwent a series of critiques regarding its reliability and validity. The major criticism is the possibility of mere rote repetition in the EI tasks. In recent years, a resurgent interest in EI has been witnessed along with an increasing number of empirical studies validating and refining the EI tasks. An item analysis using Classical Test Theory (CTT) was conducted in this project. By examining the item difficulty, item discrimination and score reliability, this study explores how the EI tasks function as a measure of general proficiency. This study analyzed 200 test samples of the Assessment of College English–International (ACE-In), which is a locally developed language test for post-entry international undergraduate students.

[21] Li, Y. (2016). The validity and reliability of grammar quizzes for ESL undergraduate students [Unpublished English 674 course paper]. Purdue University.

This project is to conduct item analyses for the GS 100 quiz of PLaCE program in Purdue University. The researcher has conducted statistical analysis for three GS 100 quizzes which were taken by over 200 participants. The elicited data presents several statistic results of the test which includes the testing coefficient of reliability and items’ indices (difficulty and discrimination indices). From analyzing the data set, the researcher have uncovered two noticeable findings regarding differences in coefficient of reliability amongst different forms and parts of a form when they have the same testing construct. The result of the explanation of these findings in the data is related to the item writing and also the item analysis format which is the 0/1 format. In addition, the researcher will also explain the part that the statistic results of this project fails to shed light on.

[22] Rodriguez-Fuentes, R. (2018). Linguistic and cultural factors in graduate school admissions: An examination of Latin American students at Purdue University [Unpublished doctoral dissertation]. Purdue University.

While the number of graduate students from different parts of the world in the United States is decreasing, the trend in Latin American populations is the opposite. Nonetheless, the current lack of information regarding the reasons behind this tendency, in terms of English language proficiency and cultural aspects, affects all parts involved: graduate students do not know what type of opportunities they can make use of; American universities do not have enough information to provide Latin American students with a sheltering environment; and Latin American governments are unable to make policies that encourage the application and facilitate admission to graduate school in American universities. The aim of this study is to establish a starting point for understanding the linguistic and cultural complexities of the Latin American population in graduate school in the United States. To do so, surveys and interviews were carried out to explore academic experiences, cultural influences and socioeconomic patterns that influenced the admission of Latin American students to graduate school. Mixed methods were used to describe the patterns of the survey responses quantitatively while leaving room for confirmatory quantitative analysis using the information of the interviews. The participants of this study were graduate students from Purdue University, one of the American universities with the highest number of Latin American graduate students. The results of this study underscore the importance of effective English language instruction during college years for reaching the graduate school admission scores, especially in cases when English language training during school was not possible or had little impact on the functional proficiency of the learner. Also, there is a large body evidence indicating that undergraduate research internships could be one of the opportunities with the highest potential to recruit graduate Latin American students, regardless of their socioeconomic background.

[23] Shin, J. (2021). Fluency, accuracy, and their relationships in second language development: Insights from cross-sectional and longitudinal analyses of elicited imitation performance. [Unpublished doctoral dissertation]. Purdue University.

Accuracy and oral fluency are important aspects of oral proficiency. Investigating objective features and scores of accuracy and fluency may provide diagnostic information on L2 proficiency and language development (Skehan, 2003). Elicited imitation (EI), an oral sentence repetition task, is a reliable and efficient test to measure accuracy and fluency (Van Moere, 2012), but less is known about using EI to diagnose fluency. My dissertation investigates the relationships among objective measures and human ratings of fluency and grammatical accuracy on the EI subsection of the Assessment of College English—International (ACE-In), a post-entry assessment of general English proficiency used in the Purdue Language and Cultural Exchange (PLaCE). Through the use of Python-based and R-programmed tools, acoustic fluency features and grammatical errors will be extracted and estimated; the relationships among objective features and corresponding holistic scores will be examined, in addition to changes in fluency and accuracy features in EI performance over 7 months of instruction in a sequence of two American language and culture courses, not covering EI, though. The findings from the cross-sectional and longitudinal observations will also shed insight on the roles of accuracy and fluency and dynamic trade-offs between fluency and accuracy in second language acquisition and development.

[24] Shin, J., & Crouch, D. (2018, March 21–23). Fluencing: A reliable and simple tool to measure temporal measures of oral fluency [Paper Presentation]. Language Assessment Research Conference (LARC), Ames, IA.

Temporal measures of oral fluency (TMOF) have been shown to be positively correlated with holistic scores of oral English proficiency (Ginther, Dimova, & Yang, 2010). This finding has positive implications for the use of measured variables in assessing oral English proficiency. However, the main obstacle to the use of TMOF in language assessment is the difficulty of computing such measures accurately, reliably, and efficiently. Solving these problems would provide language testers with more objective evidence of test-takers' language proficiency, thus improving the validity of interpretations that can be made from test results. To that end, the purpose of this demonstration is to introduce the use of a software application, Fluencing, as one solution. Fluencing is a software that automatically calculates TMOF based on audio-visual aided user annotation. Other speech analysis systems designed for acoustic analysis, for example Praat, can be used to compute TMOF, but they require more tedious and complicated calculation by hand. When using Praat, users have to count the number of syllables in each speech run, input the syllable count into the Praat syllable tier, and then manually calculate the TMOF. This process is time-consuming and prone to human error. Fluencing, a Python-based system developed by the Purdue Oral English Proficiency Program (Park, 2016), simplifies the process in a user-friendly way. It calculates speech rate, mean syllables per run, pause ratio, and expected pause ratio with the help of some basic user input or annotation. The automatic calculation of TMOF requires relatively simple user preprocessing. First, the user segments speech samples into pauses and speech runs and then annotates the speech runs using a built-in speech annotation function. The system then counts syllables using a customizable syllable dictionary to which the user adds all new words as they are encountered in speech samples. Our presentation will demonstrate this process of Fluencing using a response from an elicited imitation, also known as Listen and Repeat task. The responses were collected from a post-entry English language proficiency exam for college ESL speakers. The presentation includes a step-by-step demonstration of the graphical user interface and the annotation tools. In addition, we will report inter-annotator agreement on pre-test and post-test elicited imitation responses (n=72) The agreement between two annotators using Pearson correlation was .987 (p<.0001) for speech rate and .917 for mean syllables per run (p<.0001). The high reliability of this system will provide empirical support for studies examining the importance of fluency and thus, contribute to the understanding of the construct of fluent speech and human perception of it.

[25] Thirakunkovit, S. (2016). An evaluation of a post-entry test: An item analysis using classical test theory (CTT) [Unpublished doctoral dissertation]. Purdue University. 

This study is an analysis of test reliability of two screening tasks (C-test and cloze-elide) in the Assessment of College English-International test (ACE-In), a post-entry test developed at Purdue University. The study uses Classical Test Theory (CTT) to assess the reliability of these test items. CTT is selected because this theory is the standard comprehensive procedure for developing, evaluating, and scaling test items (DeVellis, 2006). This reliability analysis is important because it is a prerequisite to the test validation process. This study has three major research questions: 1. What is the item characteristics of C-test and cloze elide? 2. What are the average values of item difficulty and item discrimination of C-test and cloze elide items? 3. What are the internal consistency coefficients for and correlation coefficient between the C-tests and cloze elide tests? The results of the pilot study showed that the average score of C-test is 77.8 (SD = 9.98), and that of cloze-elide test is 36.59 (SD = 14.86). Considering the average values of item difficulty and item discrimination of both tasks, C-test items are generally considered easy (item difficulty > 0.7), while cloze-elide items are of medium difficulty (item difficulty ≈ 0.6). Even though C-test items have acceptable discrimination i.e., the average biserial correlation indices (r_pb) are 0.3, cloze-elide items are shown to have much better discrimination values on average i.e., r_pb indices are higher than 0.5. The Cronbach’s alpha coefficients, a measure of internal consistency, of C-test and cloze-elide are .88 and .96, respectively. The Pearson product-moment correlation analysis revealed that the correlation between the C-test and cloze-elide is high (r = .66), and it is significant with the p-value less than .01. These analyses of test reliability indicated that the test items were measuring the same underlying construct – generally language proficiency. Even though the key results of the item analyses showed that C-test did not meet the standard of item difficulty and discrimination, it does not necessarily mean that C-test cannot sufficiently serve its intended purpose as a preliminary screening tool. After examining the score distributions of both C-test and cloze-elide scores, the scores of both tasks range widely. With fairly wide standard deviations, there is a potential to combine the scores of these two screening tasks to identify the students who had a uniformly low performance across both tasks. 

[26] Yan, X. (2015). The Processing of Formulaic Language on Elicited Imitation Tasks by Second Language Speakers [Unpublished doctoral dissertation]. Purdue University.

The present study investigated the processing of formulaic language, in an effort to examine how the use of formulaic language may or may not contribute to second language (L2) fluency in speaking performance. To examine the effect of formulaic language on L2 fluency, this study utilized elicited imitation (EI) tasks designed to measure general English language proficiency in order to compare repetition of individual sentences containing formulaic sequences (FS) to repetition of sentences that do not. In addition to the presence of FS, the length of stimuli sentences was manipulated and compared to a second independent variable. Responses to EI tasks were automatically measured for articulation rate (AR) and number of silent pauses (NumSP), two important measures of L2 fluency. Repeated measures ANOVAs were conducted to examine the main and interaction effects of FS and sentence length (SL) on AR and NumSP. Results of analyses of EI performances showed that both SL and FS had a significant effect on L2 fluency in speech production; however, these two variables had differential effects on AR and NumSP. SL had a strong effect on NumSP on EI performances: as the stimulus sentence becomes longer, NumSP on EI performances increases. The presence of FS had a larger effect on AR than on NumSP: higher proportion of formulaic sequences in language use contributes to faster articulation rate, while the processing advantage of formulaic sequences helps reduce the number of silent pauses when the processing load is large. Findings of this study suggest that the presence of formulaic sequences create a processing advantage for L2 speakers and that EI tasks prompt language comprehension and processing. Findings have important implications for language teaching and assessment, in particular with respect to the teaching of formulaic sequences and the use of EI as a measure of L2 proficiency. Recommendations for future research of formulaic sequences and development of EI tasks are discussed. 

[27] Yan, X. (2020). Unpacking the Relationship Between Formulaic Sequences and Speech Fluency on Elicited Imitation Tasks: Proficiency Level, Sentence Length, and Fluency Dimensions. TESOL Quarterly, 54(2), 460–487. https://doi.org/10.1002/tesq.556  

Previous research has provided ample evidence for the processing advantage of formulaic sequences, leading researchers and teachers to argue for its facilitation of speech fluency. However, few studies have examined how the processing of formulaic sequences interacts with speaker proficiency and task difficulty; even fewer studies have investigated how formulaic sequences facilitate different dimensions of speech fluency. To address these gaps, this study examined the impact of formulaic sequences on speech fluency for both first and second language speakers (N = 269) across proficiency levels on elicited imitation tasks. Participants' speech fluency was measured on both rate and pausing features. Results from linear mixed-effects models reveal that formulaic sequences had a significant effect on the reduction of pauses, but not on speech rate. The effect on pausing was stronger in the processing of long sentences and on intermediate second language speakers. These results suggest that formulaic sequences have differential impacts on the rate and pausing dimensions of speech fluency, and the impacts are further conditioned by speaker proficiency and task difficulty. Findings of this study have both theoretical and pedagogical implications regarding the acquisition of formulaic sequences and the development of speech fluency.