Developing a set of criteria to evaluate the authenticity of EAP materials
The mediating role of materials writers between research and teachers/learners is an area worth investigating because teaching materials are often criticised for their misalignment with findings from relevant research. For example, Tomlinson (2022) condemns the continued popularity of the Presentation, Production and Practice (PPP) approach despite research having proven its ineffectiveness. In a recent post on this same blog, Susanna Schwab makes a similar point.
One of the language learning principles I looked at in my study is authenticity. The traditional, and narrow, definition of authenticity refers to the use of language input, i.e. texts, that were not intentionally created for language teaching purposes (e.g. journal articles). A broader definition of authenticity considers aspects of a task, such as the sequencing of activities and the interaction patterns. From a task-sequence perspective, for example, EAP materials are perceived as more authentic if they require students to engage with the content of a text before looking at its linguistic features, as content is what readers would usually focus on (McGrath, 2002). Classroom interaction patterns are also important. For example, authentic EAP materials should encourage students to work collaboratively and independently from the class tutor, as this is how students are expected to work on their degree programmes (De Chazal, 2014).
So, authenticity is an important aspect of not only the language input but also the task, context and purpose of learning activities (Jordan, 1997; Tomlinson, 2010). Central to this principle is the idea that students should be framed as language users, rather than language learners, who are working on tasks that have meaning-based objectives (Ellis, 2003). It is this broader conceptualisation of authenticity I was keen to investigate in my exploratory study.
The set of materials (nine worksheets) I analysed in my study are my own, so mine is also a reflective account. The materials were those I designed for an academic language and literacy course offered to students on the International Foundation Programme (IFP) at the University of Bristol.
My main research question was: ‘To what extent are the materials I design informed by the principle of authenticity?’. To answer this question, I first sought to determine a set of criteria (Table 1), drawing on the work of Littlejohn (2022). This set of criteria is something I am still working on, and I would welcome your suggestions. (Feel free to comment on this below!)
What is the source of the input?
☐ Previous Learner (exemplars)
☐ Published material (expert)
☐ Published material (adapted)
How long is the input?
☐ Words
☐ Sentences
☐ Paragraphs
☐ Extended discourse
What is the topic of the input?
e.g. night safety in Bristol, transitioning to university …
What is the target genre/sub-genre?
Questionnaires, research articles, introductions, methodology sections, results sections, marking criteria …
What mode is the input in?
☐ Linguistic (written)
☐ Linguistic (spoken)
☐ Visual
☐ Aural
Table 1 - Section on authenticity in the materials writers’ criteria
With these criteria, I was trying to capture what makes teaching materials authentic. So, the section on language input draws on the idea that input should be meaningful, whether this might mean topics likely to interest students and/or genres and modes relevant to their needs (Hedge, 2000). I also included source and length because I thought this type of data could yield some interesting information in relation to the perhaps narrowest conceptualisation of authenticity as the ‘use of texts not intentionally created for language teaching purposes’. The task section in my set of criteria is an attempt to go beyond the narrow definition of authenticity and investigate the broader conceptualisation of authenticity I mentioned earlier in this blog post.
I believe that using these criteria to reflect on the materials I design proved useful in identifying how I could make them more authentic. For a start, the data analysis revealed that the majority of the language input in my materials was sourced from assignments written by previous students. The quality of these samples varied from high-scoring samples to less successful ones. I didn’t make any changes to the original student samples (in this sense, they were indeed authentic). This is a fairly popular approach in EAP, mainly for two reasons. Firstly, authentic texts such as research articles tend to be perceived as too long and difficult for EAP learners, especially pre-undergraduates. Secondly, they encourage students to be teachers, so to speak. For example, when looking at student exemplars in class, learners are encouraged to identify both strengths and areas for development, which involves using their judgement. When the EAP tutor analyses these samples with the learners, what the tutor is doing is essentially demystifying what good academic writing is or, using Smyth & Carless’s (2021) words, they make what would otherwise remain ‘tacit knowledge’ visible to students. The ability to judge a draft is invaluable for university students, who often do not get feedback on their drafts.
This approach, however, has recently come under criticism. The use of student exemplars seems to exacerbate the mismatch between the texts students read and have access to and what they are asked to produce: the so-called ‘genre paradox’ (Walková, 2023). Students usually have access to published papers written by expert writers, and these are what they read for their assignments. Students do not have access to a sufficiently large bank of student samples, and this prevents the development of formal schemata for which frequent exposure is necessary. An additional issue is that students are often not encouraged to engage with the content of student exemplars. This issue also transpired in my study: even when the topics of the exemplars in my materials were authentic in the sense that they were potentially interesting or relevant to students (e.g., night safety in Bristol, transitioning to university, gender and students’ choices in my study), the activities in the worksheets did not ask students to share a personal response or engage with the content. Table 2 provides a sample of the questions used in some of these activities. As you can see, the focus is on reading strategies and textual features of the text. The fact that these materials do not encourage a personal response in learners might impact the extent to which they perceive the content of the lesson as interesting and informative (McGrath, 2002).
A further issue arising from the exclusive use of student exemplars is the limited length of discourse to which students are exposed. When looking at my materials using the criteria in Table 1, I started noticing that the student exemplars were only a few paragraphs long (usually because of concerns over possible intentional or unintentional plagiarism). This is somewhat problematic because linguistic features operating across extended stretches of discourse might go neglected in teaching. For example, the importance of macro-Themes and hyper-Themes in organising a long piece of text (Miller & Pessoa, 2016) might go unnoticed if students are not given access to full samples.
In addition, a focus on short stretches of discourse might fail to give students an overview of what Bruce (2015) calls ‘cognitive genres’. The cognitive genre is the internal organisation of a passage that fulfils a specific purpose (e.g., to argue, explain); in contrast, ‘social genres’ are ‘socially recognised constructs’ (ibid, p.4; e.g., an essay, a research article) and refer to the overall purpose of a text. Students analysing full samples could develop an appreciation of how a text unfolds by noticing how different sections within the same text instantiate different cognitive genres (e.g., explanation, discussion). This kind of analysis would teach students the importance of rhetorical shifting in writing assignments. (You don’t only argue in an argumentative essay; in some parts you might explain or even recount.)
On a more positive note, the data analysis revealed my effort to ensure the authenticity of my materials by drawing on relevant research. In this sense, I was acting as a mediator between the results of research and learners/teachers. For example, the extracts in Table 3, which were taken from the teacher’s notes (bolded for ease of reference) include quotes and terms from Basturkmen’s (2009) move analysis of results sections and Hyland’s (2005) corpus study of stance and engagement in research articles. Although the linguistic analysis should ideally be carried out in-house, this ‘mapping of the field’ using existing secondary linguistic analysis is an important stage in designing materials (Coffin & Donohue, 2014, p. 264).
On reflection, this exploratory study has prompted a revision of my current approach to designing materials. Although the student exemplars analysed in my study can be considered authentic in that their genre and topics are relevant to the majority of IFP students, the accompanying activities could be made more authentic in a number of ways.
First, more authenticity could be achieved by ensuring that the materials adopt an emergent approach (De Chazal, 2014, p. 275) to the study of language, where texts are central to the course, and students are encouraged to first engage with the content and then notice relevant linguistic choices.
The initial focus on content should also include a discussion of purpose and context to acknowledge the EAP genre paradox mentioned above. And this is why in my materials I would like to include more of the kind of contrastive analysis that features in Coffin and Donohue’s (2014) ‘language as a social-semiotic’ (LASS) approach. This type of analysis always starts by asking students three seemingly simple questions (Figure 1) that aim to clarify the relationships between purpose and context, on the one hand, and stages, language and interpersonal relations on the other, while also prompting a reflection on alternative ways of writing.
This focus on alternatives is then explored in Question 3 from the perspective of appropriateness to student assignments (Figure 2). Thus, Coffin and Donohue’s materials seem to go some way towards acknowledging the EAP genre paradox and helping students navigate the complexities of academic writing.
Do your materials feature student exemplars too? What is your solution to the EAP genre paradox? I’d love to hear from fellow materials writers, so please leave your comments below!
References
Bruce, I. (2015). Use of cognitive genres as textual norms in academic English prose: University essays in English literature and sociology. Bulletin VALS-ASLA, n° spécial, tome 2, 161–175.
Coffin, C. & Donohue, J. (2014). A language as social semiotic-based approach to teaching and learning in higher education. Chichester: Wiley.
De Chazal, E. (2014). English for academic purposes. Oxford: OUP.
Ellis, R. (2003). Task-based language learning and teaching. Oxford: OUP.
Hedge, T. (2020). Teaching and learning in the language classroom. Oxford: OUP.
Jordan, R. R. (1997). English for academic purposes: A guide and resource book for teachers. Cambridge: CUP.
Littlejohn, A., (2022). The analysis and evaluation of language teaching materials. In J. Norton, & H. Buchanan, (Eds.). The Routledge handbook of materials development for language teaching, pp.263–276. Oxford: Routledge.
McGrath, I. (2002). Materials evaluation and design for language teaching (2nd ed.). Edinburgh: Edinburgh University Press.
Miller, R. T. & Pessoa, S. (2016). Where’s your thesis statement and what happened to your topic sentences? Identifying organizational challenges in undergraduate student argumentative writing. TESOL Journal, 7(4), 847–873.
Smyth, P., & Carless, D. (2021). Theorising how teachers manage the use of exemplars: Towards mediated learning from exemplars. Assessment & Evaluation in Higher Education, 46(3), 393–406.
Timmis, I. (2022). Theory and practice in materials development. In J. Norton, & H. Buchanan, (Eds.). The Routledge handbook of materials development for language teaching, pp. 30-46. Oxford: Routledge.
Tomlinson, B. (2022). The discipline of materials development. In J. Norton, & H. Buchanan, (Eds.). The Routledge handbook of materials development for language teaching, pp. 3–16. Oxford: Routledge.
Walková, M. (2023). EAP genre paradox [PowerPoint presentation]. BALEAP Conference: Caution! EAP under deconstruction, 19–21 April, Warwick.