Research and analysis

Origins and Evolution of the CASLO Approach in England - Chapter 7: Conclusions

Published 18 November 2024

Applies to England

In this final chapter, we will consider lessons that we have learnt concerning:

• the CASLO approach
• its fitness for purpose, and
• TVET qualification reform more generally

The CASLO approach

At the outset of our research programme, we defined the CASLO approach in terms of qualifications with units that shared 3 core characteristics:

1. unit content is specified via learning outcomes
2. the unit standard is specified via assessment criteria for each learning outcome
3. to pass the unit, a learner must acquire all of the specified learning outcomes, which we refer to as the mastery requirement

This was a very pragmatic definition, based simply on our observation that very many regulated TVET qualifications shared these 3 core characteristics, even when they shared little else. We now know much more about the origins and evolution of the approach, and the following sections set out some of our key insights.

How to understand the approach

We begin by reflecting on how best to understand the approach, including:

• the significance of diversity within the CASLO family
• the historical contingency of the approach
• the challenge of characterising the approach
• the need to locate our understanding within a broader theory of qualifications

Diversity

In the report of our first investigation in this area, report 3, we attempted to characterise the ‘archetypal’ CASLO qualification (beyond its 3 core characteristics). We concluded that qualifications of this sort tended to be designed with flexibility in mind, to accommodate learners studying under different circumstances, within different locations, at different times, and so on. And we noted that their heavy assessment load tended to dispose them toward a phased, or continuous, assessment model.^{[footnote 1]} But it is fair to say that the very idea of an ‘archetypal’ CASLO qualification was tricky to articulate, given the wide variety of qualifications within this very broad family.

A more interesting question, perhaps, was why the 3 core characteristics became so prevalent across such a widely divergent landscape. One part of the answer to this question was quite straightforward: the features that comprise the CASLO approach were specified as accreditation criteria for both the NVQ framework and the QCF. However, another part of the answer was more subtle, suggesting that these frameworks were pushing at an open door. After all, as early as the 1970s, much of the sector had bought into the importance of outcomes and mastery when designing vocational and technical qualifications. As such, the 1980s NVQ model was continuous with the 1970s TEC and BEC models, just as the 1990s BTEC model was continuous with the 1980s NVQ model.

Significantly, although both 1980s NVQs and 1990s BTECs fully adopted the CASLO approach, it seems appropriate to locate these models at opposite ends of a continuum. At one end was the original NVQ model, which was neutral to teaching and learning approaches, on the assumption that a qualification ought to be achievable in a variety of different ways. At the other end was the original BTEC model, which was committed to a particular approach to teaching and learning, on the assumption that it was optimal for its targeted cohort.

The prevalence of distinct traditions of this sort makes it hard to learn lessons about the CASLO approach, per se, from historical analysis alone. For instance, where the approach came to be associated with progressive, student-centred teaching and learning – which was true for BTECs, GNVQs, and many other qualifications – it is hard to distinguish strengths and weaknesses associated with the CASLO approach from strengths and weaknesses attributable to broader aspects of their philosophy, design, development, and delivery.

The most important insight, though, is the simple fact that there is a lot of diversity within the CASLO qualification family. Different manifestations of the CASLO approach – including the NVQ model, the GNVQ model, the BTEC model, and the QCF model – are best understood as distinct nodes within a broad network of approaches circumscribed by the general CASLO definition. It is also important to appreciate that the CASLO approach itself is best understood as just one node within a far broader network of qualification models that emphasise the importance of outcome specification and outcome mastery.

Historically contingent

Bearing in mind for how long the CASLO approach dominated the TVET qualification landscape in England, it would be easy for newcomers to fall into the trap of assuming that it must simply be the ‘occupational way’ of designing qualifications. Yet, even a cursory analysis reveals that this is not really true. Before the introduction of outcome-based models, college-based TVET qualifications were designed classically, and some still are today, both nationally and internationally. The approach makes a lot of sense in certain occupational contexts – where it certificates full occupational competence – but it is not the only game in town.

It is more appropriate to conclude that adopting the CASLO approach was historically contingent, in the sense of representing a particular response to a particular confluence of problems at a particular point in time. As just noted, factors such as qualification accreditation criteria were key to explaining why it became so dominant, and if these factors had not operated then history might have been quite different. England might, for instance, have explored the potential of a far wider variety of outcome-based and mastery-based approaches. We will return to this issue in the final report of our research programme (report 9).

Adoption of the approach was also historically contingent in the sense of riding the wave of a number of North American educational movements, related to objectives, mastery learning, and criterion-referencing. The concept of criterion-referencing became particularly influential in England during the 1980s. For a variety of reasons, it struck a chord with both scholars and politicians alike. Both general education and technical training qualifications changed in response to this criterion-referencing zeitgeist, but in different ways.

The exam boards were instructed to investigate making their exams more criterion-referenced. Having investigated this, they concluded that radical reform was not appropriate for GCSEs or A levels. Consequently, General Qualification reforms of the 1970s and 1980s tended to be more evolutionary than revolutionary. This included developing far more detailed syllabuses, which, in addition to content, included: clear statements of aims, assessment objectives, grade descriptions, specification grids for papers, advice for teachers and students, suggestions on teaching approaches, details of recommended reading, and so on. Consequently, syllabuses that had spanned perhaps 2 pages in 1960 spanned some 20 pages by 1980 (Kingdon & Stobart, 1988).^{[footnote 2]}

Although GQ reforms were extensive, TVET qualification reforms of the 1970s and 1980s were far more revolutionary. Recognising the inadequacy of planning curriculum, pedagogy, and assessment on the basis of meagre syllabus content lists, a new foundation for planning was devised. This involved specifying learning outcomes in detail, meaning that everyone involved in teaching, learning, and assessment could refer to the same, precise statement of the outcomes that students would be expected to acquire (rather than having to rely on a brief, ambiguous statement of what teachers would be expected to cover in their teaching). This distinction was captured in the idea that the new qualifications would be ‘outcome-based’ (learning-focused) rather than ‘input-based’ (teaching-focused).

Inputs ‘versus’ outcomes

In fact, although this distinction appears to tap into something important, consistent with the idea of a genuine revolution in TVET qualification design, it is conceptually problematic, and overstates the differences between outcome-based and classical approaches. One problem with the utility of the distinction is that it was invented to highlight a solution to a recognised design flaw with the classical approach: relying upon meagre syllabus content lists risked low-level examining, which risked low-level teaching and learning. The solution involved stating high-level (and low-level) intended learning outcomes explicitly, as the foundation for planning curriculum, pedagogy, and assessment. In short, the distinction was not invented to characterise 2 conceptually distinct approaches (outcomes versus inputs), it was invented to foreground the solution (outcomes) to a problem associated with the old approach. The ‘outcome’ idea was meaningful and useful, but not so much the ‘input’ idea.

A related problem with the utility of the distinction is that the new approach was intended to correct the old approach, not to oppose it. Note that the grids that Tyler introduced simply supplemented traditional content lists with information on what students needed to ‘do’ with that content. The outcome-based approaches adopted by both the TEC and the BEC worked in essentially the same way. The BTEC approach supplemented this with insights into effective teaching and learning, which included providing information and guidance on delivery approaches, indicative reading, links to related resources, and so on. In other words, outcome-based approaches were not fundamentally opposed to classical ones. And, despite how the distinction is sometimes portrayed in the literature, outcome-based approaches were not forced, as a matter of principle, to reject anything that might be construed as an ‘input’ to teaching – like syllabus content, learning programmes, and suchlike.

The passage of time has rendered the ‘input versus outcome’ distinction even more problematic because, as we have just seen, the classical approach evolved too. The most significant development for classical qualifications in England was the inclusion of assessment objectives within all GCSE and A level specifications. They did not function in exactly the same way as learning outcomes in CASLO qualifications, but they clearly helped to elaborate the nature of the outcomes that students were expected to achieve while studying for a qualification.

We therefore conclude that the ‘input versus outcome’ distinction is neither very meaningful nor very useful nowadays. We recommend that it should no longer be used. This is not to reject the concept of an outcome-based qualification, which is specified principally in terms of learning outcomes, which are used directly for planning curriculum, pedagogy, and assessment. But there does need to be greater clarity and precision in articulating what an outcome-based approach might stand in contrast to (as the term ‘input-based’ fails to articulate this). This position behoves us to revisit our pragmatic distinction between the ‘classical approach’ and the ‘CASLO approach’ to qualification design.

If the CASLO approach is best understood as just one node within a broader network of outcome-based and mastery-based models – and if the classical approach is capable of incorporating features normally associated with outcome-based models – then is the ‘classical versus CASLO’ distinction still tenable, even if simply in relation to regulated qualifications in England? Well, it is certainly not the case that all regulated qualifications fall neatly into one of these 2 categories, depending on whether their designers adopted the CASLO approach or the classical one. Indeed, there are numerous hybrid qualifications nowadays, which incorporate both classical units (typically assessed externally, via mark-based exams) and CASLO ones too (typically assessed by centres, on an outcome-by-outcome basis).

Hybridisation became particularly significant during the mid-2010s, in the wake of Department for Education design requirements for any qualification that was to contribute to performance table calculations. These rules effectively proscribed adopting the CASLO approach for a certain proportion of each qualification. A number of qualifications that had previously been designed entirely according to the CASLO approach, including many BTECs, were reformed to incorporate classical units. On the one hand, hybridisation can be seen as a rational response to external pressures. On the other hand, it can be seen as an uncomfortable melding of quite different educational philosophies. The idea of hybridisation certainly begs questions that merit further research and analysis.

Although not all regulated qualifications in England can straightforwardly be categorised as embodying either the CASLO approach or the classical approach, we think that there are still enough qualifications that are far more like one of the 2 approaches for the distinction still to have meaning and utility.

A theory of qualifications

The classical approach was heavily influenced by a pragmatic tradition of examining for university matriculation, which became established in England during the second half of the 19th century. This tradition had a reputation for shrouding its methods, principles, and machinery in a cloak of mystery and secrecy (Wallis, 1927).

Well into the second half of the 20th century, General Certificate of Education exams continued to have an aura of mystery and secrecy about them – even in terms of their syllabuses – to the extent that Pearce felt a need to theorise the “information structure” of public exams, identifying 5 sources of insight potentially available to ordinary teachers (Pearce, 1972, page 28):

1. syllabuses (which were generally implicit, meaning that their interpretation relied heavily upon the second source)
2. professional expertise (that comes from being a subject specialist within the same community of practice as the syllabus writer)
3. past exam papers (which provided clues concerning how the chief examiner might sample from the syllabus)
4. student results (which indicated the adequacy of their preparation)
5. chief examiner reports, where available (which provided more or less detailed accounts concerning the strengths and weaknesses of student performances)

Pearce noted that syllabus implicitness presented a particular challenge to novice teachers, especially arts teachers, whose syllabuses were the least explicit.

But how explicit should a qualification syllabus (specification) be? This is surely a legitimate topic for a theory of qualifications to engage with. It is not a straightforward question to answer though. In England, this question became entangled within long-standing ideological debates related to control of the curriculum. From the outset, there was a tension between what universities wanted students to be able to demonstrate for matriculation purposes and what individual schools wanted to teach their students. This led to a proliferation of syllabuses from which schools were able to choose. Tension of this sort also discouraged exam boards from specifying syllabuses too explicitly, on the assumption that looser specification would enable teachers to follow different routes through any particular syllabus.

During the second half of the 20th century, government became increasingly involved in these debates between universities and schools. This pressure led to a reduction in the number of GCE syllabuses on offer, in an attempt to secure greater confidence in the comparability of standards (Tattersall, 2007). This also led to the development of A level common cores and GCSE national criteria – which specified the content that would need to be included within all syllabuses in a particular subject area at each level – resulting in far tighter specification of syllabus content during the 1980s and 1990s. Their development also opened the door to greater central control over the content of exam board syllabuses.

In response to this pragmatic tradition – and in the midst of these ideological battles – the TEC, the BEC, and the NCVQ decided that their qualifications needed to be specified far more tightly than qualifications had previously been, drawing inspiration from outcome-based approaches that had been associated with North American curriculum theory. They encountered a certain amount of resistance to the idea of tighter specification, particularly from those who believed that government should not exert so much influence over the curriculum, and that control should ultimately reside locally. However, it seems fair to conclude that most of the resistance to outcome-based approaches in England, particularly within the academic literature, related to exactly how these far tighter specifications were articulated. Of particular concern was the risk that unitised specification of learning outcomes would detract from the idea of an integrated, overarching competence.

So, what does the theory that underpins qualification policy and practice in England have to say of relevance to this debate? Apparently, not a great deal. There seems to be a surprising lack of scholarship devoted to the aims and functions of qualification syllabuses (now known as specifications). In fact, there are good reasons to question whether there even exists a body of work that might legitimately be described as a coherent underpinning theory for this area of policy and practice. Again, we will return to this issue in our final report (report 9).

Why the approach was introduced

Reading some of the more extreme critiques of the first CASLO qualification of national prominence, the NVQ, we might struggle to understand why on earth such a radical reform had ever been contemplated in the first place. Yet, we should not underestimate the problems that outcome-based qualifications were introduced to help solve. It was not just that qualifications for off-the-job education and training were unsatisfactory, provision for on-the-job training was unsatisfactory too. Reform was certainly due, if not long overdue.

Serious problems

It is sometimes said that the NVQ model (and therefore the CASLO approach) evolved out of a need to certificate training schemes for young people who might otherwise have remained unemployed and become increasingly unemployable. The implication is that many of the problems associated with NVQs can be attributed to tailoring what was supposed to be an all-compassing qualification framework to the needs of a relatively small cohort of learners. There is certainly some truth to this claim. Indeed, exactly the same criticism could be made in respect of the Qualifications and Credit Framework, which was also problematically tailored to the needs of a relatively small cohort of learners, in that instance, returning adult learners. As far as the TVET landscape is concerned, one size never fits all, so the key to system rationalisation must be to seek an optimal balance between generic and bespoke.

In fact, the features that comprised the NVQ model, including the CASLO approach, were not introduced purely to solve the problem of YTS certification. They were intended to address multiple, serious, long-standing, widely-recognised issues within the TVET landscape of the 1960s and 1970s. These included problems related to:

• on-the-job work-based training – that was not well specified, that was largely uncertificated, and that was highly variable in quality
• off-the-job college-based education and training – that was skewed toward book knowledge, that sometimes had limited relevance to employment, that was criticised for failing to target the high-level competencies that employers really needed, and that was plagued by wastage, retardation, and failure
• the apprenticeship system generally – that was rapidly falling out of favour with employers
• the many young people and adults who received no education or training beyond school or college

The outcome-based approach to qualification design proved to be particularly useful in relation to those aspects of occupational competence that were not directly associated with the kind of book knowledge that was often the focus in college. Within a well-established academic discipline, it is possible to imagine how a meagre syllabus content list might (just about) be capable of providing sufficient conceptual scaffolding to enable a subject specialist with considerable experience and expertise to create an appropriate programme of learning. Yet, this was not the situation that the Industrial Training Boards faced during the 1960s. The idea of writing specifications for on-the-job training along the same lines as a traditional book knowledge syllabus would have made no sense. So, they developed specifications defined in terms of outcomes, albeit often very narrowly defined ones, concerning the tasks that an employee would need to perform (task analysis).

The NCVQ believed that it could generalise this approach to embrace outcomes arising from both on-the-job training and off-the-job education and training. This would require a broader specification of the occupational role that an employee would need to perform. Learning outcomes were defined in terms of what competent performance of those roles looked like (functional analysis).

This approach left implicit that which a traditional book knowledge syllabus would traditionally have foregrounded, that is, the knowledge and understanding that underpins competent performance. In retrospect, it seems fair to conclude that this was a serious mistake, as it appeared to signal that book knowledge was unimportant for completing an NVQ, and this mired the implementation process in controversy for years. NVQs might have been received more positively if they had adopted a less radical proficiency model, perhaps more akin to those that had been developed by the TEC and the BEC, which were also intended to help counter many of the serious problems identified above.

Although the CASLO approach does not necessitate centre-based assessment, it naturally gravitates towards it, given its emphasis on mastery learning and certification. Centre-based assessment became part of the solution to two of the major problems that affected existing qualifications, which typically relied heavily upon terminal external exams. First, it helped to address the problem of wastage, retardation, and failure, by breaking down the overarching proficiency model into more manageable elements of competence. The requirement that all learners would have to achieve each element of competence established more clearly the need for progression to be carefully monitored, to reveal any emerging gaps in learning for immediate intervention. Second, centre-based assessment helped to address the problem of exams not targeting the higher-level competencies that employers really needed. This involved specifying those outcomes more explicitly and encouraging teachers to adopt alternative assessment formats whenever necessary to assess them.

Variety of purposes

Reasons for adopting the CASLO approach were typically not made explicit independently of reasons for adopting broader features of each of the qualification models considered in preceding sections (NVQs, GNVQs, BTECs, and so on). Consequently, we made a particular effort to unpack the multiplicity of goals that qualification designers appeared to want to achieve by adopting the CASLO approach. We identified 3 distinct perspectives on qualification goals:

1. the certification perspective – to improve the technical quality of assessment (validity)
2. the educational perspective – to improve teaching, learning, uptake, completion, and so on
3. the sociopolitical perspective – to improve the structure of the TVET system

One of the more explicitly stated goals that led to the CASLO approach being incorporated within NVQs was to safeguard the technical quality of NVQ assessment (particularly via domain alignment). Yet, ironically, one of the most serious criticisms of the CASLO approach was that it threatened technical quality. For instance, it was argued that because written criteria cannot possibly express standards with sufficient precision to make perfectly consistent judgements, assessors would be liable to judge students according to different standards, which evidence revealed to be the case (Wolf, 1995; 2011). If so, then was it always naïve to assume that the CASLO approach ought to have a positive impact on the technical quality of assessment? There are 2 points to bear in mind here.

First, validity requires an overarching integrative judgement concerning all of the empirical evidence and logical analysis that bears upon claims concerning the accuracy of interpretations arising from qualification results. Because there is no such thing as perfect assessment, we always end up making design trade-offs that enhance certain aspects of technical quality while accepting a hit on others. This is true whatever qualification model we might be considering.

Advocates of the CASLO approach were keen to ensure that assessors assessed all of the right things (to ensure construct representation) and none of the wrong things (to avoid construct-irrelevant variance). They tackled these 2 principal threats to validity by modelling the target proficiency at the heart of each qualification in terms of learning outcomes and assessment criteria. In doing so, they aimed to specify the qualification construct both comprehensively and authentically. Unfortunately, there is often a correlation between authenticity and judgemental inconsistency in assessment contexts. Pragmatically, a proponent of the CASLO approach would hope that they had engineered a situation in which the increment in validity arising from improved construct representation (in particular) would outweigh any decrement in validity arising from judgemental inconsistency. Empirical evidence would be necessary to substantiate this hypothesis, of course. But, the point is simply that it is possible, in theory, to construct a plausible validity argument in favour of CASLO approach, which trades off different sources of validity and invalidity.

Second, the argument in favour of adopting the CASLO approach needs to be understood in terms of attempting to improve qualification validity, and not in terms of attempting to achieve perfect validity. With its plethora of learning outcomes and assessment criteria, the approach provides a great deal of scaffolding for assessment judgements. The less experienced the assessor, the more important scaffolding of this sort becomes. NVQs, in particular, were intended to support on-the-job training, where assessment expertise could not necessarily be guaranteed. The need for scaffolding would presumably have been high in this context. In short, the CASLO approach was never intended to turn workplace assessors into infallible judges, but it was certainly intended to enhance the validity of qualifications that relied heavily on their judgements.

Although improving validity was one of the more explicitly stated goals that led designers to adopt the CASLO approach, many other goals, particularly educational ones, were far less clearly articulated. It is unclear why, especially as the goals that we associated with the educational perspective were obviously extremely important.

We identified 4 distinct educational goals:

1. domain alignment – to align curriculum, pedagogy, and assessment as closely as possible with the intended domain of learning (and therefore also with each other)
2. domain mastery – to ensure that all students achieve a satisfactory level of attainment across the full domain of learning
3. qualification efficiency – to make the process of becoming qualified as efficient as possible
4. domain personalisation – to enable the domain of learning to be tailored to the personal situation, interests, or needs of learners (or customised to meet the needs of local employers) ^{[footnote 3]}

Note that the domain alignment goal is simultaneously a certification perspective goal (to improve the quality of assessment) and an educational perspective goal (to improve the effectiveness of teaching and learning).

As will have become clear throughout this report, adopting the CASLO approach did not always result in goals of this sort being achieved. More work is required to understand both the circumstances that facilitate their achievement and the circumstances that frustrate them. More fundamentally, though, we need to consider whether these goals are still as attractive today as they might have been in previous decades. It seems likely that some will be, and it seems likely that some will not. Once again, this is an issue to which we will return in our final report (report 9).

How the approach was received

Because the CASLO approach was introduced via qualifications that were innovative and idiosyncratic in many different ways, it is hard to distinguish between observations that relate directly to the approach and observations that relate to broader aspects of their philosophy, design, and delivery. This is particularly relevant in relation to how the CASLO approach was received.

Perhaps the most important point to note is that the CASLO approach was largely imposed on the TVET qualification landscape by the extended machinery of government, via bodies like the TEC, the BEC, and the NCVQ (in association with the bodies that controlled qualification funding decisions). Furthermore, it was introduced as one component of a network of policies designed to rationalise the landscape, which involved challenging extant power structures and provision. The context was therefore necessarily one of conflict.

While reactions to the new TEC and BEC awards were mixed, reactions to the NVQ model tended to be more extreme, and frequently more negative. The NCVQ was generally perceived to be imposing change rather than facilitating it. This imposition was received differently in different quarters.

To the extent that the awarding organisations had been the agents of qualification ‘proliferation’ in the past, they were part of the problem that the new NVQ framework was supposed to solve. Ultimately, they worked closely with the NCVQ to establish their new roles in the landscape, as did the relatively new BTEC validating body. Although relations were sometimes extremely strained – particularly between the NCVQ and the BTEC – this conflict should not be overstated in relation to the CASLO approach. The principal awarding and validating bodies had largely embraced the shift towards outcome-based qualifications significantly before the NVQ framework was rolled out. This prefigured a wider embrace of criterion-referencing during the 1980s, which attracted scholars and policy makers alike.

NVQs received a mixed reaction from colleges, training providers, and employers. Funding incentivised their uptake, and many stakeholders bought into the system on that basis. There were plenty of employers who did not buy into the system, however, raising questions concerning the extent to which it was really employer-led. Funding incentives also drove GNVQ uptake, although there was a genuine sense that GNVQs were serving an important new purpose for full-time students in schools and colleges, for whom neither general education nor technical training routes seemed appropriate. Serious implementation challenges affected how NVQs and GNVQs were received. The CASLO approach received a better reception as the high-level design template for other awards, including OCN and BTEC awards.

Many education scholars reacted negatively to the introduction of NVQs and GNVQs. Again, these were innovative and idiosyncratic qualifications, and the CASLO approach was not always the principal bone of contention. But, it was definitely part of the critique. Debate concerning the introduction of outcome-based and mastery-based approaches was sometimes so heated that it became characterised as a battle ground.

Protagonists included government-sponsored agencies, especially the NCVQ, plus a contingent of education scholars who saw value in the new models. Although NCVQ publications tended to stick to procedural details, the NCVQ pursued an active research agenda, and collaborated with supportive academics in the production of scholarly outputs. In addition to Gilbert Jessup’s own book on ‘NVQs and the Emerging Model of Education and Training’ (Jessup, 1991), this also included collections such as:

• ‘Competency Based Education and Training’ (Burke, 1989)
• the ‘Competence & Assessment’ quarterly journal, including occasional compendia such as ‘Competence & Assessment Compendium No.2’ (ED, 1992)

Beyond the relatively small group of academics who defended the new outcome-based approach, which included John Burke at the University of Sussex, a slightly larger group of academics was constructively critical. This included scholars like Alison Wolf, Michael Eraut, and Phil Hodkinson. However, a considerably larger contingent of scholars was heavily opposed, including Kenneth Marshall, Alan Smithers, and Terry Hyland. Although the NCVQ actively engaged with critical voices (Tim Oates, personal communication), these debates rarely made it into print. Indeed, it would inevitably have been hard for the NCVQ to commit sufficient time to engage with this rapidly expanding literature, while simultaneously rolling out the NVQ and GNVQ models and responding to implementation challenges.

The hostility of the radical NVQ-GNVQ critique has been commented upon in the literature (see, for example, Bates, 1995; Ecclestone, 1997; Hargraves, 2000). No doubt passions were raised by factors beyond the underlying conceptual issues at stake, for instance, heavy encroachment by government into matters that would previously have been negotiated between awarding organisations, education providers, and professional bodies.^{[footnote 4]} The fact that the NCVQ was an agency of the Employment Department, with limited involvement from the Education Department, might well have aggravated concerns. But it is also important to remember that outcome-based approaches were introduced in England at a time when debates over control of the curriculum would have been at their height. In this context, the very idea of prespecifying educational outcomes would have raised the hackles of anyone in favour of teachers retaining professional autonomy over such matters.

In this context, we noted that the NVQ literature includes strands that are not simply extreme, but problematically so. This includes the claim that the NVQ model – and outcome-based approaches more generally – are inherently behaviourist and therefore fundamentally flawed and unworkable. This claim has been repeated so many times in the literature that it has effectively become a matter of TVET dogma. We need to take this seriously, of course. If it is literally true that outcome-based approaches are fundamentally flawed – as a consequence of fundamental flaws within behaviourism itself – then this behoves the regulator to ban them from the TVET qualification landscape. Period.

Having considered the most radical strands of this academic critique in some detail, we did not reach this conclusion. For instance, on the particular issue of the supposed behaviourist roots of outcome-based approaches, we concluded that certain of the most radical claims were simply without foundation, including the idea that the NCVQ based the NVQ model on behaviourist learning theory. Other criticisms we judged to be unduly extreme, based upon a misleading account of the development of outcome-based approaches in North America, and to some extent on a misleading account of the nature of the NVQ model. We exit this debate with lingering concerns over the legacy of the radical critique.

Its trajectory

Part of the empirical argument against the claim that outcome-based approaches are fundamentally flawed – and therefore inherently unworkable – relates to the trajectory of the CASLO approach in England. It is certainly fair to say that the CASLO approach was imposed on the TVET qualification landscape by the extended machinery of government, for instance, via accreditation criteria for the NVQ and QCF frameworks. As such, the CASLO approach did not simply ‘evolve’ in a survival of the fittest sense. Indeed, its survival was often secured by funding levers, which incentivised uptake of CASLO qualifications.

However, it is not fair to say that the origins and evolution of the approach can straightforwardly be explained in terms of central enforcement. The NCVQ had limited statutory powers, and the levers available to the government, including funding levers, could have been pulled more forcefully than they actually were. The truth no doubt lies somewhere between the extremes of central enforcement and natural selection.

Although the NVQ model remained controversial until its official demise, it never abandoned the CASLO approach. Furthermore, the fact that NVQs survived for a long time suggests that the model must have got something right (or, at least, not entirely wrong) and, despite sustained criticism, many employers remained reasonably content with its assessment approach (West, 2004).^{[footnote 5]} While the post-2010 official policy reviews, including the Wolf report, raised important questions related to qualification standards, it is also the case that NVQs enabled many learners who might never have completed a qualification to obtain one. This stood in stark contrast to many TVET qualification suites that preceded NVQs, which had been plagued by drop out and failure. NVQs also provided a viable solution to the problem of certificating off-the-job training, which had not been adequately addressed prior to their introduction.

The GNVQ story is more complicated as far as the CASLO approach is concerned. Like NVQs, GNVQs were plagued by implementation challenges, such as assessment overload. Unlike NVQs, however, GNVQs were gradually redesigned to move away from the CASLO approach. The final iteration of the GNVQ model was the AVCE, which could no longer be described as a CASLO qualification. AVCEs were criticised as being neither seriously vocational, nor consistently advanced, yet their assessment regime still remained excessively complex, bureaucratic, and hard to understand. It was not until almost all of the remnants of the CASLO approach had been eliminated from the model that the qualification ultimately failed.

As GNVQs failed, so BTEC Nationals began to thrive. The BTEC family inherited new cohorts of learners from GNVQ schools that would not previously have considered a BTEC route. The BTEC model that thrived during the 2000s embodied the CASLO approach in full, in contrast to earlier models which had adopted somewhat different outcome-based approaches.

Into the 2010s, the CASLO approach was seen as key to the success of the QCF, so it was established as a formal design requirement for all accredited units and qualifications. By the mid-2010s, it had become clear that the CASLO approach was now the high-level design template for the vast majority of regulated TVET qualifications in England.

Central control

The origins and evolution of the CASLO approach are inseparable from the story of increasing central control over the TVET landscape in England. Back in the 1950s and 1960s, government actively influenced these systems, for example, through membership of the Joint Committees that oversaw the original National and Higher National awards. Yet, it was primarily the awarding, examining, and accrediting bodies that oversaw and co-ordinated these systems. However, precisely because there were so many bodies with responsibilities of this sort, there was no single national system, and the systems that coexisted were not co-ordinated. From the 1970s, government increasingly assumed control of TVET qualification systems in England, in an attempt to rationalise and simplify provision. It operated through a succession of independent non-departmental public bodies, which included the:

• Technology Education Council (1973 to 1983)
• Business Education Council (1974 to 1983)
• Business and Technician (Technology) Education Council (1983 to 1993)
• National Council for Vocational Qualifications (1986 to 1997)
• Qualifications and Curriculum Authority (1997 to 2008)
• Ofqual (2008 to present)

These organisations operated alongside a succession of bodies with related responsibilities for co-ordinating training and training standards at a national level, as well as alongside the existing awarding, examining, and accrediting bodies. Government departments also exerted control via qualification funding rules and, more recently, via school and college performance table rules.

These non-departmental public bodies influenced the structure of qualification systems in England, particularly through the design of qualification frameworks. However, they also influenced the design of the qualifications that populated those frameworks, through qualification (and unit) accreditation criteria. Both NVQ and QCF accreditation criteria established the CASLO approach as the national approach to designing TVET qualifications.

As the approach was rolled out nationally, through NVQs and GNVQs in particular, many problems were encountered. During the mid-1990s, a number of policy reviews were commissioned in response (including the Beaumont review, the Capey review, and the Dearing review). While acknowledging the scale of these implementation challenges, these reviews largely supported continued adoption of the CASLO approach, despite high-profile criticism of the model itself. A minor exception to this was the Dearing review, which argued that the concept of mastery was appropriate for NVQs but not for GNVQs (the Capey review expressed similar, but more nuanced, reservations concerning the mastery requirement for GNVQs, despite strongly supporting continued adoption of an outcome-based approach).

The CASLO approach continued as the national approach to designing qualifications for the technical training route, located at the heart of the NVQ model. This contrasted with the national approach to designing qualifications for the general education route, which continued to be the classical approach. Interestingly, though, the national approach to designing qualifications for the applied education (middle) route did change. The final GNVQ model (the AVCE) largely rejected the CASLO approach. And the national middle route qualifications that followed in its wake also adopted a classical approach (Applied A levels, the Diploma qualification, T Level Technical Qualifications). At the same time, however, the BTEC, which was no longer a non-departmental public body – meaning that BTECs were no longer a ‘national’ qualification as such – fully embraced the CASLO approach. So, the middle route, from the 2000s onwards, included qualifications based on both approaches.

Somewhat ironically, soon after the system had formally embraced the CASLO approach as the high-level design template for all QCF qualifications, it became a matter of concern within a number official policy reviews (from the Wolf review to the Sainsbury review). None of these post-2010 reviews focused primarily on the approach, but they all raised CASLO-related concerns that were judged to be sufficiently important to be acted upon by government and its agencies in various ways.

The Department for Education responded to concerns raised in the Wolf report by specifying design rules for any qualification that was to be counted within a performance table calculation (from 2016 onwards). These included requirements for a certain proportion of external assessment and for the inclusion of synoptic assessment. This effectively ruled out the CASLO approach for a certain proportion of each performance table qualification, and led to some qualifications adopting a hybrid approach, with both CASLO units and classical ones too.

The approach to certificating apprenticeships changed radically in response to the Richard review. This included rejecting extended, or continuous, assessment in favour of a terminal independent End-Point Assessment model. This represented a significant shift away from the CASLO approach – as it had been operationalised within the NVQ model – albeit without rejecting the articulation of learning outcomes or the idea of mastery.

In the wake of concerns expressed by the Wolf report and the Whitehead report, Ofqual reconsidered its approach to regulating TVET qualifications under the QCF. In 2015, it withdrew these regulations, subsequently regulating the vast majority of VTQs solely through its General Conditions of Recognition. As these conditions applied to all regulated qualifications, including GCSEs and A levels, they made no reference to the CASLO approach. From 2015, Ofqual no longer required any regulated qualification to adopt the CASLO approach (which remains true today).

Given this account, it is fair to say that the CASLO approach has fallen out of favour with policy makers in recent years. It is officially proscribed as a design template for certain qualification types. And for many other qualifications it is no longer required. Having said that, there are still many regulated CASLO qualifications in England, which is why we described the state of the CASLO approach as ‘down but not out’.

Fitness for purpose

We now turn to the issue of what lessons we might be able to learn from preceding chapters concerning the fitness for purpose of CASLO qualifications and outcome-based qualifications more generally. Unfortunately, it is not straightforward to reach conclusions of this sort. First, and most obviously, this project was not designed as an evaluation of the CASLO approach, in relation to the goals that tend to drive it. Indeed, part of the project rationale was to develop clearer insights into the nature of these goals, to provide a more solid foundation for subsequent evaluative work.^{[footnote 6]}

Second, as we have just seen, it is hard to distinguish strengths and weaknesses associated with the CASLO approach from strengths and weaknesses attributable to broader aspects of the philosophy, design, development, and delivery of the qualifications that we have studied. NVQs, GNVQs, and BTECs all incorporated innovative and idiosyncratic features in addition to the 3 core CASLO characteristics (outcomes, criteria, and mastery). Furthermore, the implementation failures that bedevilled NVQs and GNVQs make it hard to judge the viability of their underlying models on the basis of evidence from rollout alone.

Caveats aside, the preceding sections have clearly illustrated a variety of threats and tensions, and raised numerous questions related to the CASLO approach. After highlighting issues of this sort, we will consider what we might be able to conclude from this strand of our research programme concerning fitness for purpose.

Note that the following sections make repeated reference to issues raised by Alison Wolf. Wolf is one of the most significant figures in the field, having produced one of the most important scholarly texts (Wolf, 1995) and one of the most influential policy reviews (Wolf, 2011). Her reflections on fitness for purpose in relation to NVQs, GNVQs, and QCF qualifications – which have strongly influenced policy making in recent years – therefore provide an important point of reference. They are complemented by reflections from other scholars and policy reviewers.

Threats

We can classify threats that have been linked to the CASLO approach into one of 3 broad categories: threats to viability, threats to the quality of assessment, and threats to the quality of teaching and learning. Under each of these headings, the following subsections capture a range of issues that affected NVQs, GNVQs, BTECs, and other CASLO qualifications, and that help to explain why the approach began to fall out of favour with policy makers during the 2010s.

Threats to viability

Perhaps the most predictable threat associated with the CASLO approach derives from its requirement for exhaustive (all-encompassing) assessment. Because CASLO qualifications are intended to certify domain mastery, they require evidence that each and every specified learning outcome has been achieved to required standards. This tends to make the assessment process burdensome. Summarising experiences during the early 1990s, as NVQs and GNVQs were being rolled out, Wolf observed that every observer of the system – whether official body or independent researcher – agreed on the “sheer quantity of assessment” that teachers faced (Wolf, 1998, page 433).

For instance, evidence arising from her own 1994 investigation into GNVQ grading concluded that the average amount of time spent on assessment-related activities by each GNVQ teacher was 13 hours per week, with wide differences between teachers largely explicable in terms of class size. To the extent that the CASLO approach blurs distinctions between assessment, teaching, and learning – with its heavy emphasis on formative assessment – this would not necessarily have been time spent unproductively. But the administrative load associated with portfolio assessment on this scale is undoubtedly challenging for both students and their teachers, trainers, and assessors. Indeed, Wolf argued that an important factor in explaining high GNVQ non-completion rates was the failure of many students to sustain a steady rate of portfolio completion.

Threats to the quality of assessment

Although the CASLO approach was often introduced to help secure the quality of assessment – as part of the rationale set out in the domain alignment goal – evidence from implementing CASLO qualifications illustrates significant threats too.

Inconsistent judgements

Wolf expressed even more concern over the threat of inconsistent judgements than over the threat of assessment burden. This line of criticism was developed across much of her research and analysis into NVQs and GNVQs during the 1990s, and it was also emphasised in her 2011 policy review. She claimed that evidence of inconsistent judgements challenged what she considered to be a widespread false belief concerning criterion-referencing:

Correspondingly, it is believed that it can provide – indeed, that its use guarantees – information about a candidate’s competence (skills, knowledge, etc.) that is substantive and specific, and highly reliable.

(Wolf, 1995, page 54)

It is debatable the extent to which protagonists of the approach genuinely believed that criterion-referencing guaranteed highly reliable judgements.^{[footnote 7]} Yet, it is not debatable that NVQ and GNVQ judgements often exhibited limited reliability (see, for example, Eraut, Steadman, Trill, & Parkes, 1996).^{[footnote 8]} Furthermore, it is hard to argue with Wolf’s more general conclusion that written statements alone are insufficient for ensuring judgemental consistency. Wolf proposed that effective assessor networks – active communities of practice – were key to ensuring the consistent application of assessment criteria (Wolf, 1995).

Deficient judgements

A different kind of threat to the quality of assessment was emphasised in the Richard review. This related to the risk that certification failed to capture information related to the integration and organisation of the elements that comprise occupational competence. By design, the CASLO approach deconstructs the overarching competence construct into discrete learning outcomes located within discrete units. If there is significantly more to the ‘whole’ (occupational competence) than might be captured by assessing individual ‘parts’ (discrete learning outcomes) then this presents a threat to assessment quality. This concern was echoed in Ofqual’s 2014 evaluation of the QCF.

Threats to the quality of teaching and learning

Although less prominent during the 1990s, concerns over the potential for backwash impacts on teaching and learning became more prominent during the 2000s, and featured within the post-2010 TVET policy reviews. The need to shift attention from assessment back to curriculum and pedagogy was emphasised by the 2013 CAVTL report.

Superficial learning

The threat of superficial learning is closely related to the threat of deficient assessment judgements, stemming from the same phenomenon of discretely specified learning outcomes. The policy reviews written by Richard, Whitehead, and Sainsbury all expressed concern that the occupational competence acquired while studying for a CASLO qualification might end up being insufficiently integrated or holistic. The decision to specify learning outcomes far more generally, and succinctly, within new apprenticeship standards was a response to this threat of negative backwash impact from detailed CASLO specifications, as was the requirement for synoptic assessment within performance table qualifications.

Insufficient learning

If CASLO qualification implementation results in undue assessment burden, then this can lead to one of 2 possible consequences. The first is that additional time may need to be committed beyond the period already allocated to teaching and learning. The second is that the additional time required for assessment may eat into the allocated period. If so, then there will be less time available for teaching and learning. Undue assessment burden has often been linked to the CASLO approach, although often without unpacking its consequences in any detail. However, certain policy reviewers, including Doug Richard, expressed specific concerns over the threat of reduced teaching and learning time.

Tensions

Threats to assessment quality led to calls for greater rigour in relation to NVQ and GNVQ rollout. Reflecting on conclusions from the Beaumont and Capey reviews, Dearing noted that their: “greatest concerns related to the whole business of assessment, and at the heart of this lies the need to establish assessment that is rigorous” (Dearing, 1996, page 75).

Unfortunately, particularly in relation to NVQ implementation, this resulted in a fundamental tension between the need for qualification rigour and the need to be responsive to the demands of those who were charged with implementing the system, especially when (some but not all) employers valued simplicity and accessibility above rigour. This tension played out in numerous ways over the decades.

For instance, Beaumont was critical of the process of developing standards for NVQs, which he believed had been “marred by complex, jargon ridden language” (Beaumont, 1996, page 13). He argued that standards needed to “be written for employers” and that they “must all be in plain English” (both quotations from Beaumont, 1996, page 5). Yet, while system designers recognised these concerns, some were wary of what they saw as dumbing down the approach:

[The standards] sometimes include jargon – technical shorthand which is understandable only to occupational specialists. This is essential if the standard is to be used properly by the people who design and implement the systems based on the standards. If the standard is informal and colloquial it may be vague and often useless.

(Mansfield & Mitchell, 1996, page 154)

Tension of this sort is hard to reconcile. Assessment is a technical process, and the argument for expressing its requirements technically is persuasive. Yet, if this technical nuance proves to be inaccessible to those with primary responsibility for conducting the assessment, then it is not going to ensure rigour anyhow.

The tension between rigour and responsiveness played out even more starkly in relation to functional analysis itself, which had been introduced and advocated by the NCVQ and the Training Agency during the late-1980s. A decade or so later, when the QCA took over from the NCVQ, it sought views on how effectively the NVQ system was operating, particularly from the perspective of employers. The QCA was keen to make NVQ structures more flexible, to reduce bureaucracy, and to improve ease of use. Under new regulations that were published in 2000, functional analysis – and the detailed prescription that it entailed – was no longer formally required.

A decade or so later, the system for developing National Occupational Standards was reviewed by the UKCES with a view to improving its quality. It put functional analysis firmly back on the table as a formal requirement of its new quality criteria, although its approach was not quite as stringent as the original methodology. Subsequently, the Sainsbury review rejected the very idea of detailed prescription that lay at the heart of functional analysis, and employers were invited to play a more significant role in developing standards.

Questions

After more than 3 decades of implementing the CASLO approach, there still remain numerous technical issues that would benefit from further research and development.

Learning outcomes

Plenty of questions remain to be answered concerning how best to specify learning outcomes for CASLO qualifications. These include how to tailor learning outcomes to the needs of particular qualifications, as well as more general questions related to learning outcome format, content, and grain size.

Format

The most important lesson to learn from NVQ implementation was that it matters greatly how learning outcomes are specified. The NVQ proficiency model was unusual in being specified purely in terms of the achievements that comprise performing an occupational role competently. Although this model made sense, in theory, it lacked credibility for many users. By leaving implicit exactly that which traditional qualifications had foregrounded – namely underpinning knowledge and understanding – it left itself open to widespread misunderstanding. It was caricatured as (and ridiculed for) shunning underpinning knowledge and understanding.

Although there are general lessons to learn concerning the consequences of adopting one or another format for specifying learning outcomes, questions still remain concerning how best to tailor learning outcome formats to the needs of particular qualifications, related to their purposes, cohorts, and contexts. Perhaps the most fundamental outstanding question concerns the relationship between outcomes that are more knowledge-like and outcomes that are more performance-like. Should they be specified in similar formats or quite differently? Should they be specified together or separately? The idea of certificating an integrated overarching competence seems to argue in favour of using essentially the same format within a single qualification. Yet, this was exactly the principle that underpinned the NVQ model, which was heavily criticised. In response to this critique, knowledge-like outcomes ended up being assessed quite independently, within Technical Certificates, which was also not ideal. The issue remains to be resolved.

Content

NVQs were also criticised in terms of the content of their learning outcomes. A major bone of contention concerned whether NVQ outcomes ought to be limited to, or be far broader than, specific occupational roles. By limiting NVQ standards to the parameters of specific occupational roles, the NCVQ was criticised for discouraging broader learning, and this criticism was not fully addressed by adding core skills requirements. It is important to remember, however, that concerns of this sort are not directly related to the CASLO approach, as it would be entirely possible to specify a far broader education and training programme in terms of learning outcomes and assessment criteria.

Related to the issue of outcome format, NVQs were accused of shunning traditional syllabus content, which seemed to be implied by their lack of explicit reference to underpinning knowledge and understanding. As we have discussed in some detail, this was technically untrue of NVQs, and it is more obviously false as a criticism of many other CASLO qualifications.

Indeed, the basic idea of an outcome-based approach – as epitomised in the work of Ralph Tyler – is that it is intended to provide a more detailed specification than would be provided by syllabus content alone, to provide a more comprehensive and authentic foundation for planning curriculum, pedagogy, and assessment. We saw how TEC and BEC awards manifested exactly this idea, transforming traditional syllabus content lists into explicitly stated learning outcomes. We also saw how later BTEC awards adopted a variety of different approaches to specifying both syllabus content and learning outcomes. In other words, the idea that outcome-based qualifications are somehow inherently opposed to the idea of syllabus content is straightforwardly wrong.

However, practical and theoretical questions do remain concerning the relationship between syllabus content and learning outcomes. For instance, are there qualification purposes, cohorts, or contexts for which it is important to dial down the central specification of syllabus content, and to rely instead upon more generic and therefore more widely applicable learning outcomes? This relates to the goal of (minimal) cross-context domain personalisation, which has been linked to the CASLO approach over the years. So, when is this flexibility a good thing, and just as importantly, when might it become problematic?

Finally, to the extent that central specification of syllabus content is deemed to be important for a particular CASLO qualification – as has often been the case – questions remain concerning how best to specify it. For instance, is it best to overlay outcomes directly on content (as per Tyler) thereby explaining what learners are expected to ‘do’ with the content? Or is it best to specify content quite separately?

Grain size

The idea of grain size is intended to capture the amount of detail in which learning outcomes are specified. This highlights another critical tension: the greater the detail in which learning outcomes are specified, the greater the potential for clarity, but also the greater the risk of mis-specification, where the proficiency in question can no longer be represented meaningfully or usefully in the level of detail being attempted.

Identifying the right grain size is one of the most important of all design decisions, if not the most important. Remember that the basic idea of an outcome-based qualification is to provide more detail than a classical qualification. This involves explicating what we want learners to be able to ‘do’ with the syllabus content to which they are exposed. Or, in the words of the authors of the most recent articulation of Bloom’s Taxonomy, this involves making “general and abstract learning goals more specific and concrete” (Anderson & Krathwohl, et al, 2001, page 12). The more clarity we can provide, the better. But the pursuit of clarity forces us to consider when enough is enough. That is, we need to identify the point at which attempting to provide further detail becomes counterproductive, and clarity is lost.

This has always been the trickiest question for advocates of criterion-referencing to answer. Experiences from both England and overseas teach us that this typically involves trial and error. Reflecting on both the international literature and experiences in England, Wolf referred to the threat of “a never-ending spiral of specification” (Wolf, 1995, page 55). From a purely practical perspective, overstepping the mark is counter-productive because it results in specifications that are too burdensome to be useful, and that consequently fall into disuse.

In England, pitching learning outcomes at the right level of detail proved to be challenging right from the outset. The BEC was able to learn from the experience of TEC awards, and pitched its original outcomes at a slightly higher level of generality. Subsequent generations of BTEC awards experimented with differing levels of detail. The NCVQ methodology for specifying standards was supposed to culminate in outcomes that were fairly broad. Yet, in practice, these outcomes were often specified quite narrowly. The addition of range statements also led NVQs down the route of increasingly detailed specification.

Wolf has argued that the goal of “total clarity” means that outcome-based approaches are inherently doomed to overstep the mark, as written specifications can never deliver total clarity no matter how detailed they become (Wolf, 1995, page 54). Yet, as noted earlier, this seems to be an uncharitable reading of these approaches, which are fundamentally pragmatic, and which therefore aim for optimal clarity, not total clarity. The Goldilocks principle seems to be apposite here.

The question of how to specify outcomes at the right level of granularity, or detail, does not have a straightforward answer. Again, it is likely to depend to some extent on the purposes, cohorts, and contexts for which a particular qualification is to be designed. The international literature provides useful insights (for example, Bloom, et al, 1971; Popham, 1994; Mager, 1997; Gronlund & Brookhart, 2008; Cedefop, 2017).

Mastery

We turn now to the question of whether there might be a threshold beyond which it becomes inappropriate to attempt to specify the proficiency model in more detail, for fear of mis-specifying it. Under the CASLO approach, this links directly to the idea of mastery, because the smaller the grain size, the greater the number of parts into which the overarching competence is deconstructed, and therefore the larger the number of elements that need to be mastered independently. This also relates to nature and function of assessment criteria, which also add to the mastery burden.

The BTEC grappled with the grain size challenge when transitioning from the BEC model (Generation 1) to the BTEC model (Generation 2). Instead of nesting lower-level outcomes within higher-level ones, the G2 model retained only the higher-level outcomes for which it also provided a list of indicative content. So, rather than prescribing the exact set of knowledge, skills, and understanding that comprised each higher-level outcome, this content list provided a broad indication of the sort of knowledge, skills, and understanding that would be relevant to achieving it. This meant that the higher-level outcome could be achieved in different ways, facilitating (minimal) cross-context domain personalisation.

This illustrates the more general point that, unless a learning outcome is specified in extreme detail, there are likely to be different ways of achieving it, and sometimes very different ways. It seems reasonable to suggest that, the less detail in which a learning outcome is specified, the greater the number of ways in which it might legitimately be said to have been achieved. One of the fundamental goals of the CASLO approach is to reduce ambiguity by providing greater detail concerning the elements that comprise the targeted proficiency. Yet, the greater the detail provided, the greater the risk that the model mis-specifies it, by attempting to deconstruct elements that cannot meaningfully be split into further parts, or by enforcing the mastery requirement at a level where the amount of detail is too great for the concept of mastery to be meaningful.

Related to this point, Wolf discussed the “inherent variability of the contexts in which competence is tested and displayed” such that a learner might genuinely be able to demonstrate competence in a number of contexts yet not in others (Wolf, 1995, page 68). This suggests that competence is not actually a binary concept, and that even the concept of mastery must have an element of compensation, or perhaps charity, built into it.^{[footnote 9]}

Perhaps, as a general rule-of-thumb, the idea of mastery makes less sense and becomes less useful the smaller the grain size with which outcomes are specified? If so, then any CASLO qualification designer will need to be able to judge the borderline beyond which the benefits of disambiguation (by specifying an outcome in greater detail) are outweighed by the risks of mis-specification (by implying that each and every additional detail is indisputably critical to the outcome). Beyond this borderline, it would no longer be profitable to characterise a higher-level outcome as the sum of its lower-level parts.

The discussion of inherent variability prompts a related question concerning the function of assessment criteria within CASLO qualifications. Should they function definitively (such that satisfying all of the criteria defines having mastered the learning outcome) or should they function indicatively (providing an indication, but not a prescription, of what it means to have mastered the learning outcome)?

QCDA guidance on the QCF insisted that assessment criteria had to function definitively – that is, each and every criterion that was specified for a learning outcome had to be satisfied – which was also how NVQ performance criteria functioned. Perhaps, though, for certain qualifications, it might be more meaningful and useful for assessment criteria to function indicatively? The Eraut report appears to have reached the same conclusion in arguing that criteria “should be aids to judgement rather than rules for judgement” (Eraut, et al, 1996, page 68).

It is worth noting that the use of more holistic (indicative) criteria has sometimes been seen as the best solution to the challenge of awarding higher grades within CASLO qualifications, for instance, within GNVQs and certain BTECs (see also Newton, 2018).

Assessment criteria

Our investigation into the origins and evolution of the CASLO approach raises further questions concerning the nature and function of assessment criteria within CASLO qualifications.

Outcomes versus criteria

Particularly when criteria are presumed to function definitively, there is a risk of blurring the distinction between outcomes and their standards. That is, when having mastered a learning outcome is defined in terms of having satisfied all of its associated criteria, this risks giving the impression that criteria need to be ‘mastered’ in essentially the same way as outcomes, which can lead to treating criteria as though they are little more than mini learning outcomes, as opposed to standards for judging mastery of learning outcomes.

Although, confusingly, assessment criteria do sometimes look like mini outcomes, they need to function quite differently, because their fundamental purpose is to help articulate the difference between having achieved the outcome and not (yet) having achieved it.^{[footnote 10]} Gealy argued that this distinction became increasingly blurred over time as the CASLO approach generalised far beyond NVQs. The Further Education Unit actually defined assessment criteria as mini learning outcomes (FEU, 1995a). The QCDA provided examples of how to specify knowledge units for QCF qualifications, which looked like this:

Respond to customer requests for repairs L34

LO 1     Know the organisation’s housing stock and possible defects which require repair.

AC 1.1  Describe the types of properties which the organisation manages.

AC 1.2  Identify, using the appropriate terminology, the types of faults which can occur in these properties.

The way in which these were written renders them far more like mini outcomes than clearcut criteria, which raises the question of how assessors are supposed to identify the boundary between, say, a satisfactory description and an unsatisfactory one. This challenge becomes even more significant when knowledge and understanding requirements are abstracted from their presumed contexts of application (in the workplace) and delivered as a freestanding qualification like a Technical Certificate.

Bloom’s Taxonomy

A particularly enigmatic question concerns the role of Bloom’s Taxonomy in designing and developing CASLO qualifications. In many ways, Bloomian thinking is fundamental to the CASLO approach, as it is to many outcome-based approaches. This relates to the domain alignment goal, which identifies a need to differentiate explicitly between less complex and more complex forms of engagement with qualification content. Hence the idea of articulating what students are expected to be able to ‘do’ with the content that they are taught – recall it, analyse it, apply it, or suchlike.

In certain CASLO contexts, Bloom’s Taxonomy was used not simply as a tool for characterising learning outcome complexity but also as a tool for establishing comparable complexity across learning outcomes. This was true of the QCF, for instance, which incorporated the idea of an association between learning outcome complexity and command verb complexity, such that: ‘stating’ something related to content X was less complex than ‘explaining’ something related to content X, which was less complex than ‘evaluating’ something related to it. This helped to justify the idea of writing unit standards – assessment criteria – as though they were little more than mini learning outcomes. In other words, being able to ‘explain’ represented a certain level of competence (suitable for assessment criteria at, say, Level 2 of the QCF), while being able to ‘evaluate’ represented a higher level of competence (suitable for assessment criteria at, say, Level 3 of the QCF). Having said that, the use of command verbs to indicate both level-worthiness (across levels) and grade-worthiness (within levels) certainly muddied this water (see also Newton, 2018).

Ultimately, while command verbs like ‘state’ and ‘explain’ and ‘evaluate’ can help to scaffold qualification standards, it is important to recognise that they can do so only very roughly. For instance, as already mentioned, there is nothing in AC1.1 (above) that helps to clarify the boundary between a satisfactory description and an unsatisfactory one. Unless that gap is somehow bridged – whether by professional insight, by formal training, by detailed exemplars, or whatever – standards are likely to be applied inconsistently. Bloom’s Taxonomy has no more than heuristic value, and it should not be taken too literally.

Judgements against criteria

If the foregoing analysis holds water, then writing learning outcomes (and assessment criteria) in too much detail runs a variety of risks, including:

1. increasing the assessment burden (by increasing the number of assessment judgements that need to be made)
2. decreasing the feasibility of the assessment process (by providing assessors with more judgemental scaffolding than they can deal with)
3. increasing the risk of mis-specification (by attempting to explicate the proficiency model at a level of detail that is no longer compatible with the mastery approach)

Weighing these risks against the intended benefit of disambiguation, it might well be appropriate for a qualification designer to err on the side of providing too little detail rather than too much. If so, then this would increase the need for additional mechanisms to underpin the consistent application of assessment criteria – perhaps of the sort recommended by the FEU, including exemplification materials, professional networks, and so on – especially when those criteria are written as mini learning outcomes.

Written testing

Finally, questions remain about the role of written testing within CASLO qualifications. These concern how effectively the CASLO approach can be implemented:

1. using the written test format, or
2. alongside classically designed tests (within hybrid qualifications)

In theory, there is no obstacle to implementing the CASLO approach via the written test format. It simply requires that each learning outcome is amenable to being assessed validly via a written test, and that each learning outcome has, in effect, its own mini test to enable a discrete judgement concerning mastery.

Having said that, where assessment criteria are intended to function definitively, this complicates the process by requiring a mini test for each criterion. So, the challenge is more practical than theoretical: the more learning outcomes (and assessment criteria) a unit has, the greater the burden of testing on all involved. Furthermore, if these tests were required to be taken terminally, at the end of each unit, then the consequences of failing a mini test (and therefore failing the unit) would be high. Yet, at the same time, the likelihood of failure would also be high, given the number of hurdles at which each candidate might fall. Each failed mini test would need to be resat during a subsequent session, which might actually mean having to resit full tests. So, the practical challenges are more far serious than the theoretical ones when attempting to implement the CASLO approach via written test formats, especially when they are required to be taken terminally.

Conversely, in practice, there is no obstacle to combining CASLO and classical units within a single qualification. As long as they report results in the same metric – pass, merit, or distinction, for instance – they can be combined straightforwardly. The more serious challenges to hybridisation are theoretical. For instance, if only certain units apply a mastery certification principle – while others aggregate judgements across multiple (potentially sampled) learning outcomes using a compensatory certification principle – then is that still aligned with a philosophy of mastery learning? And how should we expect qualification users to interpret results from hybrid qualifications? More pragmatically, to the extent that CASLO and classical units are designed, developed, delivered, and quality assured in quite different ways, how does hybridisation affect the financial viability of a qualification, or perhaps the quality of its implementation if resources are spread more thinly?

Lessons

We began this section by noting that it is hard to draw definitive conclusions from our analysis concerning the fitness for purpose of CASLO qualifications. After noting a couple more caveats, we will conclude with just 2 high-level lessons learnt.

Caveats

First, some of the insights that arose from our analysis were linked to, but extend far beyond, the CASLO approach. For instance, the approach played a role in making the principle of mixing and matching units (more of) a reality under the QCF. The fact that this risked undermining the coherence of study programmes – as noted by the Richard review in particular – is an important lesson to learn, but not one that bears directly on the fitness for purpose of the CASLO approach. Mixing and matching units risks undermining the coherence of study programmes regardless of how units might be designed.

Second, certain insights that appear, at first blush, to be specific to the CASLO approach are actually far more general. A major investigation into impacts associated with different modes of assessment in the learning and skills sector illustrates this (Torrance, Colley, Garratt, et al, 2005). On the one hand, the authors concluded that the move towards criterion-referenced assessment and competence-based assessment had increased uptake and improved learner achievement and progression. However, on the other hand, the very transparency that was fundamental to these approaches had also encouraged instrumentalism, thereby reducing the challenge of learning and the validity of qualification results:

We have identified a move from what we characterise as assessment of learning, through the currently popular idea of assessment for learning, to assessment as learning, where assessment procedures and practices may come completely to dominate the learning experience, and ‘criteria compliance’ comes to replace ‘learning’. This is the most significant challenge confronting assessment in the LSS: balancing the explicitness of learning objectives and instructional processes against the validity and worthwhileness of learning outcomes.

(Torrance, Colley, Garratt, et al, 2005, page 2)

Remember that exactly this concern was identified in the CAVTL report, and then in the Whitehead report. In terms of this second caveat, however, the most important point to note from the Torrance report was that they expressed exactly the same concerns regarding learners who were studying AVCE and A level programmes. In other words, criteria-focused coaching and the concomitant risk of criteria compliance was proving to be similarly problematic for post-millennium classically designed qualifications.

Putting caveats of this sort to one side, we believe that our investigation into the origins and evolution of the CASLO approach enables us to reach 2 major conclusions related to fitness for purpose:

1. the CASLO approach cannot be said to be universally fit, nor universally unfit, for purpose
2. it is not easy to render CASLO qualifications fit for purpose, and in some ways they are harder to render fit for purpose than classically designed qualifications

Neither universally fit nor unfit

On the one hand, our analysis has provided insufficient reason to conclude that the CASLO approach, or outcome-based approaches more generally, are fundamentally flawed and therefore universally unfit for purpose.

Particularly in recent years, there has been a tendency for scholars to treat the NVQ model as though it reflected the essence of all outcome-based approaches to qualification design: “a ‘pure’ learning-outcomes approach” (Young & Allais, 2009, page 3), or “a ‘full-blooded’ version of the outcomes-based design philosophy” (Winch, 2020, page 169). The implication is that, by striking at the heart of the NVQ model, we strike at the heart of the outcome-based approach, per se, and therefore at the heart of the CASLO approach too. On the basis of an analysis of this sort, Winch recently declared that: “outcomes-based qualifications are not fit for purpose” (Winch, 2023, page 20).

Our own analysis does not lead to such an extreme conclusion, at least, not as we define the essence of an outcome-based qualification.^{[footnote 11]} More specifically, we are not convinced that it is legitimate to generalise from criticisms of NVQ theory and practice – both of which involved features that were highly idiosyncratic – to outcome-based qualifications more generally. Furthermore, our analysis reveals that there is a positive as well as a negative story to be told concerning the introduction of the NVQ system in England, which is also true in respect of CASLO qualifications more generally, and of outcome-based qualifications more generally than that.

On the other hand, our analysis has provided sufficient reason to conclude that the CASLO approach – and outcome-based approaches more generally – are not universally fit for purpose. For instance, we believe that it was a mistake to assume that all qualifications can straightforwardly be shoehorned into this design template, as those who designed the QCF appear to have assumed. There will be contexts and purposes that make adoption of the CASLO approach more challenging, such as when results are used for purposes that have high stakes for teacher-assessors (although it is fair to say that classically designed qualifications may also be less fit for purpose under such circumstances). There will also be cohorts for whom adoption of the CASLO approach is likely to be less beneficial. For example, the benefits of implementing a mastery learning philosophy might be lower for qualifications that tend only to attract highly motivated and highly capable students.

The decision to adopt the CASLO approach ought to follow from a rigorous analysis of the purposes that a particular qualification (or type of qualification) is intended to serve, given the nature of the cohort for whom it is being designed, and the contexts within which it will need to operate. Sometimes an analysis of this sort will recommend an outcome-based approach, such as the CASLO approach, but not always.

Challenging to ensure fitness for purpose

One part of the explanation for why it is not easy to render CASLO qualifications fit for purpose relates to the fact that the CASLO approach has tended to be used for TVET qualifications, and it is harder to design qualifications to be fit for TVET contexts than for traditional academic ones.

Traditional academic qualifications benefit from their grounding in disciplines that have accumulated wisdom, status, and rigour over decades if not centuries. They are overseen by scholarly communities of practice, which include established teaching and learning communities, and they typically involve large numbers of students and well-qualified staff. The same cannot be said for many TVET qualifications. TVET qualifications also tend to attract students who have been less successful on prior academic qualifications, including students who have straightforwardly been failed by them (in both senses of the word). Finally, whereas many of the outcomes associated with academic qualifications are readily amenable to the written exam format, this is often not the case for TVET qualifications. Alison Wolf put it like this:

However, in drawing a line under the history of NCVQ, it is important to remember what first inspired the ‘competence’ movement and the advocacy of portfolio based techniques. In most vocational contexts, it is virtually impossible to develop timed, paper based, externally marked assessments which carry conviction as valid samples of the skills and competencies concerned. Research evidence (summarised in Wolf, 1995a) confirms that such tests generally have very limited value as predictors of workplace performance. Legitimate concerns with the reliability and manageability of assessments do not, in themselves, help us to develop techniques which also generate valid judgements about capabilities and performance. In Britain, the failures of NVQ and GNVQ policies leave these problems unsolved but no less pressing than before.

(Wolf, 1998, page 442)

Another part of the explanation for why it is not easy to render CASLO qualifications fit for purpose stems from the fact that outcome-based qualifications attempt to provide far more structure than classical ones. This additional structure is intended to help scaffold teaching, learning, and assessment. Yet, it generates risks too.

The present report highlights the difficulty of designing an outcome-based qualification model that is right first time. Significant difficulties were encountered with all of the earliest innovations, including the TEC and the BEC models, the NVQ model, the GNVQ model, and so on. Each of these models was refined and reformed over time.

Of course, all qualification models are likely to be refined and reformed over time. Yet, outcome-based qualifications can be particularly difficult to get right because, by design, they model outcomes and criteria far more explicitly and in far more detail than classical ones. This increases the risk of modelling them inadequately or inappropriately. This includes the risk of specifying outcomes (and criteria) in too much or too little detail. It also includes the risk of pitching standards at a level that is either too demanding or insufficiently demanding. These challenges can be harder for school-based qualifications than for work-based ones.^{[footnote 12]}

These challenges are exacerbated by the fact that outcome-based qualifications are less forgiving of inadequate or inappropriate design decisions. If a classically designed qualification specified too much content, then this would no doubt place students and teachers under unnecessary pressure as they studied for their exams. Yet, because exam standards are ultimately applied post hoc, adverse consequences can be mitigated by setting grade boundaries lower than originally anticipated. In contrast, outcome-based qualifications specify required outcomes and standards in advance, which makes it hard to mitigate adverse consequences arising from having specified too many, or unduly demanding, outcomes.

These challenges emphasise the critical importance of piloting and trialling outcome-based qualifications. The broader the scope of the reform process – whether at the level of a single qualification, or a qualification type, or a qualification framework – the more extensively the outcome-based model will need to be piloted and trialled. These challenges also underline the level of professional insight and expertise that will be required of anyone responsible for designing and developing outcome-based models and qualifications. We will return to this point shortly.

Figure 17. Securing the success of a CASLO qualification

The CASLO template

Perhaps the most important lesson to learn is that it is hard to say anything definitive about the CASLO approach, per se, because the validity of any particular CASLO qualification, as well as the nature of its educational and societal impacts, will depend upon how the basic CASLO template is brought to life. Ultimately, the CASLO approach is little more than a high-level design template that provides a foundation for building a broader qualification design template, which is bespoke to the qualification in question. It is the adequacy and appropriateness of this broader qualification design template that really matters (as well as how it gets put into practice, of course). Figure 17 attempts to illustrate this idea, indicating that multiple layers of planning and decision making are required to secure the success of a CASLO qualification.

Qualification design always needs to begin with a rigorous analysis of the purposes that a new qualification (or qualification type) will need to serve, given the nature of the cohort for whom it is being designed, and the contexts within which it will need to operate (Newton, 2023). Let us assume that an analysis of this sort recommends adopting the CASLO approach. Ultimately, all this means is that our proficiency model needs to be specified in terms of learning outcomes, assessment criteria, and a stringently applied mastery principle. This is the ‘CASLO template’ level of analysis from Figure 17. Upon this foundation, CASLO qualification design might diverge in all sorts of different ways, and these subsequent design decisions will be critical to the success, or otherwise, of the qualification in question.

At the ‘CASLO detail’ level of analysis, we will need to decide exactly how our learning outcomes and assessment criteria are to be represented. This is far from a trivial decision, as the NVQ story taught us: the approach that was advocated by the NCVQ – the NVQ competence model – proved to be highly controversial. The subsequent QCF approach was far more traditional, tending to frame learning outcomes as elements of knowledge, skill, or understanding. QCF assessment criteria were not unproblematic, though, often having been written more like mini learning outcomes than as definitive criteria. TEC and BEC designers also experimented with different approaches to representing outcomes and criteria.

Note that the decision concerning how best to represent learning outcomes and assessment criteria is just the first step at this level of analysis. Subsequent steps include determining an appropriate grain size for writing learning outcomes, deciding how the assessment criteria ought to function and how many to include, determining how to build depth into outcomes and criteria (for example, via Bloom’s Taxonomy), and so on.

The next level of analysis concerns the ‘broader design template’ and this is where all of the remaining qualification design decisions get made. On the one hand, this means fleshing out details of the assessment procedure that underpins the qualification. On the other hand, it means explicating implications for teaching and learning, particularly when a specific approach is anticipated, which might also affect the design of the assessment procedure. Again, decisions at this level will be critical to the success of the qualification.^{[footnote 13]} Obviously, the mastery principle will be relevant here, as mastery learning is fundamental to mastery certification. However, there might be broader issues to consider, for instance, the extent to which the qualification promotes a problem-based approach to learning. Considerations at this level of analysis explain why the CASLO family is so broad, embracing both NVQs (at one end of the spectrum) and BTECs (at the other).

This brings us to the ‘learning programme’ level of analysis, which is where qualification requirements get translated into learning journeys and learning experiences. Ultimately, this is the responsibility of teachers and trainers in colleges and workplaces. Yet, there is plenty of scope for awarding organisations to provide support and guidance at this level too. More to the point, qualifications need to be designed with this stage in mind, so as not to incorporate features or processes that might frustrate the production of effective learning programmes. This is why Stanton has consistently argued that teachers and trainers need to feed into qualification design decisions, both from the outset and throughout the process. We will return to this level of analysis later under the ‘whole system reform’ heading.

Reforming TVET qualifications

The story of the CASLO approach comprises a very large part of the story of regulated TVET qualifications in England over the past half century, spanning multiple major reforms. So, we end this report by considering lessons that we might be able to learn about TVET qualification reform more generally, based on experiences from designing and implementing CASLO qualifications in England.

One of the key challenges that we have faced so far (in this report) has been to distinguish goals that were specific to the CASLO approach from far broader ones (that had little or no bearing on why the approach was adopted). However, we are now going to begin this final section by focusing specifically on one of most important of the broader TVET reform goals, namely rationalisation.

Rationalisation

Rationalisation through increasingly explicit regulation has been the dominant meta-narrative to stories concerning qualifications in England over the past century. This has involved creating systems, frameworks, and agencies, where none previously existed, and attempting to reduce the number and variety of qualifications on offer. This has been true of all types of pre-university qualifications. For instance, the Acland report on ‘Examinations in Schools’ (Acland, 1911) led to the first national exam system in England, based upon the School Certificate and Higher School Certificate. These exams were administered by a small number of universities and overseen by the Secondary Schools Examinations Council, a new co-ordinating body that reported to the Board of Education.

Although a number of TVET qualification subsystems were established during the early years of the 20th century, including the system of Ordinary and Higher Nationals – and although major players like the City & Guilds and the RSA had their own organisation-specific qualification systems – the general lack of structure and co-ordination became increasingly poignant over time as problems became increasingly apparent. Government was resistant to exercising control over the TVET landscape, seeing training principally as a matter for industry and commerce. Yet, by the end of the 1950s, it was clear that this hands-off approach was not working, and strong central intervention became part of the solution, initially in relation to training and subsequently in relation to qualifications.

The TEC and the BEC were established to rationalise the technician qualification subsystem during the early 1970s. By the end of the 1980s, the NCVQ had been established to rationalise the entire landscape of TVET qualifications. The NCVQ was the first agency to introduce a national qualification framework, to which all qualifications would be accredited. This was a powerful rationalising tool, as it was designed to ensure that all qualifications adopted exactly the same high-level design template, thereby attempting to prohibit the variety of approaches that had become a hallmark of TVET qualification provision prior to that. Since the 1980s, government-sponsored agencies have controlled the TVET qualification landscape using a succession of qualification frameworks. Some of these have been more prescriptive (for example, the Qualifications and Credit Framework) and others less so (for example, the current Regulated Qualifications Framework).

Fewer qualifications

Consistently, over time, the goal of reducing the number of qualifications within the TVET landscape has been pursued in the name of creating a simpler system. The idea of counterproductive complexity was captured in the idea of there being a qualification ‘jungle’ that needed to be cleared, leaving only the finest specimens behind. Clearing the so-called ‘jungle’ in this manner would make it more navigable for learners and for employers – and for teachers, trainers, parents, careers advisors, and other stakeholders too – increasing the likelihood that learners would locate high value qualifications (and progression routes) best suited to their needs and aspirations.

Unfortunately, experience has taught us that it is harder than it sounds to reduce the number of qualifications in the market. Indeed, reforms intended to rationalise the system can result in a proliferation of new qualifications, which is a phenomenon that was recognised as early as the 1960s (Haslegrave, for instance, identified a proliferation of new courses in the wake of the 1961 white paper).

Prescriptive qualification frameworks that formally introduce new qualification types, such as the NVQ framework and the QCF, appear to be particularly vulnerable to the risk of qualification proliferation. When the NVQ framework was introduced, the NCVQ anticipated that all regulated TVET qualifications would be subsumed within it. Yet, many existing qualifications continued to exist beyond the framework, and even the new GNVQ was not actually accredited to the NVQ framework. This necessitated a broader framework, and the National Qualifications Framework was introduced in 2000.

Half a decade after NVQs had been introduced, the FEFC observed that there was “little evidence” that this had “led to a rationalisation of qualifications offered within colleges” (FEFC, 1994a, page 12). Indeed, Williams concluded that the introduction of NVQs had “simply added to the existing jungle” (Williams, 1999, page 218).

A similar thing happened when the QCF framework was introduced, which was also originally anticipated to subsume all regulated qualifications. A larger proportion of the qualification market ended up being regulated under the QCF than under the NVQ framework. Yet, plenty of TVET qualifications continued to exist beyond it, regulated under the NQF. Furthermore, just as under the NVQ framework, qualifications actually proliferated under the QCF (Oates, 2013a).^{[footnote 14]}

Part of the challenge in exercising control over the system via qualification frameworks is that the state, in England, has only limited powers, given that qualifications are provided by independent awarding organisations. Government can attempt to persuade these organisations to replace their existing qualifications with new ones – and can heavily influence this via funding and performance table requirements that impact on qualification consumers – but it has tended not to go so far as to prohibit awarding organisations from continuing to offer old qualifications (alongside new ones). Where the old qualifications have continued to be sufficiently attractive to a sufficiently large number of consumers, the awarding organisations have had an incentive to continue offering them. Bear in mind that even school and college qualifications sometimes have national or international markets that are not necessarily swayed by funding or performance table pressures.

Furthermore, on a number of occasions, government has capitulated to pressure to retain certain qualifications, ultimately recognising that they do fulfil an important need (for certain students and qualification users) that was not adequately fulfilled by their notional replacements. This happened with the NVQ framework, when funding was approved for certain existing qualifications that failed to satisfy the NVQ accreditation criteria. It also happened when successive GNVQ reforms ran out of steam, and pressure to withdraw BTECs was lifted.

The BTEC example is also interesting for highlighting the importance of branding. Even qualification users (like employers) rarely understand how qualification systems operate, beyond familiarity with the qualifications that they (as individuals) took when they were at school and college, and familiarity with qualification suites that have traditionally underpinned their own sector. For instance, it was widely observed how poorly employers understood the NVQ system, and they proved to be particularly resistant to being educated. In a context like this, the maxim ‘better the devil you know’ is likely to hold true. The BTEC council was therefore very shrewd in retaining the ‘National’ and ‘Higher National’ nomenclature following withdrawal of the old system of Ordinary and Higher technician awards. And, when the Council was ultimately disbanded, continuation of the BTEC brand was equally shrewd.

One final observation concerns a more subtle risk of qualification proliferation. New prescriptive frameworks tend to act like ‘new brooms’ that inadvertently open the market up to new providers. This occurs when a new framework aims to replace existing qualifications with a new type of qualification, like an NVQ or a QCF qualification, which is built to comply with new design rules. Where a single provider might previously have won market share for an established qualification over an extended period of time, new design rules can wipe the slate clean, levelling the playing field for the new awards. The replacement qualification might now be offered by multiple awarding organisations.

Qualifications certainly proliferated under the QCF as identical units were packaged into the same qualification offered simultaneously by multiple awarding organisations. Beyond proliferation, per se, this approach ran the risk that some of these new qualifications would be developed and delivered far less effectively than others. This was a problem under the NVQ framework (Raggatt & Williams, 1999) and proved to be so under the QCF. Indeed, one of the principal reasons for withdrawing the QCF was that it had (intentionally) opened the market up to new, inexperienced awarding organisations, some of which developed qualifications that managed to satisfy QCF accreditation criteria whilst ultimately lacking validity (Ofqual, 2014b).

Homogenised qualifications

Consistently, over time, the goal of homogenising the design of qualifications within the TVET landscape has been pursued in the name of creating a stronger system. Instead of leaving the finest specimens to flourish, post-deforestation, government and its agencies in England have tended to want to cut the entire jungle down and replant it as an arboretum of perfectly cultivated trees. This was the logic of the NVQ framework, which specified accreditation criteria in such detail that it effectively called for universal qualification reform, and it was the logic of the QCF too.^{[footnote 15]}

The rationale for starting from scratch with a brand new qualification model appears plausible and attractive. In the NVQ context, it resulted from having identified serious problems with existing qualifications that appeared to be linked to specific design features. By specifying a different approach to qualification design, these problems were intended to be addressed. This rationale directly explains the origins of the CASLO approach in England.

However, while plausible and attractive, the desire to homogenise qualifications – which is typically characterised as ‘strengthening’ them – runs multiple risks, which relate to the possibility that these newly specified design features might:

1. not actually solve the problems that they were supposed to solve
2. introduce new unanticipated problems
3. work well for certain qualifications but not for others
4. prevent the use of even better design features (thereby also stifling innovation and competition)

Successive reforms to qualification systems and frameworks in England have swung the pendulum of change a very long way away from the situation that pertained in the 1950s, when the leading TVET awarding organisation was proud to declare that there was no such thing as a typical City & Guilds qualification. Back then, responsiveness to local needs and aspirations was the guiding principle. Even when the TEC and the BEC took steps to rationalise the technician qualification subsystem, during the 1970s, they still embraced the idea of responsiveness by implementing a validation model, whereby centres were responsible for programme development and were encouraged to customise provision accordingly, albeit within parameters. Having said that, even during the 1970s and 1980s, the demand for customisation of this sort was not overwhelming, and centres often preferred to rely on standard units (which, it is fair to say, still offered a certain amount of flexibility). Nowadays, customisation of BTEC units occurs only rarely, by exception.

Interestingly, CASLO qualifications have tended to be homogenised in a manner that retains a certain amount of responsiveness. On the one hand, their core characteristics mean that all CASLO qualifications are highly constrained by the detail in which learning outcomes and assessment criteria are specified, and by specifying the mastery requirement. On the other hand, when those learning outcomes are specified at a fairly high level, this can facilitate a certain amount of customisation, which was encapsulated in the goal of (minimal) cross-context domain personalisation. This can also be facilitated by incorporating optional units, or routes, which also helps to improve responsiveness while keeping proliferation in check (as emphasised by both the Beaumont and CAVTL reports). Likewise, by exploiting the transparency inherent in the approach, this permits a certain amount of operational flexibility (which was encapsulated in the goal of qualification efficiency).

Yet, qualification homogenisation is no panacea or silver bullet, even when cunningly operationalised. For instance, centrally specified NVQ standards proved to be problematic when defined either too narrowly, where learners failed to acquire competencies that they actually needed, or too broadly, where learners were required to acquire competencies that they did not actually need. Although each NVQ standard was intended to represent a consensus position, which established a certain amount of democratic legitimacy, this also meant that they were vulnerable to not reflecting the needs of any particular employer very well, or to only representing the needs of certain employers when the consensus building was unrepresentative (see Oates, 2004, on NVQ standards as pragmatic constructs).

In certain areas of certain sectors, problems of this sort challenged the very idea of a national qualification. The review of NVQ assessment produced by Eraut, et al (1996) brought this challenge home. To some extent, a qualification framework is a useful fiction, which helps to make sense of similarities across otherwise disparate contexts, but that cannot be taken too literally. For instance, just because 2 NVQs were both classified at Level 2, that did not literally mean that they had exactly the same level of demand and required exactly the same amount of training time – particularly not under the NVQ framework where standards were designed to be role specific. Thus, the Eraut report explained how carpentry and joinery were much more demanding than painting and decorating, which needed to be recognised in terms of both training time and funding. The report went even further by arguing that:

It is impossible to have a national system of qualifications based on current competence at work; because we do not have a national system of working practice. But it is possible to have a national system of competence-based units which are not mandatory for all trainees.

(Eraut, et al, 1996, page 67)

The Wolf report reached a similar conclusion:

However, because a complex modern economy has a correspondingly complex occupational structure, central attempts to impose a neat, uniform and ‘logical’ structure on it always fail.

(Wolf, 2011, page 57)

It seems likely that the ‘sweet spot’ lies somewhere between the extremes of, on the one hand, qualification design being determined by the accreditation requirements of a single overarching framework and, on the other hand, of qualification design being entirely bespoke. Exactly where on that continuum the sweet spot lies is unclear. Indeed, it might well be in a different location for different qualifications (see also Oates, 2013b).

Clearly, though, the higher the level at which a qualification framework is implemented – a single overarching framework being the highest level – the greater the risk of it failing to accommodate the diversity of the qualifications that it subsumes. This underlines the necessity of comprehensive trialling and piloting: the higher the level of the framework, the more extensive this trialling and piloting will need to be. Yet, the very complexity of occupational structure in England renders trialling and piloting (and, subsequently, evaluating) extremely challenging. Trials undertaken in one area of one sector are quite likely not to generalise to another.

In theory, the decision to implement one qualification model as opposed to another – for instance, the CASLO approach rather than a classical one – ought to be taken on the basis of a detailed analysis of the purposes and cohorts that it is intended to serve, and the contexts within which it is intended to operate. If, instead, this decision is taken (by fiat) at the qualification framework level – by building it into accreditation criteria – then the framework designer will need to be correct in assuming that the model will not be inappropriate for any of the qualifications subsumed by the framework. Both the NVQ framework and the QCF fell short in this respect.

One final homogenising tendency is worthy of mention, which involves making TVET qualifications look more like classical ones, typically in the name of strengthening them, but sometimes also in the name of improving parity of esteem. This happened repeatedly to GNVQs. By the time the AVCE model was introduced, in 2000, the qualification had effectively lost its way. Shorn of its original defining characteristics, it no longer seemed to be fit for any particular purpose, and it was soon withdrawn. The qualification that followed in its footsteps was literally an A level, the Applied A level, although it was not a great success.^{[footnote 16]}

School and college performance table qualifications were partially homogenised by DfE requirements that followed in the wake of recommendations from the Wolf report. This led to the hybridisation of many CASLO qualifications, whereby the changes effectively enforced a classical approach on certain units, while other units continued to operate under the CASLO approach. The impacts of hybridisation along these lines are still to be thoroughly investigated.

Control

As government has increasingly intervened to rationalise qualification systems in England, it has increasingly exercised control over those systems through networks of co-ordinating and regulatory agencies, of which Ofqual is one. This raises important questions concerning where responsibilities for different aspects of qualification design, development, delivery, and review ought to reside, and what the consequences of different configurations might be.

These questions are particularly challenging because of how widely shared these responsibilities have traditionally been, owing to the fact that qualifications operate at the interface between curriculum, pedagogy, and assessment (that is, at the interface between quite different stakeholder groups). This is further complicated within the TVET landscape by the disparate needs of employers, not to mention ambiguity over who ought to be responsible for meeting those needs, let alone how to ensure sufficient engagement within a largely voluntaristic system.

Central intervention

The assumption that employers ought to take responsibility for their own training needs acted as a brake on central intervention until the early 1960s. The establishment of a Central Training Council and Industrial Training Boards began to change this. Initially, the focus was upon training, and training standards. But attention soon turned to qualifications, and the certification of competence, when the TEC, the BEC, and the NCVQ were established. Through the NVQ system, government simultaneously exercised control over standards for education and training (National Occupational Standards) and methods for assessing and certificating those standards (National Vocational Qualifications).

Critically, the CASLO approach was part of this initiative. The features that comprise it were implemented fully within NVQ awards, and partially within TEC and BEC awards. The existing awarding organisations were already largely signed up to the general idea of specifying qualifications in terms of learning outcomes and assessment criteria, but it was government-sponsored agencies that implemented this approach at scale.

Responsibilities

Because qualifications operate at the interface between curriculum, pedagogy, and assessment, there will always be some debate over how best to achieve their co-ordination. For over a century, awarding organisations in England have played a central co-ordinating role, and were assumed to be where the buck stopped in terms of accountability for the quality of those qualifications. Within the TVET landscape, the preeminent awarding organisation during the first half of the 20th century was City & Guilds. It relied heavily upon advisory committees for guidance on preparing course syllabuses, ensuring that the views of educationists, employers, and trade unions were all heard. For the original Ordinary National and Higher National awards, this co-ordinating role was assumed by Joint Committees representing the relevant professional institution(s), the Ministry of Education, and teacher associations.

Although central coordination is clearly critical, and although there are good reasons for government to be part of that co-ordinated activity, experience also teaches us that there are challenges associated with increasingly centralised control. Indeed, the Wolf report argued against government control, on the basis that having to assume ultimate responsibility will make it “correspondingly impossible to be honest” when qualification reforms go wrong (Wolf, 2011, page 9).

This raises an important question concerning ownership of, and responsibility for, the CASLO approach. If a qualification model, like the CASLO approach, is specified as a qualification framework requirement, then there is a lot of sense in it being ‘owned’ centrally by government. Yet, it is the responsibility of an awarding organisation to design, develop, and deliver qualifications using this high-level design template. This makes it unclear exactly where the buck stops in terms of the quality of qualifications within the framework, particularly where it is impossible to disentangle issues arising from the centrally specified model and issues arising from features and processes specified by an awarding organisation. In the worst case scenario, it disposes awarding organisations to abrogate responsibility for quality, instances of which appear to have occurred under the QCF, when it was entirely unclear where responsibility for the quality of units was located.

Expertise

Increasing central control also affects the structure of knowledge within a sector, in this instance the qualifications sector. While the awarding organisations were experimenting with outcome-based approaches during the 1970s and 1980s, the NCVQ was to become the powerhouse of the CASLO approach, with an increasingly powerful internal research function (Ecclestone, 1998). With the Training Agency, it oversaw the design and implementation of the NOS-NVQ model. Based on functional analysis, this model was technically demanding to implement. It was an expert system that required expert practitioners, who were few and far between even then, and the system relied heavily upon centrally contracted consultants.^{[footnote 17]}

Perhaps the biggest risk associated with locating critical knowledge centres within government-sponsored agencies relates to what happens when their functions are dissolved and they are disbanded. The agencies that followed the NCVQ and the Training Agency did not ‘own’ the CASLO model in quite the same way or to the same extent. Furthermore, because the awarding organisation sector increasingly developed qualifications on the back of NOS, rather than developing standards from scratch, they increasingly became users rather than owners of standards. In essentially the same way, they increasingly became users of the CASLO approach rather than owning it. This may be at least part of the reason why there is so little published research into the approach. If the awarding organisations were not ultimately responsible for it, then were they similarly absolved of responsibility for researching and evaluating it?

So, who are the CASLO experts today, and in what kind of organisation might we expect to find them? It is not entirely clear. The fact that the present programme of research was even necessary raises a fundamental question concerning the structure of knowledge within the qualifications sector. One thing is for sure, though: however that knowledge structure is configured – and it could legitimately be configured in a variety of different ways – it must always be the case that whichever organisation is ultimately responsible for qualification, or qualification system, design decisions, it must be equipped with the necessary expertise to make those decisions effectively.

As far as expertise is concerned, the qualifications sector faces an unfortunate, if not ironic, impediment: there is no formal ‘certification profession’ in England and there are no generally accepted certification qualifications. Unlike professions such as medicine, or accountancy, you do not qualify to become a certification professional, you tend to fall into the qualifications sector, and accumulate relevant knowledge, skills, and understanding on the job. At the very least, this lack of formal recognition of the expertise required to design and implement certification systems is odd. At worst, it constitutes a systemic threat to the quality of professional practice in the qualifications sector in England.

Pragmatics

Finally, our analysis provides insights into how TVET qualification reform has been undertaken over the years, including lessons concerning how reforms can go wrong.

Speed

The introduction of Curriculum 2000 is remembered less for the new AVCE, which was to fizzle out within a few years, and more for the first new modular A level awards, which exploded with a bang shortly after A level results day in 2002 (Tattersall, 2007). Anomalously low results in certain units led to multiple official inquiries and ultimately to 18 unit grade boundaries being lowered and to 9,800 unit grades being raised. It also led to the Chairman of the QCA being sacked, and to the Education Secretary resigning shortly afterwards. A Select Committee on Education and Skills report into the crisis concluded:

On the evidence presented to us, we conclude that the events of last Summer were not caused by the manipulation of the examination system but by confusion arising from the introduction of the A2 exam without adequate trials.

(HCESC, 2003, paragraph 78)

The government responded by acknowledging:

We recognise absolutely that there are lessons to be learnt for the future about the way in which we implement major reforms of this sort. Detailed planning and extensive trialling is essential so that we can be confident that all systems are in place and that teachers and examiners are fully trained in new requirements before they are introduced.

(DfES, 2003b, paragraph 5)

Qualification reforms are always under pressure to deliver results as soon as possible. Even when piloting and trialling activities do take place, there is no guarantee that problems with the new systems and qualifications will be spotted and resolved. And, of course, it takes a long time to pilot and trial a qualification.^{[footnote 18]} Both the QCF and the AVCE were at least notionally ‘trialled’ before going live, yet, they both ultimately failed.^{[footnote 19]}

What seems eminently clear, however, is that systems and qualifications as complex and novel as the NVQ system, or the GNVQ, stand little hope of succeeding without extensive piloting and trialling over a significant period of time. Scaling up too soon can cause serious reputational damage to the qualification brand when things go wrong on a national scale. In this situation, even if the qualification model is rapidly revised to address these issues, the initial reputational damage may be sufficient to have undermined the credibility of the new qualification brand (and, for qualifications, credibility is the bottom line, more so even than validity). In the words of Bent Flyvbjerg and Dan Gardner – think slow, act fast. It is far better to make mistakes in the planning stage than during delivery (Flyvbjerg & Gardner, 2023).

Given their complexity and novelty, it seems fair to conclude that both NVQs and GNVQs were scaled up to national rollout too quickly, with shambolic consequences. The fact that they were both based upon a new qualification model – not to mention adopting other innovative and idiosyncratic features and processes specific to each of them – greatly magnified the risks associated with rushing their implementation. Critically, these reforms did not simply have radical implications for assessment, they also had radical implications for teaching and learning, which is a point that we will return to shortly.

Engagement

These radical reforms also provide insight into the importance of engaging stakeholders effectively. The introduction of NVQs provides an important case study in its own right, given just how radical the reform was, and just how extreme the critique from certain quarters proved to be. It seems reasonable to conclude that the NCVQ should have adopted a better strategy for engaging stakeholders, although communications would always have been highly challenging to manage bearing in mind that the reform was explicitly intended to disrupt existing practices.

Unfortunately, the NCVQ shook the system so violently, and rapidly, that it ended up metaphorically at war with many academic educationists. Although NCVQ research teams did actively collaborate with academics who generally supported their approach (such as John Burke), and with those who were prepared to contribute as critical friends (such as Alison Wolf), they struggled with those who were more openly hostile (such as Alan Smithers). As researchers surveying the scene 2 to 3 decades later, we see evidence of these rifts even to the present day.

Employer engagement is a perennial challenge for TVET in England, for many varied and complicated reasons (see, for example, Stanton & Bailey, 2004; Unwin, et al, 2004; BIS & DfE, 2016; Huddleston, 2020). The basic challenge is how to ensure that the voice of employers is sufficiently loud, representative, and clearly articulated to ensure that their needs are adequately reflected in TVET qualification design. However, to the extent that employers are also expected to play an active role in qualification delivery – the idea at the heart of the NVQ model – the challenges multiply significantly.

Inevitably, the more we expect employers to be actively involved in the technical details of qualification design and delivery, the more accessible those technical details will need to be. This runs the risk of creating tension between the need for qualification rigour and the need to be responsive to employer demands for simple systems and processes (as discussed above). When employer engagement is required, yet can only be incentivised and not enforced, this increases the risk of capitulating to demands that result in oversimplified systems and processes.

Once again, the challenge is how to balance the scales, such that employers are sufficiently engaged, but not so much so that they end up with responsibilities that qualification specialists could deliver more effectively.

Overstepping the mark

One of the most important risks highlighted by our research is that reforms overstep the mark. This can happen when things go wrong, but the proposed solution swings the pendulum of change too far in the opposite direction. When this happens, it is likely that the new system will fail for exactly the same reason as the old one, albeit operating in reverse.

Once again, the NVQ model provides a good illustration of this phenomenon. Previous qualifications had been tailored to off-the-job education and training, and were rightly criticised for focusing too much on theory and book knowledge. Yet, the solution – the NVQ model – ended up being tailored to on-the-job training, and was rightly criticised for focusing too much on practice and informal learning. This threat was identified early on (see Black & Wolf, 1990) but the model was already too entrenched by then for this pendulum swing to be reined in effectively. This underscores the risks associated with reactive, knee-jerk reform (Lum, 2015). One final consideration is that the more radical the reform in question, the more challenging it becomes to implement, which increases the importance of not overstepping the mark with a reform that is more radical than it actually needs to be.

The QCF provides a slightly different illustration of overstepping the mark, which involved generalising to all learners a solution that seemed to work well in the context of adult learners. The utility of a flexible unit-based framework had been demonstrated in the OCN context: effectively empowering adult learners – especially women who wished to return to the workforce – to re-engage with the discipline of learning, one small step at a time. However, when generalised to the QCF context, the idea that all learners would benefit from being able to mix and match units to form bespoke qualifications was never substantiated. Indeed, the flexibility built into the QCF had the unintended consequence of facilitating, if not promoting, incoherent curriculum programmes.

In addition to overgeneralising a solution to the genuine problems faced by returning adult learners, the QCF also engineered solutions to problems that proved not to be significant after all. Paramount, in this respect, was the facility for transferring credit across awarding organisations, for which there proved to be very little demand indeed (Ofqual, 2010b; 2011b).

Whole system reform

By far the most important risk highlighted by our research is that qualification reforms are conceptualised and operationalised too narrowly, with insufficient attention to the wider education and training changes that are necessary for those reforms to bed in (see also CAVTL, 2013; Oates, 2004; 2013a; 2013b). Qualification reforms are best understood as education and training reforms that are initiated through changes to certification requirements. When considered from this perspective, the importance of adequately involving and supporting teachers, trainers, centres, and learners right from the outset becomes harder to overlook.

In terms of involving key protagonists, Stanton (2012) argued that standards need to be developed in collaboration with teachers and trainers, to maximise the likelihood of their being meaningful and useful when subsequently translated into learning programmes (see also Callender, 1992; CAVTL, 2014). The same principle would apply when standards are translated into certification arrangements, which suggests that standards ought to be developed in collaboration with awarding organisations too. Stanton returned to this theme a few years later, arguing that one of the reasons why agency-designed qualifications have failed in the past was:

The use of linear rather than iterative development processes, without piloting, and with teachers being regarded as implementers of schemes rather than having a role in their design.

(Stanton, Morris & Norrington, 2015, page 79)

In terms of supporting key protagonists, Stanton (2012) also emphasised the risks associated with handing standards over to teachers and trainers without further elaboration, as though they somehow constituted a programme of learning, which is how (in the absence of satisfactory support) they were often treated. Teachers and trainers were required to become qualified in NVQ assessment principles and practices, but not in NVQ teaching and learning principles and practices.^{[footnote 20]}

Similarly, the speed with which GNVQs were rolled out left too little time for developing effective support systems and guidance materials for curriculum and pedagogy (FEU, 1994). Nor was there adequate time or resource for staff development. Indeed, although materials became available over time, including materials designed to facilitate the consistent interpretation of standards, Higham (2003) found little evidence of any systematic form of staff development, other than teachers having to complete units on assessment. Once again, these were units on assessment, not teaching and learning, despite mastery certification being premised on mastery learning, which implicated an entirely different philosophy of teaching and learning, not to mention entirely different teaching and learning strategies (see also FEFC, 1997). As one of the awarding organisation officials interviewed by Kathryn Ecclestone put it:

So you had school teachers blundering into this with a lack of specialist knowledge, with no materials and a style of learning that they had never had to use before.

(Ecclestone, 2000, page 549)

Even toward the end of the 1990s, Ofsted was still insisting that GNVQ teachers needed more help with curriculum planning, teaching methodology, and assessment (Ofsted, 1999). Butcher (1998) described how Initial Teacher Training paid scant attention to the particular requirements of GNVQ. More generally, Ecclestone (2010) emphasised how the state of Continuing Professional Development continued to be extremely poor. These challenges were compounded by the more fundamental challenge of appointing suitably qualified teachers and trainers in the further education sector (Wheatley, 1959; FEFC, 1997; Lingfield, 2012).

It was not just the NVQ and GNVQ reforms that failed to involve and support teachers and trainers adequately from the outset. The same was true when outcome-based qualifications were first introduced by the TEC and the BEC. Again, these were radically new qualifications, which changed what it meant to teach and learn, as well as carrying an expectation of needing to be customised to local needs.

None of this was adequately factored into the qualification reform process. Former Minister of State for Education and Science, Gerry Fowler, put it like this:

This takes me back to BEC and TEC. In my experience, such is the burden upon some colleges of devising new syllabuses for the approval of the Councils that there will have to be either an acceptance of the need to use (in the short term at least) prepared syllabuses and even prepackaged course materials, or a recognition by the DES and Local Education Authorities that bricks cannot be built without straw. Additional staff resources would solve the problem; their provision seems unlikely. There is a lesson in this for the future. Further changes in the machinery of course validation may be highly desirable, as I have argued. But they will have to come gradually and piecemeal, if the system is not to collapse under the strain.

(Fowler, 1978, page 57)

Some years after the introduction of the new BTEC model, an evaluation conducted by the National Foundation for Educational Research emphasised that many staff still lacked necessary skills. The evaluation report explained that there was a “need for radical development of the complex of skills required for the role of manager and facilitator of learning” (FEU & BTEC, 1990, page 7). An earlier report from the Further Education Unit had raised similar concerns (FEU, 1986).^{[footnote 21]}

This concluding section can be summarised in a quotation from Alan Brown, who explained that the clear lesson from framework-driven reforms in England is that:

an emphasis on qualifications development needs to be balanced with equal concern about how learning and development will be facilitated in practice

(Brown, 2011, page 04-2)

Brown is just one of many scholars who have reached a similar conclusion (see Hodkinson & Issitt, 1995, for another example). It remains unclear why this lesson was not straightforwardly learnt from earlier TVET reforms to later ones, although a lack of policy memory may have something to do with this.

Conclusion

It is hard to draw a pithy conclusion from our nuanced and multifaceted investigation into the origins and evolution of the CASLO approach in England. So, we will end with just a couple of high-level observations concerning the CASLO approach in the context of outcome-based and mastery-based qualifications more generally.

First, the very fact that we can distinguish the CASLO approach so clearly – as a high-level design template that has underpinned so many TVET qualifications over the past few decades – is interesting and important in its own right. As we have seen, the approach is just one way of operationalising an outcome-based qualification model, and just one way of operationalising a mastery-based qualification model. So, why did it become ‘the’ way in England, achieving almost hegemonic status as a TVET qualification model?

Clearly, the answer has much to do with the homogenising tendency of qualification regulation in England, which led to the approach being specified as accreditation criteria within both NVQ and QCF regulations. However, it also has something to do with the general zeitgeist of enthusiasm for outcome-based and mastery-based approaches, which took root during the 1970s in the wake of the Objectives Movement, and which persisted within the TVET qualification sector despite not taking root in the general qualification sector nor in the national curriculum assessment context. Once the CASLO approach rose to national prominence within the NVQ model, it set a precedent for qualification designers to follow. Some followed under duress. Others followed willingly.

Second, given its roots in the Mastery Movement, the CASLO approach is fundamentally a philosophy of teaching, learning, and assessment, as embodied in certification requirements. In other words, it is not merely the ‘occupational way’ of designing qualifications. Of course, it makes a lot of sense in certain occupational contexts, where achieving mastery means becoming fully competent in an occupational role. Yet, even in occupational contexts, it is still premised upon mastery learning, so the underpinning philosophy is still there, albeit only implicitly.

Contrary to conclusions from certain policy reviews, the CASLO approach is not straightforwardly inappropriate in general education contexts. Far from it. The very idea of mastery learning was a response to the tendency, within general education, for teachers to treat failure as normal and unproblematic. However, it is entirely legitimate to question whether the CASLO approach is appropriate for any particular general education context. Indeed, it might well be judged appropriate for certain contexts but not for others. Furthermore, it is quite possible to adopt a mastery approach to teaching and learning without that necessarily culminating in mastery certification. Finally, the CASLO approach incorporates a plausible approach to mastery certification, but it is not the only approach. So, this begs the question of whether it is the optimal approach for any particular context.

Over the years, we have come to conceptualise and operationalise outcome-based qualifications quite narrowly. Hence, the CASLO approach achieving almost hegemonic status. Yet, there is nothing sacrosanct about it. There are certainly better and worse ways of implementing the CASLO approach, but there might also be better ways of designing outcome-based qualifications, more generally, and better ways of certificating mastery. This behoves us to think more broadly and creatively about the significance of outcomes and mastery when designing vocational and technical qualifications for the future.

1. We also noted that many CASLO qualifications only awarded the passing grade, and that many were not heavily time constrained. ↩

2. Syllabuses came to be known as ‘specifications’ during this period, such that the idea of a ‘specification’ superseded the idea of a ‘syllabus’. Having said that, because the idea of a specification has less everyday currency, and is far more generic, the term ‘syllabus’ is still quite useful, and we sometimes use it even when discussing current arrangements. We distinguish both ‘specification’ and ‘syllabus’ from the idea of ‘curriculum’ planning, which goes beyond the requirements of a particular specification, integrating broader considerations related to the organisation of teaching, learning, and assessment. ↩

3. We also distinguished minimal from radical domain personalisation, and we distinguished domain personalisation from what we labelled ‘teaching and learning approach’ personalisation and ‘assessment format’ personalisation. ↩

4. Concerns such as these can be seen in the literature that attempts to unpack the sociopolitical goals underlying the introduction of the CASLO approach (for example, Young, 2008). ↩

5. Note also that, while England moved away from the NVQ model, Scotland has retained its SVQs. ↩

6. Nor does it answer the even more complicated comparative question of the pros and cons of the CASLO approach versus the classical approach in relation to the different goals that they set out to achieve. The fact that CASLO qualifications are often designed to achieve quite different goals – mastery learning being an important case in point – would certainly complicate a comparative analysis of this sort. ↩

7. Wolf suggested that this was implicit in the relative lack of attention paid to this threat by those responsible for NVQ and GNVQ policy and practice. ↩

8. A QCA evaluation of BTEC awards also raised concerns over inconsistent assessor judgements (QCA, 2005), as did a couple of Ofqual investigations (Ofqual, 2010a; Cuff, et al, 2018). ↩

9. This does not undermine the idea of a mastery judgement. It simply underscores the fact that mastery can only legitimately be inferred on the basis of multiple sources of relevant evidence. ↩

10. Or, for graded qualifications, the purpose is to help articulate the difference between operating at a lower level and operating at a higher one. ↩

11. We have had many fruitful conversations with Chris Winch over the different ways in which outcome-based approaches are defined in different contexts, and how unhelpful this can be. His definition seems to be somewhat narrower than ours. ↩

12. Where qualification design is driven primarily by the need to certify occupational competence, certain of these factors (such as qualification demand) will not be negotiable, which may necessitate holding other factors open (such as qualification time). Where qualification design is simply driven by the need to provide a suitable course of learning for a particular cohort of students who will be studying for a set period of time, the challenge of pitching outcomes and standards effectively will loom large. ↩

13. This, incidentally, explains why some awarding organisations were extremely uncomfortable with the idea of building the Qualifications and Credit Framework on the foundation of a bank of shared units. It left open the possibility that an inexperienced awarding organisation might build suboptimal procedures and approaches on the foundation provided by those shared units, ultimately undermining their credibility (and the credibility of the qualifications that they were included within). Unfortunately, this did happen. ↩

14. The significance of this proliferation is hard to judge, as part of the rationale for the QCF was to accommodate awards that would not previously have been accredited (under the NQF). This shines a light on the elephant in this particular room, which is the plethora of qualifications that have always existed beyond the regulated qualifications market (see PricewaterhouseCoopers, 2005). ↩

15. Stanton, Morris, & Norrington (2015) identified a failure to learn lessons from the past – alongside the clear preference of inexperienced policy makers for a ‘clean sheet’ approach to problem solving – as 1 of 4 main reasons why agency-designed qualifications have failed in the past. ↩

16. They were more popular as AS awards than as full A levels, but uptake of both qualifications declined steadily over time until they were withdrawn (Sutch, Zanini, & Benton, 2015). ↩

17. Raggatt & Williams observed that these technical consultants were alleged to be the only ones who truly understood functional analysis (Raggatt & Williams, 1999). The credibility of the methodology was also heavily dependent upon which particular industry experts the technical consultants enlisted. Lester (2015) noted that functional analysis was particularly vulnerable to relying on senior members of relevant occupations who lacked insight into current working practices or contexts. Parkes (1994) noted the credibility threat that arose when standards were developed in collaboration with sectors that lacked a strong tradition of commitment to, and investment in, training. Callender (1992) identified risks associated with relying upon experts within industries that were neither co-ordinated nor co-operating in their contributions. ↩

18. New syllabuses take a long time to be agreed. They need to be disseminated significantly before first teaching, and the new courses will run for a year or two before the first awards are made. ↩

19. Arguably, these might be better described as ‘phased introductions’ rather than as genuine ‘trials’ given the widespread presumption of system change. ↩

20. To be fair, even the assessor qualifications appear to have been something of an afterthought (Raggatt, 1991). ↩

21. Of course, in the wake of the withdrawal of the QCF, where qualifications began to transition away from the CASLO approach, essentially the same challenge arose with the need to support teachers and trainers in delivering more classically designed qualifications. ↩