Leading & Managing Educational Technology
Making Judgements part 2: summative assessment in ICT
By Terry Freedman
Thu, 19 Feb 2004, 00:56

Terry Freedman, who worked on the assessment of ICT at the QCA, looks at the tricky area of summative assessment.

In Making Judgements, we looked at Assessment for Learning (AfL) techniques, and what they might mean in terms of ICT. AfL, as its name suggests, is an approach designed to help the teacher help the pupil learn more -- and learn better.
It is, to risk stating the obvious, assessment for learning.
One of the benefits to the teacher of AfL is that she doesn't have to pin down exactly what level a child is working at in order to help the child progress. Indeed, research suggests that if the teacher awards a grade for individual pieces of work, the pupil actually suffers as a result, in terms of not benefiting as much as if a comment had been made instead.

However, there comes a point where the teacher really does need to make a judgement one way or the other as to the level the pupil has reached in ICT. This is where summative, or assessment of learning, comes in.

Summative assessment also has another side to it: it is, or should be, an objective measure of progress that means something to, and is trusted by, agencies that are "outside" the school: ofsted, putative parents, and the DfES for example.

That really does make life harder. ICT is a subject in which there is still a dearth of subject experts, and even fewer experts who agree with each other about levels of attainment of pupils. The fact of the matter is that ICT is still quite a young subject and so there is not (yet) a solid body of knowledge or wisdom concerning pedagogy and standards. But it is starting to emerge, as we'll see in a moment.

Add to this subject-specific difficulty a requirement for objectivity that floors even the most experienced teacher in other subjects, and you could be forgiven for believing that the task is impossible.

The situation is not aided by the fact that the Level Descriptions in the ICT Programme of Study have been written in such a way that it's hard to pin down exactly what a level 5, say, actually looks like. The general view is that the Level Descriptions are "vague". However, the truth of the matter is that they were written in such a way as to be future-proofed against advances in technology and software. They are also intended to be used in a "best fit" kind of way which relies on a teacher's professional judgement. That in itself is problematic in ICT: there aren't that many teachers of the subject who have the self-confidence to stand up and declare that they know how to assess ICT properly!

Fortunately, though, QCA recognises these problems and have come up with a couple of creative solutions. The first is to encapsulate what the Level Descriptions mean in terms of a few key characteristics, and the second is to develop a website containing pupils' work which exemplifies different Levels. Conveniently enough, you can find the former on the latter! Go to http://www.ncaction.org.uk/subjects/ict/progress.htm and you'll come to the page called Progression in ICT. More of that in a moment: for now, let's consider the NC Action website itself.

The point of the site is to help teachers actually see what a particular Level or, to be more accurate, what working at some aspects of a particular Level, looks like. It's hard -- impossible in fact -- for teachers to be able to say categorically that a pupil has reached Level X when there is no objective form of assessment available -- or at least, no generally accepted form of assessment (see the box on validity and reliability also). The Level Descriptions describe what is expected; the examples of work on the NC Action website show you.

The work is organised very well. Each piece consists of several elements: screenshots of the work, a description of what the pupils were asked to do, a commentary and further analysis. You can search for work at particular Levels, or particular key stages, or both.

When you read the commentary for a piece of work, you'll see that it does not say "This is a Level X", but "This illustrates some aspects of Level X": these aspects are given in bold print. By the same token, the text does not read "To get to Level Y the pupil should…", but "To make further progress…". The reason for this style of writing is to avoid misleading teachers into thinking, say, that if the pupil were to look at 6 websites instead of 3 she has reached a higher Level. It also recognises that, in real life, pupils' work does show elements of different Levels. The idea, as stated earlier, is to achieve "best fit".

The progression document, which lists the key characteristics of each Level Description, with examples, is a life saver. Take Level 6, for example, where the key characteristics are described as follows:

Level 6 is characterised by increased integration and efficiency in the use of ICT tools. A greater range and complexity of information is considered.

This means that when you are assessing a pupil's ICT capability (over a range of work rather than just one item), you can ask yourself whether integration and efficiency are in evidence. If the answer is no, then they are not demonstrating Level "six-ness". You can then see if they are demonstrating Level "five-ness" through the same process. This is much easier and more accurate than trying to match the pupil's work to the original Level Descriptions!

A word of caution though: do read the examples, and all of the notes on the website to do with assessing ICT, because it's very easy for teachers to misinterpret what is meant. Take the concept of "efficiency" for example. Some teachers, understandably, assumed it meant using keyboard shortcuts or touch typing. It doesn't. But to find out what it really means, you'll have to look it up for yourself!

Top tip #1: Validity and reliability
To be certain that a test (or any other form of assessment) -- in any subject --actually measures what it purports to measure, it has to be both valid and reliable.
There are several types of validity, but the most important one, at least initially, is face validity. That is, it must at least look as though it measures what it says it measures, or will not have any credibility.

A test can be said to be reliable if it produces consistent results between groups of similar pupils.

If you are thinking about buying or creating your own objective assessment of ICT, you should take these factors into account. On the subject of validity, for example, is the test really measuring a pupil's ICT capability, or her reading skills?)

Bear in mind also that the commercially available tests that have been developed will almost certainly have been trialled with teachers. That sounds good, until you realise that most teachers award a higher Level than is really the case.

Obviously, this may lead you to ask, quite reasonably, who has decreed what constitutes working at a particular Level, if there is no objective assessment available? The answer is the QCA. And before you protest that your interpretation of the Level Descriptions is much more accurate or robust, for whatever reason, you need to take into account the fact that OFSTED, BECTa and the Key Stage 3 ICT Strategy all base their assessment-related work on the QCA's interpretation. There are ongoing discussions between those groups and practitioners, which results in the guidance being clarified as time goes by, but QCA is the ultimate arbiter. (For this reason, LEAs who have their own exemplar sites and moderation procedures do need to ensure that they use the QCA examples (see main text) as a baseline for their work, rather than simply ignore it.)

Getting back to the tests, they may be very helpful in suggesting what Level a pupil is working at, but you will still need to exercise your own judgement, and come up with other ways of assessing the pupils' ICT capability (see box called Better Assessment). Even the test which the QCA is developing, which has gone through very rigorous procedures to ensure validity and reliability, and the correct evaluation of the contribution to the final outcome of the pupils' mastery of skills and techniques, is not expected to replace teacher assessment.

Top tip#2: Better Assessment
Beware the one-off assessment in the form of a written test, or a computer-based exercise. To be worthwhile, the assessment should take place over time, and should not impose a glass ceiling on what the pupil can show he knows, understand and can do.

The best way to achieve this, especially with older pupils such as at Key Stage 3, is to run projects. To be successful, these projects should:

* Be interesting

* Run for several weeks

* Involve pupils working independently, but with help available should they need it

Involve the pupils documenting what they have done, and saving versions of their work as they progress: taken together, these create an audit trail to aid you later on, or to help you justify your decisions; it also provides a record of who did what if pupils have to share computers.

A prerequisite is to make sure that the pupils are capable of working independently. This has to be built up gradually. For example, start them off by having single lessons of working independently (micro projects), then two or three lessons (min projects).

It's important to realise that pupils don't learn how to work independently simply by doing it. You need to teach them how to organise their work and their time, and how to document it. You can provide templates for these activities. You will almost certainly need to take them through the process as a kind of worked example.

Finally, some of the help available should be in the form of technical help on using the software, because pupils shouldn't be prevented from doing what they want to do just because they don't know how to do it in a particular application. Teaching pupils how to use the Help in applications, and providing crib sheets and posters on how to do the basics in various applications, are fundamental to the success of project-based assessment.

Of course, no matter how well you run this sort of activity, you should never assess the pupil on the basis of one item of work, but on a range of activities throughout the Key Stage.

See also the articles on record keeping.

A variation of this article was first published in InteracTive in 2003.


&

© Terry Freedman Thu, 19 Feb 2004