THE IMPORTANCE OF ASSESSMENTS

INTRODUCTION

In a recent survey to identify ways of improving basic knowledge and understanding of key principles in the maritime professions, a large group of ship managers and operators were invited to comment on the need for better assessments.

The result: an overwhelming 93% believe better assessments are a major factor in raising standards. Their answers also included words of wisdom and caution, for example:

• People don’t listen to just words – assessments need to be reality-based.
• Assessments are difficult to do on board.

WHY ARE ASSESSMENTS IMPORTANT?

If we could not measure the starting point, key points in progression and the completion of a learning process we would not know:

• Where to start?
• What to teach?
• Whether the student is progressing satisfactorily?
• When the learning outcome has been achieved?

As a lecturer on simulator courses for many years, where there were no formal assessments, I was surprised to be asked by one of my very well-respected senior colleagues, Keith Jones: “Without proper assessments, how do we know whether our teaching is effective or not?” His wise words were illustrated some weeks later when I was debriefing a class, to discover what they had learned during the week. One experienced Coastal Master, who prior to this course, had had very little formal training in radar navigation and collision avoidance said, “I have learned a lot! I now realise that many of the things I have been doing for the last 20 years were highly dangerous and I have only survived by good luck.” I asked if he would now do things differently, in the way he had been taught. After some thought he said, “No, I shall still do the same things as before, but I will worry a lot more!” My teaching had not changed his attitude or his performance. The week had been wasted.

PERFORMANCE IN THE WORKPLACE: THE ONLY REAL MEASURE

In STCW95, which is a competency-based certification system, the best assessment is the measurement of performance in the workplace or in a simulated workplace, such as a simulator, and it is not easy. Any other testing of knowledge gives only background information. Written examinations can measure whether seafarers can talk knowledgeably about doing the job. They do not test whether they can do it in practice. That is why we changed to a competency-based system¹. 

The problem with many Authorities today, is that they have become obsessed with paperwork and computers, and have forgotten that we are trying to test people’s ability to do a complex and demanding job on board. The fact that many countries do not have the equipment or the expertise to carry out effective practical tests has resulted in a “dumbing down” of assessments. The real danger for the future of the industry is that some people believe the dumbed down computer-based tests can measure competence – they cannot!

ASSESSMENT IN COLLEGES AND TRAINING CENTRES

The classroom, laboratory, workshop, library and simulator suite of colleges and training centres are valuable, but the most important asset for shore-based learning, is the teacher or lecturer. These “resources” need to be used effectively. It is a waste to teach material that is out-of-date or irrelevant. The curriculum content needs to be carefully selected, effectively taught and properly assessed. It is not possible to teach the detail of every trade, every type of equipment, every ship type, etc. The focus must be upon the principles, theories and rules. Hopefully not too many rules! I remember talking to David Webb, now a manager, about the Masters course he had just completed. There was too much about regulations. “I now know all the penalties and fines for getting things wrong. I wish people had spent more time teaching me how to do things right!”

We work within a framework of regulations, but the most important thing is to learn the principles on which our professional practices are based. We then have a reasonable chance of making sensible decisions when life changes and we meet a situation in which the new rule has not yet been written.

Assessments in college should ensure that all students understand the principles on which their work is based, and can apply them to realistic problems. Typical examples are damage stability, disciplinary problems, and many other situations that can be explored through case studies.

Now that there is less emphasis on written examinations in the certification process, the responsibility of shore-based teachers/assessors has increased enormously. Senior ships’ officers rely on you to ensure that the students you send on board know the theory and need only to be taught the practice of their profession. If an officer at sea has to teach basic principles, the teachers ashore have let him/her down.

ONBOARD ASSESSMENTS

In 1991 when Hong Kong Shipowners’ suggested that I write a book on how to do shipboard training², I included a short section on Assessment. The immediate reaction was “What have you done that for?” Admittedly, at that time, much of the shipboard training was delivered and monitored by an officer or travelling trainer and there was less need for formal assessment than there is today. The best training is still led by a trainer and incorporates motivational training aids, followed by a drill where the ideas can be carried through into performance. Most of the training material at that time was on video and the booklet supplied with each package gave a study plan, notes, questions and sources of further information, for use by the trainer.

Today, things have moved on, and we are asked to deliver the training as if there was a good trainer leading it, but without the trainer being there. So most of the packages are made in Videotel Interactive format, now normally on CD-ROM or VOD. They still contain the motivational video. They provide the accompanying printed text with key points, further explanations and sources of further information. A tutorial section asks questions and helps the trainee to find the answers in the video or text. They finish with a test with random questions and no assistance. If the final test is supervised the training officer can be reasonably reassured that the key points of the package have been learned and understood.

The move towards distance learning courses on board has added a new dimension in terms of assessment, many requiring the students to complete assignments and have them assessed ashore. This is a more effective assessment than just certifying attendance at a shore-based course where there was no requirement to demonstrate what you had learned.

SHIPBOARD ASSESSMENTS: HOLDING THE KEY

The most important person in the whole assessment process is the officer on board, who sees the work the seafarer carries out and signs it off in his or her record book. When you sign that document, you are telling the Authorities ashore and the future shipmates of that person, that in your view, he or she has been judged as competent by you, carrying out the particular task in a variety of circumstances, and that you believe, if asked to do the task in the future, you think he or she would be able to do it effectively. That puts a huge responsibility on your shoulders. There is a temptation at the end of a voyage to sign the cadet’s record book off in a hurry, even though you are not sure they have carried out the tasks. At the time it makes the cadet/trainee momentarily happy, if not reassured (knowing you are incompetent and having a piece of paper saying you are competent can cause sleepless nights!). By signing it without evidence, you undermine the whole competency-based certification system. There is a debate which says ships’ staff certify “performance” (meaning they have seen the job done once or twice) and the assessor ashore signs off “competence” (meaning that they have received evidence of performance in several different situations and the results of tests of underpinning knowledge).

NATIONAL CERTIFICATION

Some colleges, national administrations and shipboard programmes use multiple-choice questions. Have you heard “multiple choice” referred to as “multiple guess”? Have you heard people say they are useless? Well, they are not useless. If the assessment is properly structured and good questions are chosen, they can tell us something about the candidate’s knowledge of a subject. With appropriate pass levels, they can distinguish between those whose knowledge is inadequate and those who know the subject well. Just as we cannot say they are “useless”, neither can we say that they are an adequate measure of the candidate’s ability to do a job, to perform in the workplace, to be certified as competent. There is more to the assessment of competence than that.

The shore-based assessor must, in part, judge competence on the evidence s/he receives from the senior staff at sea. If that evidence is unreliable, the whole system is unreliable.

THE NAUTICAL INSTITUTE’S ROLE

The Nautical Institute was formed in order to raise standards. Members share that common goal. One group of people we should be trying to support and motivate are the college lecturers and assessors. One of the best ways lecturers can keep themselves up-to-date is to join the Nautical Institute and take an interest in their own profession.

The other group the Nautical Institute supports is the officers at sea. Again, the best way to motivate and encourage officers to take a serious interest in their profession and keep themselves up to date is to encourage them to join the Nautical Institute and participate.

We are about to revise STCW again. Let us remember that college lectures and senior sea staff hold the key to the future standards in our industry and deserve more support and recognition for what they do. They will determine the level of knowledge and skills for the next generation.

 

 

 

¹  1986 Technology and Manning for Safe Ship Operations in the 1990s. A UK study on future manning, which was to form the basis of the STCW 95 revised certification structure.

²  2002. Training and Assessment on Board. Witherby. Third Edition