Overview

TRIADS is a powerful e-Assessment authoring and delivery system complemented by an interactive results analysis and reporting program. The unique characteristic of TRIADS is that it allows very flexible, visual question styles with sophisticated controls on question behaviour and scoring. Assessments may be run in summative or in formative mode with context-sensitive feedback either as an immediate response to a user action or as a summary at the end of each question. TRIADS can support scenarios that require concurrent, multiple interaction types overlaid on graphics and has some functionality that is not available in any other system.

History of Development

The direct precursor to TRIADS was initially developed by Professor Don Mackenzie as the University of Derby (UK) Interactive Assessment Delivery System or ‘DIADS’ and used to support computer-delivered assessment in Earth Sciences within the University. The system was used for some components of the course-ware developed by the UK Earth Sciences Course-ware Consortium project under the leadership of Dr Bill Sowerbutts at the University of Manchester, UK, funded as part of the national TLTP initiative (http://www.naec.org.uk/organisations/the-teaching-and-learning-technology-programme). This application has strongly influenced the development of highly visual course-ware style of interactions that now underpin all elements of its design.

Between 1994 & 1999 the system was further developed to provide assessments in a wider range of disciplines across the University and boosted nationally when used to underpin the TRIADS – Assessment of Learning Outcomes project. This was a collaboration between the Universities of Liverpool (Professor Chris Paul, Lead), Derby and the Open University funded as part of the national FDTL initiative (http://pcwww.liv.ac.uk/apboyle/triads/). The renaming of the system from DIADS to the Tripartite Interactive Assessment Delivery System (TRIADS) occurred at this time in recognition of the contribution made by the three Universities. The system was used by 16 disciplines across 32 academic departments in19 universities as part of the project. The state of development at this stage is summarised in Mackenzie (1999) .(http://caaconference.co.uk/pastConferences/1999/proceedings/mckenzie%20with%20images.pdf) which contains examples of some of the question styles available at this time. (See one example question from the paper below.)

In 1999 the University of Derby set up the Centre for Interactive Assessment Development (CIAD) in response to demand from a wide range of academic disciplines. This was a dedicated e-assessment design consultancy, production, monitoring and results reporting department to support all academic staff across the University.

Between 1999 and 2005 the wide ranging functionality provided by the system attracted the interest from a number of external organisations including pilot e-assessments for schools’ examination bodies (e.g.Maughan.S & Mackenzie.D., 2004) (http://caaconference.co.uk/pastConferences/2004/proceedings/Maughan_Mackenzie.pdf) and for parts of the UK National Health Service.

In 2005, the external interest in both e-assessment and e-learning prompted a move of development staff from the University’s e-learning unit and the core TRIADS development team into a new commercial e-learning development department of the University (Innovation for Learning, IfL (http://www.i4learn.co.uk) ). Production of e-assessments for external organisations was then concentrated in the wider commercial arm of the University. Competency assurance and continuing professional development assessments were developed by the team for medical staff in a range of NHS hospitals.

From 2011 on the TRIAD System has been developed and maintained by Prof. Don Mackenzie at Professional e-Assessment Services which continues to support TRIADS e-assessment clients of the University and other clients largely in the field of competency assessment of medical professionals up to Consultant level.

Functionality

All versions of DIADS and TRIADS are open-coded in Authorware Professional (previously from Macromedia now © Adobe Systems). Authorware was initially selected because of its intuitive and rapid development environment for the production of a wide range of interactions suitable for e-learning course-ware.

Over time and use in a wide range of applications, the limitations of the standard Authorware interaction controls became apparent and a number of interactions were re-written using a combination of the Authorware scripting language and JavaScript in order to give finer control on question behaviour, response checking and scoring.

TRIADS can deliver all the question types listed below individually with a variety of controls on their behaviour and scoring. Additionally TRIADS can deliver text/numeric entry + checkbox matrix + hotspot + draw line interactions concurrently within a single question. This concurrent functionality can potentially be extended to all interaction types, although it is difficult to imagine a scenario that would require all of them at the moment.

All interactions can be layered and overlain on graphics

Code can be added to all question types to account for specialist scoring requirements and the system may incorporate biological and petrological microscope simulators if these are required for specialist assessments using these tools. The way in which these tools are used to answer a question can also be scored, if required.

The question display engine can be configured in a wide variety of modes which can include controlled random question delivery, menu option links to different question sets and the optional delivery of context-sensitive feedback as an immediate response to a user action or at the end of each question. Feedback can be held internally or externally in .rtf files or as links to web pages.

Assessments may be Browser-delivered for intranet, internet and LMS via a plug-in or compiled as an executable for CD or LAN delivery.

Advantages

Whilst standard e-assessments can be produced and delivered using TRIADS, it comes into its own when producing simulation and scenario-based assessments where interactions are overlain on graphics and require a high degree of control on their behaviour and scoring.

TRIADS assessments are easily integrated into Authorware-based e-learning packages.

TRIADS questions are ideal for binding seamlessly into an e-learning narrative to generate a dialogue between the user and the e-tutoring element whilst tracking the scores.

There are very few limitations on the complexity of the assessment or nature of the questions that can be delivered using TRIADS.

General Limitations

Because of its wide ranging functionality, TRIADS is very much a tool for specialist Assessment Developers in commercial or university-based e-learning/assessment development departments.

It is not generally suitable for use by most individual tutors/assessment designers on an occasional basis. This was demonstrated in the TRIADS Assessment of Learning Outcomes project when usage of TRIADS fell significantly after project funding supporting developers in each institution ended.

TRIADS is open-coded in Authorware Professional (© Adobe Systems) and is thus vulnerable to commercial decisions made by Adobe. Authorware is still currently (2013) available from Adobe but is regarded as a ‘mature’ product and will not be developed further. Thus its longevity is in question. Re-development of a system as sophisticated as TRIADS using any other development tool will be very expensive and currently there appears to be no rapid development environment that gives the flexibility of delivery mode and level of interactivity provided by Authorware in a single package.

Web delivery of software developed using Authorware is provided via the Authorware Web Player 2004 plug-in. This works well, once installed, but its installation requires ‘Administrator’ security rights on client machines together with a rather specific browser setup. This creates barriers for individual users and difficulties in distribution across some secure corporate/institutional networks.

TRIADS is developed for Windows platforms only.

Caution

TRIADS has an unparalleled and innovatory range of assessment styles and any applicant who seeks to patent any kind of question delivery mechanism would need to ensure that their application did not infringe upon or plagiarise the work already done by the TRIADS team.

The TRIADSystem is © University of Derby and Prof. Don Mackenzie

 

 

Some related references

Boyle, D.N. Bryon, C.R.C. Paul (1997) Computer-based learning and assessment: A palaeontological case study with outcomes and implications . Computers & Geosciences, Volume 23, Issue 5, June 1997, Pages 573-580 – abstract here

Mackenzie, D.M. (1997) Computer Aided Assessment at the University of Derby. A case study in: Brown, G., Bull, J. and Pendlebury,M. Assessing Student Learning in Higher Education. Chapter 13 pp 215-217. Publ. Routledge, London.

Mackenzie, D.M. (1997) Fully Interactive Assessment on the Web – A Shock to the System. Proceedings of the First Annual Computer Assisted Assessment Conference. 18th. June at the University of Loughborough.

Mackenzie, D.M., Regan, P.F, Wilkins, H. & Hutchinson, P (1998) Integrating Fully Interactive Assessments with Web Resources – A flexible learning environment for students and an easier life for academics?. Geological Society of America Annual Conference, Toronto. Abstracts p.A-389.

Mackenzie, D.M. (1999) Recent Developments in the Tripartite Interactive Assessment Delivery System. Proceedings of the Third Annual Computer Assisted Assessment Conference. University of Loughborough. 16th-17th June

Mackenzie, D.M., Wilkins, H., O’Hare, D. and Boyle, A. (1999) Practical Implementation of Recent Developments in the Tripartite Interactive Assessment Delivery System. A Workshop for the Third Annual Computer Assisted Assessment Conference. University of Loughborough. 16th-17th June

Boyle, A.P., Edwards, D.J., Mackenzie,D.M., Mills, B., O’Hare, D., Morris, E.C., Paul, C.R.C.P., Wilkins, H. and Williams, D.W. (2000) Developments in on-line assessment – experiences and evaluation of using TRIADS and its potential for assessment in electrical engineering. International Journal of Electrical Engineering Education , vol 37, part 1, pp 74-85

Mackenzie, D.M., O’Hare,D., Paul,C., Boyle,A., Edwards, D., Williams,D. & Wilkins, H. (2004) Assessment for Learning: the TRIADS Assessment of Learning Outcomes Project and the development of a pedagogically friendly computer based assessment system. In O’Hare,D & Mackenzie, D.M. (Eds) Advances in Computer Aided Assessment, SEDA Paper 116 pp11-24. Staff and Educational Development Association Ltd., Birmingham.

Maughan,S & Mackenzie,D.M. (2004) BioScope: The Assessment of Process and Outcomes using the TRIADSystem. Proceedings of the 8th International CAA Conference, Loughborough, 6th & 7th July 2004 ISBN 0-9539572-3-3

Mackenzie, D.M. (2005) Online Assessment: quality production and
delivery for higher education. Keynote Address in Enhancing Practice, Assessment Workshop Series No. 5 in Reflections on Assessment, Volume II. pp22-29, Quality Assurance Agency for Higher Education, Gloucester. ISBN 1 84482 266 4