Agile Evaluation Methods for Intelligent Transport Systems – Definitions of Agile Characteristics and Assessment Criteria

Information on the technical functioning, impacts, interoperability and acceptance of ITS systems is crucial to making decisions on the deployment of these systems. The resources, time and amount of data available for evaluation studies of a potential ITS system are often more or less limited. A need for more agile evaluation methods has been identified.

 This study provides definitions for different agile characteristics of evaluation methods for ITS systems, measurable criteria for these characteristics, and a definition for an agile evaluation method. The agile characteristics were defined on the basis of a literature study taking into account the Agile Manifesto, literature on measuring the agility of software development methods, and the experience obtained from seven case studies. The agile characteristics found to be relevant for evaluation methods for ITS include flexibility, leanness, lightness and simplicity, responsiveness, learning, speed and sustainability. The descriptions of agile characteristics, criteria for measuring the agility of evaluation methods and the definition of an agile evaluation method were validated by applying them to seven case studies. The case studies involved evaluations of technical functioning, impacts, socio-economic benefits, interoperability and user acceptance of ITS systems. Of the seven evaluated case studies, five were found to be based on agile methods as such or with restrictions. The study results indicated that agile methods for evaluating the impacts of ITS have many similar characteristics to expert assessment, such as limited requirements for time and resources. However, they can be expected to be more transparent and possibly also more accurate than expert assessment without supporting documentation. Agile methods for evaluating the technical functioning of ITS systems, based on observation of system behaviour, analysis of reliability and statistical analysis of study results, can be used to verify the operation of an ITS system or subsystem in realistic operating conditions. They can be applied also in situations in which no extensive up-to-date documentation on system requirements is produced or maintained, such as when an agile development model is used for developing an ITS system.

The study results indicate that not all simple or simplistic evaluation methods are agile methods, as simplicity does not imply that the evaluation method has other essential characteristics of agility such as flexibility and leanness. More validation will likely be needed for descriptions of the agile characteristics of evaluation methods, for the criteria for agility, and for the definition of an agile evaluation method. Further analyses should also be carried out on how to employ agile development, testing and systems engineering methods in the development of ITS systems in an optimum way, and how agile evaluation methods could be connected to them.