This Deliverable provides a report of the deployment and evaluation activities and outcomes during the final year of the CS-AWARE project. The main objective of the CS-AWARE project (https://cs-aware.eu/) is to provide a cybersecurity situational awareness software solution for Local Public Administrations (LPA’s), small- to medium-sized enterprises (SME’s) and financial institutions. This solution enables the detection, classification and visualisation of cybersecurity incidents in real-time, supporting the prevention or mitigation of cyber-attacks. It is intended that this solution will provide an intermediary step towards automation of cyber incident detection, classification and visualisation, that impacts on cybersecurity awareness of users. User involvement is crucial in all steps of design, implementation, deployment and evaluation.
Deployment can be defined as locally implementing the system and making sure that the system functions in the two LPA contexts. Like everything else in this project, the nature of this work is socio-technical. This means that ongoing feedback of users is a crucial aspect, as well as testing the technology in the authentic user context. We collected user feedback and tested the technology during three cycles of three months. For each cycle, we imple- mented user feedback as an ongoing process until the very end of the project, this is called agile programming and software development. At the start, we established the user needs and ambitions in a deployment scenario. This served to shape our deployment and evaluation activities.
For evaluation, the main qualitative input came from frequent usability testing and interactions with users from the pilot deployment teams. Through these flexible methods, we gathered valuable information about using the system and refinement of its functionalities. In combination with the agile approach, feedback could be implemented quickly. Through the use of questionnaires, we got informed about how users rated the system, in terms of general characteristics, usability, their awareness of cybersecurity, and the impact on the organisation. Our attempts to get responses from a fifth questionnaire, for potential new users in other municipalities failed to receive any response, most probably because these users were focusing on the corona pandemic rather than on cybersecurity. Together, the questionnaire scores were very positive, which confirmed the KPIs set for our system.
At the end of deployment, users successfully completed the four exercises that were part of the final usability test. This means that our users, and very probably also users in other municipalities and professional contexts, with similar motivation and experience, can work with the CS-AWARE system to perceive, comprehend, project and mitigate cybersecurity threats. Moreover, we have shown that working with CS-AWARE increases awareness of cybersecurity, this applies to individual users, such as system administrators but also to greater awareness of cybersecurity within their organisation. This is the basis for realising the objectives that users formulated in the deployment scenario: efficient detection of cyberthreats, and better communication and collaboration between departments, between system administrators and managers, and between departments and citizen users of services. Better collaboration, learning, and increased reputation can be built on this basis.
