Today I want to talk a bit about the effectiveness of the system and dependency analysis methodology we have chosen in the CS-AWARE project: the soft systems methodology. More specifically, I would like to talk about the first phase of the methodology, the information gathering phase that is done mainly by having the users of the system expressing their knowledge about the system and the problems they have with the system in a workshop setting by drawing rich pictures. The discussions among the workshop participants based on the pictures usually provides an environment that gradually, over several workshop sessions, creates a holistic understanding of the systems and the socio-technological interactions. It is understood that in complex environments this type of analysis is more effective and leads to more accurate analysis results when compared to classical analysis methods like an analyst trying to gain an understanding based on system documents, presentations and interviews.
In CS-AWARE, the system and dependency analysis of each organizational system where the CS-AWARE tool is deployed is an integral part of the solution – CS-AWARE will not work without a good understanding of the socio-technological systems and dependencies, and an understanding of how we can monitor them. Only then we can set the cybersecurity situation of each individual organization in context with the wider world and provide effective cybersecurity situational awareness. One could even say that the system and dependency analysis is what makes and breaks CS-AWARE. So we want to make absolutely sure that the information gathering and our holistic understanding of the pilot systems is as accurate as possible.
So how does our approach work in practice? We have by now conducted the first of three rounds of user workshops at our piloting partners. We have seen that, if the participants of the user workshops have prepared themselves and have understood the added value of system analysis using rich pictures, this method is a powerful tool to quickly gain a common understanding of the systems and interactions, from a high level overview down to a detailed technical understanding. The right composition of participants in the user workshops is crucial. Only if representatives from all relevant organizational levels (such as managers and technicians) are present in the workshops, a complete and holistic understanding can be achieved. We have seen that it is unavoidable to have stable workshop groups – those who decide to be part of the workshop need to be there for the whole duration of the analysis. In a situation where participants come and go based on their conception of when they are needed, two things will happen: First, information will be missing because the right person was not present to contribute their knowledge at the right time and second, information will be reiterated several times because a participant has not been present when something was already discussed. Both slow down the progress of the analysis considerably. We have also noticed that if the workshops are conducted in a cultural setting that does not facilitate participatory workshops, the participants do not necessarily see the benefit of this type of analysis. The willingness to engage and contribute to an interactive analysis that requires constant input and interaction from the participants is restrained. It can go as far as the participants start to question their role in the workshop and stop to engage completely, which of course means that the information gathering is not as effective as it could be.
To summarize, we are quite happy with the results of the first round of system and dependency analysis user workshops. In some aspects we achieved much better results than we had expected, while in others we did not yet achieve the expected results. We are confident however that with some tweaks to our approach based on the experiences we have gained so far, we will achieve the analysis results we expect for CS-AWARE during the second round of workshops scheduled next year. We are certain that with the soft systems approach we have chosen the right approach for the analysis of the complex environments we are dealing with in CS-AWARE – we can’t think of any other method that would be better suited for this task!
Thomas Schaberreiter
University of Vienna, Austria
