Applying evidence of threat to support architecture design and change

As with the evidence of risk, threat evidence is also subjective and therefore requires a similar development through methods, analysis and qualification. So what do we mean by threat and how does threat manifest itself. I like the definition of threat by NIST Federal Information and Information Systems:

Any circumstance or event with the potential to adversely impact organizational operations (including mission, functions, image, or reputation), organizational assets, or individuals through an information system via unauthorized access, destruction, disclosure, modification of information, and/or denial of service. Also, the potential for a threat-source to successfully exploit a particular information system vulnerability.

So evidence is required to understand what is the potential of the threat or threats to an organisation and to understand the business impact. It will be evident from investigations and understanding the root cause of incidents as well as any reports form logs, rule or signature based security technology that attacks are happening but understanding who and why is a lot harder to determine.

There are a long list of threat actors who pose a threat, some more sophisticated and more capable than others but they are all a risk. Each threat actor will have a particular motive and target, whether that is financial, espionage or sabotage and use different methods and approaches to reach their goal.  The table below covers the common threat actors and background to their activities.


Hobbyist Predominantly lower skilled individuals motivated by curiosity, fun or peer groups.
Corporate Espionage Some organisations may use hacking methods or employ others to attempt to steal Intellectual Property or information to gain competitive market advantage
Investigative Journalist Journalists with a remit to investigate will use less scrupulous means to obtain privileged information.
Malware authors Malware authors may provide malware to other threat actors or attempt to release malware into an organisation to steal information or disrupt a service for other gains.
Privileged Users – Internal, Suppliers (e.g. developers, service provider) Individuals with the privilege of being an authorised user may exploit this position to commit fraud, steal Intellectual Property or cause damage.
BlackHat (hacker) / Criminal Gangs Lone or small groups of experienced hackers may seek to gain illicit access for self gain or to forward stolen information to others. They are seen to possess higher levels of both skills and motivation than hobbyists.

There are two key types of threat; a general threat where any compromise and exploitation of an asset will be of value to a threat actor to either maintain a backdoor for use a a future stage or to sell the access to another threat actor; the second and most important is the specific threat where a threat actor has a particular motivation.

Apart from a privileged user, it is difficult to identify the specific threat actor and their motivations, that can only be speculated at as the evidence and information is built up. So it is easier to align threat actors types to particular critical assets within the organisation as a means to develop threat evidence. For example, personal credit card data or research into a new drug may be of interest to different threat actors and as such will have different patterns of evidence that there is or is not a threat to these assets.

If we refer back to the Evidence model the focus begins with Evidence Development and the Hypothesis. The Evidence Development will help determine the right approach to go about gathering the information and data to form the evidence to support any hypothesis. Evidence of a threat my be suspicious activity, failed attempts to compromise a network or the discovery of malware. These events may be unrelated however they may begin to support the arguments and assertions that a threat is real and an attack likely. It may require many months of gathering the right information and evidence to result in the recommendations to make changes to architecture. The threats will remain as long as the assets are of interest to threat actors and they will change their methods and approaches as well as remain persistent.



Building an Evidence Ontology

I have already highlighted in several earlier posts the aspects and importance of evidence but i am going to revisit particular key messages to explain the benefits from formalising the structure of evidence in order for it to support and improve EA knowledge.

  • The evidence ontology needs to be able to bring together a variety of research to strengthen hypotheses and recommendations in the artefacts and methods used in the architecture development method.
  • The evidence ontology needs to be able to support the internal EA content behind new views and scenarios as well as incorporate evidence from external sources.
  • Finally the evidence ontology needs to be able to support the formation of an evidence knowledge base that is flexible to help the engineering of future organisation models.
  • The evidence ontology needs to be able to enable the cross referencing required to demonstrate the importance of pattern and knowledge rules

You may be asking what is the point of building up a model of evidence to such a degree as an ontology. The main reason is that it is a relatively simple model and by doing so you are able to decide upon what constitutes evidence, how you characterise it and how you build and approve it. Otherwise any supporting evidence looses it importance if it does not have foundation. The reason i choose to use Cyber Defence evidence as an example it that the lessons and knowledge gained from understanding threats, risk and vulnerabilities in your organisation is vital to your architectural development now and in the future. That is not to say that evidence in Marketing, Sales or Product / Service development is less important, it is not, as any evidence used to support decision making is vital to develop and sustain a business.

So back to the Evidence Ontology and a proposed high level structure. at the top level the Evidence Ontology looks as follows:

  • Evidence research
  • Evidence development
  • Evidence synthesis
  • Evidence evaluation
  • Hypothesis
  • Evidence monograph
  • Evidence attributes

Each of these classes is further defined by sub classes as demonstrated in the attached image file. I have collected these classes from various sources of information and they represent my view of the necessary requirements to cover the process, classification, evaluation and provision of evidence material to support any decision based activity such as planning, strategy and enterprise architecture. I will continue in the next post with a look at the properties and relationships to the decision based activities through examples.

Evidence Ontology
Evidence Ontology

What the evidence ontology should support.

In a nutshell an enterprise architecture is a collection of facts about the organisation’s current architecture, a development methodology supported by a collection of artefacts and a model of the organisation’s future architecture. On top of that there are the capabilities of the people, processes and technologies that make up the organisation that defines what it is and what it does. An evidence based approach to enterprise architecture requires evidence in a variety of types and to varying degrees of detail to support all of the above.

The current architecture is primarily a collection of facts that have been built up, depending upon how recently the organisation has adopted enterprise architecture, under varying amounts of decision making and control. Information can and should exist about how the current architecture is operating and performing and to what level of satisfaction and success. This information can be formulated to represent existing evidence and therefore provide support for future decisions about change.

The future model of the organisation is a set of concepts that offers a new view of the organisation depending upon the scenarios it envisages. The model and concepts will evolve and impact on that evolution. The degree of possible change will depend upon how far in the future the model is set, the maturity of the strategies and organisational decisions. there will also be unexpected impacts on the organisation and on the strategic direction. This future model is dependent upon an architectural development method to deliver the new version of the organisation as well as the capabilities and tools to support it. So it is important that the evidence ontology is able to support the following principles:

  • The evidence ontology needs to be able to support the internal content behind new views and scenarios as well as incorporate evidence from external sources.
  • The evidence ontology needs to be able to bring together a variety of research to strengthen hypotheses and recommendations in the artefacts and methods used in the architecture development method.
  • Finally the evidence ontology needs to be able to support the formation of an evidence knowledge base that is flexible to help the engineering of future organisation models.

Evidence ontology – a high level overview of a model to support evidence based enterprise architecture

Evidence Ontology Concepts

The model attached to this post is a strawman collection of concepts i am evolving to represent an evidence ontology. The concepts I am exploring are:

  • Evidence research
  • Evidence development
  • Evidence synthesis
  • Evidence evaluation
  • Hypothesis
  • Evidence monograph
  • Evidence attributes

The role of an enterprise architecture repository – principle 1

Enterprise Architecture Management System
Enterprise Architecture Management System

The Enterprise Architecture Repository is both a means to store all of the artefacts concerning the enterprise architecture and a federated information system linking with other sources of data and c0ntent. This federated environment produces a enterprise architecture management system to support architectural development. There is quite an interesting overview of this on the Aris Community blog.

Along side the standard features of the repository, to hold the information pertaining to the AsIs and ToBe models, is the need for the system to support research development and evidence management. The enterprise architecture is represented by a collection of facts and statements about the enterprise. They are collected in a set of artefacts and are used through the architecture development methodology to build future versions of the enterprise. It is the change from the current state of the enterprise to a new state that evidence is required and applied to support change decisions. The evidence that is used and cited within artefacts it should be stored within the architecture repository. This enables the evidence to analysed and tracked through the lifetime of the decision.

The evidence repository is a subset of the EA repository or federated knowledge base that has defined relationships that link the evidence to the artefact. This can be achieved through simple hyperlinks but a defined approach with specific artefact metadata and unique identifiers would offer a structured relationship. This should also encompass the evidence metadata to provide the attributes to support tracking such as the value, confidence and temporal attributes.

Whilst it is important to store evidence cited within artefacts within the EA repository it is also important to store research and information to act as future evidence. This form of evidence should be built with a formal process and should be collected in line with future views of the organisation. This type of evidence can include horizon scanning of new technologies, customer or user opinions and surveys, planned legislative changes or innovations.

I have made this principle 1 because i consider an enterprise architecture repository a fundamental part of enterprise architecture and building up and storing evidence is a vital part of evidence based enterprise architecture.

Principles of evidence based enterprise architecture

These are a set of principles specifically for evidence based enterprise architecture and would follow on from those I have described in the earlier post on evidence based design.

  1. An organised enterprise architecture framework and repository must be used to provide a model of how the organisation defines enterprise architecture and a store for enterprise architecture artefacts.
  2. The enterprise architecture framework must define the development method used in practice and all of the artefacts used in the process should include evidence where appropriate.
  3. The production of all enterprise architecture artefacts must follow a controlled authoring and management process. All artefacts must have appropriate metadata and be stored in a document management system. There must also be a review process to ensure consultation and validation of content and evidence.
  4. The production of enterprise architecture documents should adopt a formal content authoring framework such as DITA – Darwin Information Typing Architecture (OASIS standard) to help structure the content and allow for reuse.
  5. A controlled vocabulary should be used to minimise ambiguity and support information relationships within the repository. The Universal Data Element Framework (UDEF) should be considered as a way to classify and categorise terms used in documentation.
  6. There must be a controlled evidence model to define how to classify evidence and evidence value. The evidence model must at least be able to define the following evidence attributes:
    – Value 
    – Sources
    – Credibility
    – Expertise
    – Impartiality
  7. All architects should be trained on how to perform evidence research, evidence analysis and how to incorporate evidence where appropriate into artefacts to support any hypotheses, propositions and recommendations.
  8. Any evidence used in an artefact must be clearly marked and identifiable to help traceability, tracking and analysis in the future.

These are just a few I have identified and I will probably add to this list and define each one in more detail in future posts.

What do we want from evidence?

Monastery of Hosios Loukas
Monastery of Hosios Loukas

Before i begin to build up a model of what evidence is and what it means for evidence based enterprise architecture i want to consider what we want from evidence. Simply put evidence is something that helps us make a decision, draw a conclusion to approve or disprove something and allow us to move forward. The sources of evidence are numerous i.e. data, statistics, research findings, expert knowledge, observations and verbal facts.

The following questions are a set i have gathered that i think set out some of the issues that the evidence ontology will need to support.

  • How is evidence used in enterprise architectural work and how does it inform architectural design?
  • How is evidence produced and how does evidence influence the architectural development?
  • What are the core methods, skills, and values needed to do evidence-based enterprise architecture or to produce evidence through architectural practice?
  • Does the use of evidence inhibit or enhance the decision making?
  • How does collaboration help to produce and evaluate evidence?
  • How much evidence is enough and what makes it credible?
  • How are the outcomes of work translated so that they can be generalized and used by others?

The other consideration is the need to support the differing evidence requirements at each stage of a decision process. This could include evidence to support the following:

  • Supporting the goal and success criteria
  • Supporting alternatives or options
  • The effects of the decisions on the end goal
  • The effects of the alternatives or options
  • How to avoid evidence overload by providing precision at the information gathering stages
  • How to help consolidate the evidence gathered through ranking or other method to effect the decision