All architects should be trained on how to perform evidence research and evidence analysis – principle 7


It was only after working for a research institute that i began to fully appreciate the capabilities and time required when conducting research for evidence. This institute employed research librarians, research analysts and specialist researchers to support the teams building clinical evidence. These individuals collectively located evidence, analysed it and stored and maintained it for use in several large knowledge databases.

In order to apply evidence based enterprise architecture, Architects should at least be trained on the basic techniques of research, analysing the information they find and how to apply it. This should include setting the objectives of the research, defining the questions to answer, undertaking literature reviews and collecting the information for analysis. Finally, identifying the appropriate evidence required to support their hypothesis and conveying that through their documentation.

I don’t however expect Architects to become experts in this area. The architecture practice within an organisation should make a judgement on how it develops this capability and whether or not it requires specialist resources.

Using an evidence model to define how an organisation classifies evidence and evidence value – principle 6


During my research into the subject of evidence based practices and how to apply the approach to enterprise architecture i have only come across three models or metamodels associated with evidence. The first comes from a publication called Enterprise Architecture – Models and Analysis for Information Systems Decision Making by Pontus Johnson and Mathias Ekstedt (a very good book and i recommend reading it) the second and third are models from the OMG website on the subjects of Software Assurance Evidence Metamodel (SAEM) and Argument Metamodel (ARM). Non of these models fulfil all of the requirements i was looking for as a model to support evidence based enterprise architecture however they provide useful associations. The two models from OMG are software related and thus would be useful in the area of software development. The model from the publication is a high level model giving the main subject areas.

For this principle I recommend that an organisation produces an evidence model to define how they classify evidence, value it, analyse it and apply it. In future posts i will offer descriptions of the model i have been developing.

The role of an enterprise architecture repository – principle 1


Enterprise Architecture Management System
Enterprise Architecture Management System

The Enterprise Architecture Repository is both a means to store all of the artefacts concerning the enterprise architecture and a federated information system linking with other sources of data and c0ntent. This federated environment produces a enterprise architecture management system to support architectural development. There is quite an interesting overview of this on the Aris Community blog.

Along side the standard features of the repository, to hold the information pertaining to the AsIs and ToBe models, is the need for the system to support research development and evidence management. The enterprise architecture is represented by a collection of facts and statements about the enterprise. They are collected in a set of artefacts and are used through the architecture development methodology to build future versions of the enterprise. It is the change from the current state of the enterprise to a new state that evidence is required and applied to support change decisions. The evidence that is used and cited within artefacts it should be stored within the architecture repository. This enables the evidence to analysed and tracked through the lifetime of the decision.

The evidence repository is a subset of the EA repository or federated knowledge base that has defined relationships that link the evidence to the artefact. This can be achieved through simple hyperlinks but a defined approach with specific artefact metadata and unique identifiers would offer a structured relationship. This should also encompass the evidence metadata to provide the attributes to support tracking such as the value, confidence and temporal attributes.

Whilst it is important to store evidence cited within artefacts within the EA repository it is also important to store research and information to act as future evidence. This form of evidence should be built with a formal process and should be collected in line with future views of the organisation. This type of evidence can include horizon scanning of new technologies, customer or user opinions and surveys, planned legislative changes or innovations.

I have made this principle 1 because i consider an enterprise architecture repository a fundamental part of enterprise architecture and building up and storing evidence is a vital part of evidence based enterprise architecture.

Principles of evidence based enterprise architecture


These are a set of principles specifically for evidence based enterprise architecture and would follow on from those I have described in the earlier post on evidence based design.

  1. An organised enterprise architecture framework and repository must be used to provide a model of how the organisation defines enterprise architecture and a store for enterprise architecture artefacts.
  2. The enterprise architecture framework must define the development method used in practice and all of the artefacts used in the process should include evidence where appropriate.
  3. The production of all enterprise architecture artefacts must follow a controlled authoring and management process. All artefacts must have appropriate metadata and be stored in a document management system. There must also be a review process to ensure consultation and validation of content and evidence.
  4. The production of enterprise architecture documents should adopt a formal content authoring framework such as DITA – Darwin Information Typing Architecture (OASIS standard) to help structure the content and allow for reuse.
  5. A controlled vocabulary should be used to minimise ambiguity and support information relationships within the repository. The Universal Data Element Framework (UDEF) should be considered as a way to classify and categorise terms used in documentation.
  6. There must be a controlled evidence model to define how to classify evidence and evidence value. The evidence model must at least be able to define the following evidence attributes:
    – Value 
    – Sources
    – Credibility
    – Expertise
    – Impartiality
  7. All architects should be trained on how to perform evidence research, evidence analysis and how to incorporate evidence where appropriate into artefacts to support any hypotheses, propositions and recommendations.
  8. Any evidence used in an artefact must be clearly marked and identifiable to help traceability, tracking and analysis in the future.

These are just a few I have identified and I will probably add to this list and define each one in more detail in future posts.

Principles of evidence based practice


Before i expand on the principles of evidence based enterprise architecture it is important to highlight the work behind evidence based practice and in particular the wikipedia page on the subject. Evidence based enterprise architecture is just another evidence based practice and as such it shares all the common features and benefits. Evidence based practices now cover a wide range of subjects such as health, education, management and design. The wikipedia page offers insights into methods and strategies as well as background information. I want to cite three sections from the wikipedia article as i think they set out a good foundation and I will then adapt these to fit the context of EBEA.

Source: Wikipedia Evidence Based Design

Methodology:

In the first instance EBD methodology could be divided in four subsequent steps:

  • reviewing existing research literature to select significant findings and recommendations;
  • matching referenced findings with data gathered from site visits, surveys results, subject matter experts;
  • prediciting the outcomes of design decisions;
  • tracking the positive outcomes for design implementation.

Meta-analysis template

In his book “Evidence-based Policy: A Realistic Perspective”, Ray Pawson (2006) suggests a meta-analysis template that may be applicable for EBD. By using such a protocol the field will be able to provide designers with a credible source for evidence-based design. Systematic review process should follow six essential steps:

1. Formulating the review question
2. Identifying and collecting evidence
3. Appraising the quality of the evidence
4. Extracting and processing the data
5. Systematizing the data
6. Disseminating the findings

Conceptual model for the application of Evidence Based Design

Hamilton (“Four Levels of Evidence-Based Practice”, The AIA Journal of Architecture, 2006

  • Level 1
    • analysing the literature in the field in order to follow the related environmental researches
    • reading the meaning of the evidence in the relationships to the project
  • Level 2
    • foreshadowing the expected outcomes of design decisions upon the general readings
    • measuring the results through the analysis of the implications, the construction of a chain of logic connection from decision and future outcome, in order to reduce arbitrary decisions
  • Level 3
    • reporting the results publicly, writing or speaking about results, and moving in this way information beyond design team
    • subjecting methods and results to others who may or may not agree with the findings
  • Level 4
    • publishing the findings in reviewed journals
    • collaborating with academic or social scientists

I think it is very clear from the quotations above that evidence based practice can be applied to a range of activities within enterprise architecture using these core principles as a guide.

Supporting alternative decisions


The alternative options are often dependent upon the type of goal an organisation is aiming to reach as well as the situation the decision maker is in. Here perhaps is an important area for evidence either to prove that the original decision is the best choice or show that an alternative is better. Alternative decisions in enterprise architecture often centre around technology choices, which product is the best fit and stands out from the others that have been considered. Product evaluation can be a lengthy process and can primarily be based upon a set of requirements issued to each vendor. The vendor which best meets the requirements and provides a cost within budget is most likely to be the winning contender. Yet does the evidence show that the product and the vendor is the best choice.

The evidence framework to support the decision making process behind this type of decision should depend upon the size and impact of the strategy or goal. If the goal is to introduce a new organisation wide application then the evidence framework needs to be broad enough to enable detailed questioning of key capabilities and situations. Using the example of a important goal of purchasing a new organisation wide application, which will impact a large number of employees and departments, then evidence should be gathered on the following:

  • The current situation of the vendor; its financial position; company size and number of clients; expertise of key staff or board members; company locations – nationally and internationally; history of the vendor and its relationships with other organisations, partners, standards bodies and associations.
  • The vendor’s product and its position within its product portfolio; history of the product; awards; key clients, case studies and reviews by organisations such as Gartner, Ovum or Forrester.
  • How has the product been received by other customers; the trade press and internet forums.
  • The development process behind the product; who are the key designers and innovators behind the product; the vendor’s product position at trade conferences – how often and at which conferences does the vendor exhibit and make presentations?
  • Product capability and technical requirements; documentation and training programmes
  • Product support and service management capability; field staff numbers and support lead times.

This list is just a set of examples of topics that should be gathered as an evidence framework for each vendor and then compared against the enterprise architecture and at each architectural level. This is to determine how each alternative product will fit within the architecture and its impact on existing structure. This may seem like a lot of effort to gather all of this information but this type of evidence gathering my highlight that an alternative vendor to the main contender is a better fit for the organisation even if it means a higher initial cost. A decision based on requirement fit and cost alone may turn out to be a bad decision in the long term.