Principles of evidence based practice


Before i expand on the principles of evidence based enterprise architecture it is important to highlight the work behind evidence based practice and in particular the wikipedia page on the subject. Evidence based enterprise architecture is just another evidence based practice and as such it shares all the common features and benefits. Evidence based practices now cover a wide range of subjects such as health, education, management and design. The wikipedia page offers insights into methods and strategies as well as background information. I want to cite three sections from the wikipedia article as i think they set out a good foundation and I will then adapt these to fit the context of EBEA.

Source: Wikipedia Evidence Based Design

Methodology:

In the first instance EBD methodology could be divided in four subsequent steps:

  • reviewing existing research literature to select significant findings and recommendations;
  • matching referenced findings with data gathered from site visits, surveys results, subject matter experts;
  • prediciting the outcomes of design decisions;
  • tracking the positive outcomes for design implementation.

Meta-analysis template

In his book “Evidence-based Policy: A Realistic Perspective”, Ray Pawson (2006) suggests a meta-analysis template that may be applicable for EBD. By using such a protocol the field will be able to provide designers with a credible source for evidence-based design. Systematic review process should follow six essential steps:

1. Formulating the review question
2. Identifying and collecting evidence
3. Appraising the quality of the evidence
4. Extracting and processing the data
5. Systematizing the data
6. Disseminating the findings

Conceptual model for the application of Evidence Based Design

Hamilton (“Four Levels of Evidence-Based Practice”, The AIA Journal of Architecture, 2006

  • Level 1
    • analysing the literature in the field in order to follow the related environmental researches
    • reading the meaning of the evidence in the relationships to the project
  • Level 2
    • foreshadowing the expected outcomes of design decisions upon the general readings
    • measuring the results through the analysis of the implications, the construction of a chain of logic connection from decision and future outcome, in order to reduce arbitrary decisions
  • Level 3
    • reporting the results publicly, writing or speaking about results, and moving in this way information beyond design team
    • subjecting methods and results to others who may or may not agree with the findings
  • Level 4
    • publishing the findings in reviewed journals
    • collaborating with academic or social scientists

I think it is very clear from the quotations above that evidence based practice can be applied to a range of activities within enterprise architecture using these core principles as a guide.

Supporting alternative decisions


The alternative options are often dependent upon the type of goal an organisation is aiming to reach as well as the situation the decision maker is in. Here perhaps is an important area for evidence either to prove that the original decision is the best choice or show that an alternative is better. Alternative decisions in enterprise architecture often centre around technology choices, which product is the best fit and stands out from the others that have been considered. Product evaluation can be a lengthy process and can primarily be based upon a set of requirements issued to each vendor. The vendor which best meets the requirements and provides a cost within budget is most likely to be the winning contender. Yet does the evidence show that the product and the vendor is the best choice.

The evidence framework to support the decision making process behind this type of decision should depend upon the size and impact of the strategy or goal. If the goal is to introduce a new organisation wide application then the evidence framework needs to be broad enough to enable detailed questioning of key capabilities and situations. Using the example of a important goal of purchasing a new organisation wide application, which will impact a large number of employees and departments, then evidence should be gathered on the following:

  • The current situation of the vendor; its financial position; company size and number of clients; expertise of key staff or board members; company locations – nationally and internationally; history of the vendor and its relationships with other organisations, partners, standards bodies and associations.
  • The vendor’s product and its position within its product portfolio; history of the product; awards; key clients, case studies and reviews by organisations such as Gartner, Ovum or Forrester.
  • How has the product been received by other customers; the trade press and internet forums.
  • The development process behind the product; who are the key designers and innovators behind the product; the vendor’s product position at trade conferences – how often and at which conferences does the vendor exhibit and make presentations?
  • Product capability and technical requirements; documentation and training programmes
  • Product support and service management capability; field staff numbers and support lead times.

This list is just a set of examples of topics that should be gathered as an evidence framework for each vendor and then compared against the enterprise architecture and at each architectural level. This is to determine how each alternative product will fit within the architecture and its impact on existing structure. This may seem like a lot of effort to gather all of this information but this type of evidence gathering my highlight that an alternative vendor to the main contender is a better fit for the organisation even if it means a higher initial cost. A decision based on requirement fit and cost alone may turn out to be a bad decision in the long term.

Evidence behind organisation goals


Goals exist at every level of an organisation and it is important that behind each goal is sufficient evidence that the target of the goal is the right decision and that it is achievable. It is often hard to measure the effects of decisions on very long term enterprise level goals unless there is a dedicated team monitoring from beginning to end.

At an enterprise level a long term goal may be dependent upon the success of many subgoals therefore creating complex interdependent relationships. As the path to reaching the goal develops the effects of the decision stages should be assessed as well as relevance and value of the underlying evidence.

The evidence used to help the decisions behind long term enterprise goals may be in several forms. Data based evidence includes such things as statistical reports, business intelligence, trends, financial projections and forecasts. Content based evidence generally includes internal reports, externally published and purchased reports from leading organisations such as Gartner, Ovum, Mckinsey etc. competitive analysis and recommendations from internal or external experts.

It is likely that an organisation will spend considerable money, time and effort assembling and analysing the evidence before decisions are made. It is therefore important to track and monitor the value of the evidence against the benefits delivered as the organisation moves towards the goal. This i believe is a role for enterprise architecture.

Realising the goal is likely to have an impact and sometimes a major impact on the enterprise architecture. The architects tasked with defining the future architecture should monitor the evidence for changes over time and consider these changes for possible impacts on the future designs.

For example, an organisational goal to adopt a new ERP system or scale up operations or service lines will have an impact on the business, information and technology architectures. The evidence behind the decisions suggests certain courses of action as well as the purchase of new technologies. Over time the evidence sources may change their opinions, views or recommendations or new evidence may arise to question the original conclusions.

This is often termed horizon scanning and if overlooked then the result could be costly errors and damage to the original outcome of the goal.

People and evidence


It is important for architects to understand and apply evidence and research findings to their work. At the same time it is important to encourage the decision-makers to pay more attention to that evidence and question it so that it can help inform their decision-making. Often the decision maker is not expecting or wanting to see negative evidence that contradicts their plans or strategies.

I was going through the electronic version of TOGAF 9 and i was surprised to find that the term evidence only appears 6 times. The first occurrence is in section 8.2.2 Developing the Baseline Description within the Business Architecture and the other 5 relate to the security aspects and gathering evidence in the event of a security breach.

Foundations for Evidence Based Enterprise Architecture


I believe that the foundations for evidence based enterprise architecture are the following:

  • An enterprise architecture framework e.g. TOGAF, Zachman supported by necessary extensions such as ITIL, BPML
  • Enterprise Content Management strategy supported by an enterprise content management system and search engine.
  • Master Data Management and Metadata Management policies supported by strategies and systems.
  • A content authoring standard and methodology such as DITA – the Darwin Information Typing Architecture which is an XML-based architecture for authoring, producing, and delivering information.
  • Authoring methods for narrative production through users stories, use cases, scenarios, storyboards, case-controlled studies, case reports, surveys and expert opinion.
  • Controls such as metamodels, taxonomies and ontologies for catagorisation and classification
  • and finally a team of architects able to provide a consultation and validation process to review all architectural content and data through systematic reviews and apply an authority to the evidence.

The final point is probably the hardest to perform as this team has to be able to provide the assurances that the evidence they have identified is necessary and sufficient to be applied to future content and will add value to the user.

So what is evidence?


Here are two definitions of evidence i came across via Google that i think support the concept of Evidence Based Enterprise Architecture.

‘evidence is factual knowledge or data that lends support to or casts doubt on a hypothesis. It is information on which we base our beliefs and ideas of how the world works.’ Morris (2004)

‘It is information or data that people select to help them answer questions… Evidence is anything that can be used as evidence to answer questions …One person’s evidence is another’s information’ Knight 2004