Proposal Assessment Criteria Leading organizations establish a set of “IT investment assessment criteria” that encompasses financial value, non-financial value, and risks. They use this set of criteria to evaluate proposed alternative solutions, before they become proposals, as well as to evaluate fully-developed proposals. The IT investment assessment criteria are higher level requirements that supplement the requirements specified to solve the problem. In other words, a proposed solution must not only solve the problem (that is, meet all the problem solution requirements), but it must do so in a way to satisfy the IT investment assessment criteria. In this way, the “best solution” is identified—the one that will not only solve the problem but contribute the most to the organization in doing so. The IT investment assessment criteria takes the form of a standard set of high level assessment criteria used organization- wide. In Template 6 (risk analysis), you already used a set of risk criteria similar to those used by many organizations, and you will use those criteria again here along with other criteria in performing a cost- benefits-risks tradeoff analysis to select the best alternative solution. Standard Set of Assessment Criteria Having a standard set of high level IT investment criteria for use organization-wide makes the following possible:
• All proposed IT investments can be subject to the same established set of decision criteria, rather than each proposal having individually created criteria
• The various components of the organization can know in advance what high-level criteria must be met in order for their internal proposal to be approved or for a proposal submitted by an external source to be selected. This helps internal groups as well as external source to prepare better proposals.
• Because all proposals must meet the high-level criteria as well as the solution criteria, more proposals will be received that are responsive to the organization’s priority goals and objectives, including alignment, timeliness, and benefit-cost-risk standards.
• The organization’s investment review committee can use the same set of criteria to evaluate and compare (a) internally-generated alternative solutions to a problem, (b) internal proposals that compete with other internal proposals, and (c) proposals submitted by external sources.
Note the fourth point above. One use of the set of high level criteria is to evaluate alternative solutions to a specific performance problem in order to identify the best solution. (This is how you will use Template 8). In leading organization, the various proposal teams use the high-level criteria to evaluate their own alternative solutions to identify the best one. They will develop the best solution into a proposal that will be submitted to an executive investment review committee. The committee receives proposals from various parts of the organization and uses the same set of high level criteria to evaluate the competing proposals submitted to them. The committee decides which among those submitted are best for the organization and should be funded.
If a request for proposal has gone out, the proposals received will first be reviewed by the proposal selection committee and then the proposal that this committee selects will be sent to the executive review committee. This proposal will have to compete with all of the other proposals for funding. The investment review committee, using the established high-level criteria, reviews the various proposals and decides which should be funded. The established high-level criteria are equally effective evaluating both internally-prepared proposals and externally prepared proposals. The Clinger-Cohen Act of 1996 requires federal agencies to establish such a set of IT investment criteria for organization-wide use in evaluating and selecting the IT proposals to fund. Using Template 8 Template 8 is an example of a scoring matrix that uses research-based high-level criteria “categories” to organize a standard set of high level IT investment assessment criteria. The categories and their definitions are based on research involving successful and unsuccessful IT acquisitions. Although the criteria categories and their definitions have broad applicability to organizations, the specific criteria within a category will vary to reflect the needs of the particular organization. For example, the category “Strategic Match” is defined as “The degree of match of the proposed solution with the strategic goals and objectives of the organization.” This definition is applicable to any organization and its proposals, but organizations have different goals and objectives so the specific criteria they will include in this category will vary. Here are some examples of goals that illustrate how diverse the goals can be. An organization is likely to have some of these goals but not all of them.
Examples of Organizational Goals
Maintain a competitive edge Gain leadership in the field Increase market share Create new products Increase number of partners Generate brand recognition Obtain international business
Increase revenues Improve customer service Increase productivity Increase efficiency Reduce cost Be environmentally friendly Many, many other possibilities
Value and Risk Criteria, Weights, and Rating Range Usually the criteria are divided into Value criteria categories and Risk criteria categories, as is done in Template 8. The term “value” is used, rather than “benefit,” because value is more encompassing. In terms of financial value, it is what it is left after the cost is taken into account. For example, if a proposal will generate $2 million in income and its cost (investment) is $1 million, the financial value of the proposal is $1 million. There are also “non-financial” values, which cannot be easily quantified in dollars. For example, keeping customers happy is an important non-financial value; they may not buy more products or services (i.e., increase the ROI), but they are likely to stay as customers, which is a value to the organization. The template also gives a certain weight to each criteria category. The amount of weight given is intended to represent the relative importance of that criteria category to the particular organization. Most organizations give ROI/NPV the highest Value weight, though sometimes (mainly in government agencies) Strategic Match is given the same or even a higher weight. With this template, the rating range for each criteria category is from 1 to 5, with 1 being the lowest score and 5 being the highest score. Defining Rating Scores Organizations determine and document the meaning of each rating for each criteria category. For example, a “1” rating on ROI/NPV would be for a proposal with a low ROI over the next three years and a low or modest NPV. A “5” rating on ROI/NPV might be for a proposals with an ROI forecast of 300% or more and a large NPV. Organizations define, document, and distribute a guide explaining the meaning of each rating. In this way, a each rating from “1” to “5” (if a 5-point scale is used) has the same meaning throughout the organization. The rating on a criteria category is multiplied by the weight given to that category to come up with the proposal’s weighted score for a criteria category. For example, if the category ROI/NPV has a weight of 8 and a proposal receive a “5” rating on ROI/NPV, the 5 rating would be multiplied by the weight of 8, so the entry in the Weighted Score cell would be 40 (8 x 5 = 40).
ROI/NPV 8 5 40
The Value criteria category given the most weight in Template 8 is ROI & NPV. All of the other Value criteria in the template are non-financial – that is, they are not easily measured in financial terms. However, they are measurable in non-financial terms. For example, Quality of Worklife can be measured via a survey of the staff expected to use the proposed system. If they respond that they believe the proposed system will enrich their jobs, broaden their skills, and increase their pay, the proposal would be rated high on Quality of Worklife. If the survey shows the opposite results, there would be a low rating on Quality of Worklife. (A low rating for a category is a sign that the proposal needs to be improved or rejected.) Of note, a high rating in this category can lead to future increases in ROI/NPV.
The Risk criteria category given the most weight in Template 8 is Organizational Risk. This type of risk is the most common cause of project failures. Many criteria are normally included in this category. The objective is to identify the presence of Organizational Risk in a proposal before it is selected for funding. Then, depending on the amount and type of risk, either eliminate the proposal or modify it to avoid as much risk as possible and mitigate the risk that cannot be avoided. The result should be less project failures. Risk Criteria Used Again As mentioned earlier, this template uses the same Risk Criteria used in Template 6 in an earlier “screening out” of evaluation of alternative solutions. As a result, you know much about the risk characteristics of the two solution alternatives you selected. You may wish to refer to your results in Template 6 as an aid to determining the risk inputs for Template 8. However, you have more information about your alternatives now, so do not be reluctant to change your risk ratings for Template 8 if you believe you should. You will also need the results from the Economic Analysis of your two alternatives (Template 7) as inputs for Template 8. Definitions of the Value and Risk Criteria Categories Used in Template 8 How are the Value and Risk criteria categories defined? Here are the definitions of the criteria used in Template 8. Value Criteria
1. ROI and NPV: The relative financial return on investment (ROI) and the net present value (NPV) of this investment compared with competing investments
2. Strategic Match: The degree of match of the proposed solution with the strategic goals and objectives of the organization.
3. Competitive Response: The extent to which this proposed solution is as good or better than what competitors are doing.
4. Definitional Certainty: The degree to which sufficient reliable data are available to enable an accurate definition of the problem and/or the future environment in which the proposed solution must function.
5. Executive Support: Strength and likelihood of continued executive support (e.g., do some executives oppose this solution).
6. Quality of Worklife: The extent to which it will improve the quality of worklife for employees and others who will staff the activities.
7. Enterprise Architecture: The degree to which it supports the enterprise architecture plan (interoperability, direction of growth, etc.)
1. Organizational Risk: Extent to which the culture and/or policies and practices of the organization may present problems. Extent to which there may be insufficient buy-in, support, or understanding on the part of key stakeholders (e.g., affected managers, employees,
customers). Extent to which employees with the required knowledge and skills may be unavailable when needed to support the project and/or the implemented system.
2. Infrastructure Risk: The extent to which it may negatively affect the organization’s IT and non-IT infrastructure (e.g., will it slow down other processes, reduce needed flexibility, place unusual work demands on some groups?).
3. Information Security and Privacy: The extent to which it deviates from or negatively effects the established standards and requirements for information security and privacy.
4. Complexity Risk: Degree of complexity of the proposed solution and/or which will be required of the implementation project.
5. External Risk: The extent to which external risks pertain to this solution (e.g., risks associated with strategic partners or with vendors or with outside overseers or interest groups, negative impacts on the environment, possible incompatibility with law or regulations).
Benefits of Template 8 Having a single set of high level criteria and weights for evaluating alternative proposed solutions as well as proposals from contractors has many advantages for an organization.
• First, it means that the same high-level criteria will be used with all proposals so that proposals can be compared. If separate criteria were used for each proposal, the proposals could not be compared.
• Second, it increases the reliability and validity of the proposal ratings. Executives, proposal teams, and other stakeholders gain experience and expertise using the high-level criteria. Executives see the value of the ratings in the results produced (e.g., cost savings, improved organizational performance, less risk).
• Third, proposal and acquisition teams use these standard high-level criteria to evaluate alternative proposed solutions in order to select the best candidates to become formal proposals—they know that proposed solutions that have high ratings on the criteria have the best likelihood of being funded.
• Fourth, the executive investment review committee uses the same high-level criteria to evaluate the proposals it receives. In other words, everyone is using the same high-level criteria.
• And fifth, the high-level criteria can be modified as needed to match the changing needs of the organization. For example, the Value criteria category “Strategic Match” refers to the extent to which a proposal is aligned with the strategic goals of the organization. If the strategic goals change, the alignment must be with the new strategic goals, not with the old ones.
Your Use of Template 8 Use Template 8 and its criteria category definitions to evaluate your alternative solutions in order to determine which one is best and should become the basis of a request for proposal. In evaluating your alternatives, do the following:
• Use the weights provided in the template.
• Use a rating range of 1 to 5 in evaluating an alternative relative to a criterion, with 5 being the highest rating. Remember that a “5” rating on a Value Criterion is good but a “5” rating on a Risk Criterion is bad.
Remember, in Template 4, the top priority risks were given low numbers, such as 1 (highest priority) and 2 (second highest priority). In this template, high numbers are used to indicate much value or much risk, depending on whether value or risk is being evaluated.
IT INVESTMENT ASSESSMENT RATING MATRIX
Solution Alternative #1 [Delete this note and enter the alternative’s name and brief description]
Solution Alternative #2 [Delete this note and enter the alternative’s name and brief description]
Alternative #1 Ratings
Alternative #1 Weighted
Alternative #2 Ratings
Alternative #2 Weighted
Scores ROI and NPV 8
Strategic Match 4
Competitive Response 2
Definitional Certainty 2
Executive Support 2
Quality of Worklife 1
Enterprise Architecture 1
Alternative #1 Ratings
Alternative #1 Weighted
Alternative #2 Ratings
Alternative #2 Weighted
Infrastructure Risk 4
Information Security 4
Complexity Risk 3
External Risk 3
Total Weighted Risk Score
Based on benefits-costs-risks tradeoffs, the best solution alternative from among those above is:
[Enter Name of Best Solution] Template 8. IT Investment Assessment Rating Form
Highest Possible Scores. Since the weights for the Value criteria total 20 and 5 is the best rating in each Value criteria category, the highest weighted Value score for an alternative solution is 100 (an alternative would need to get a “5” rating in every category). Similarly, the weights for the Risk criteria add up to 20 and 5 is the worst rating in each Risk criteria category. Therefore, the worst weighted Risk score is 100. A 100 score for Value is good, but a 100 score for Risk is bad. No, you do not subtract the Risk score from the Value score, which would be mixing apples and oranges. Incidentally, some organization regularly did this before realizing that it is illogical and resulted in poor selections of proposals. Selecting the Best Alternative The selection of the best solution from the two or three (or more) that you assess can involve many value-risk tradeoffs. Is it better to have greater value with greater risk or lower value with lower risk? It depends on the organization, its goals, problems, and other considerations. Organizations in highly competitive environments are more likely to select an innovative high payoff solution with more risk than a run-of-the-mill low payoff solution with little risk. Government agencies tend to accept less risk than commercial organizations. In your case, you need to select the alternative solution that you believe is on balance the best to solve your problem. –and you should be prepared to defend your choice with evidence and fact-based assumptions. After you have made your decision, enter the name of the selected best solution alternative in the space provided in the template. IMPORTANT NOTES: 1. This template should be used with a Microsoft Word (or other word processor) document. It should
not be used in an Excel worksheet because this is not an appropriate medium for transmitting the document to others or for printing it for distribution.
2. The template is adjustable. After the template is copied to a Word document, replace the sample
entries in the cells with your own information. You can change the length and width of the template and its cells after you copy the template to your document. If you put your cursor on a line in the template, the line can be moved horizontally or vertically, depending on the line, to best fit your information. There is no need to color any of the cells in your Word version, though color is used in the example above.