Return to Braley Consulting Services, Inc. Home Page
Problems during the installation of a new computer system are described at great length in the literature and at professional meetings. It often appears to be a contest among users to see who can complain the loudest - about their system or their vendor. While there are hints that the selection process may have not have been handled well, few people appreciate how many problems may have resulted from a poorly structured evaluation.
This article takes an in-depth look at the process of evaluating information systems. The history of system development and the resulting evaluation processes are summarized. The classical selection process which worked well in the past is examined in light of the way modern systems are developed and installed.
Project planning is discussed in detail since this activity is often handled superficially because of the desire to start looking at systems. In the worst case planning involves "doing what comes naturally". Unfortunately, what comes naturally for inexperienced organizations might not be the best course of action.
Commercial software and systems development began in earnest in the 1960's. At that time most software was skeletal in nature and required extensive customization to accomplish any real work. Computer companies were mostly hardware companies and few had any real interest in software.
Software development in these firms was a "necessary evil". Its only purpose was to help sell "iron". Such reasoning made sense when most applications were one-of-a-kind programs. Software could not be mass produced so programming was a difficult business to "manage" in the traditional sense and it certainly was not very profitable.
These business practices led naturally to a certain way of selecting computers. If a new system was to have any chance of performing as the user desired, a great deal of effort had to go into the specification. In other words, what the system was to do and how it was to perform had to be described in excruciating detail by the user. Since the software was going to be custom developed - by the hardware vendor or a third party - the details of its operation had to be documented by the purchaser. This was the specification stage.
The fact that most users had neither the time nor the experience to design good systems led quite often to disaster.
A Request for Proposals then consisted of certain bidding instructions along with a detailed specification. Few questions were included because the vendor's traditional response to questions was "we can do it any way you want."
The fact that most users had neither the time or experience to design good systems - which they had to do if the vendor was going to "meet their needs" - led quite often to disaster. To the user's complaint that a new system did not do what they wanted, the programmer's standard reply was "you didn't tell us what you wanted".
This problem was exacerbated by the division of the population into two distinct categories: computer people and non-computer people. While little real attempt was made to reach a common understanding between the two groups, both sides were accused of not understanding the needs or problems of the other.
As systems grew increasingly complex users became more frustrated and cost overruns and missed deadlines were common place. Sophisticated systems were developed which did not meet the expectations of users and expensive packages were frequently underutilized.
Lack of integration became a major barrier to system expansion in the 1980's. As organizations tried to develop increasingly comprehensive systems, the techniques they had used in the struggle to build individual components could not handle the problems associated with widespread integration of various systems.
Disputes among individuals and units of the client organization - about how systems were to function and whose fault it was when they failed to do so - became as commonplace as disagreements between the client and vendor had been in the past.
The underlying problem was that computer systems were being micromanaged. Hours were spent on designing report formats without adequate attention to whether the information on the reports was in fact useful to the end user. Vendors guessed at what users wanted because specifications were incomplete. Departments in the user organization guessed at what other departments wanted because there was little communication among the various units.
It has been a slow and difficult process but software has gradually evolved from custom one-of-a-kind programs to packages intended for a variety of users. Tailoring is possible but only to the extent options were considered by the vendor during development - or subsequent upgrades - of the system. PC packages represent the ultimate in this evolution.
Millions of copies of very complex programs are sold for less than $300 each. A company providing such software is one of the best known and most profitable firms in the country at the same time as hardware vendors lay of thousands of employees and lose billions of dollars in the process.
The quality revolution is real.
Another change has occurred which directly impacts computer systems development. The quality revolution is real. While much of the commotion surrounding this phenomena is marketing hype, there is no doubt that the core principal - greater involvement of individuals throughout an organization in decision making - is happening.
Combining this principal with the idea of a "PC or terminal on everyone's desk" and computer planning takes on a whole new meaning. It is a no longer a matter of a few experts dictating to the masses. Users are now partners in the design/selection process and the responsibility for success or failure of systems is shared by staff and management throughout the organization.
Taken to the extreme decision making extends from top to bottom in the corporate structure. It involves departmental units, upper management and specialized committees established for specific project. A typical hospital approach to system selection should involve the following people
Board of Directors
Administration
nformation Systems Steering Committee
The Medical Staff
Department Managers
Information Systems Department
An Integration Committee
Project Teams
Staff throughout the organization
Describing the roles played by each of these elements is outside the scope of this paper. Suffice it to say these roles should be agreed upon prior to the initiation of a project. The involvement of the Information Systems Steering Committee and the Integration Committee - a relatively new player - are often poorly defined. This is evident by the fact that both are concerned with "tieing the pieces together" - at the highest level of planning in the first case, and during the actual decision making process in the second - and it is precisely these connections that cause most users the greatest problem.
Many approaches are taken to evaluating information systems. They vary in complexity from elaborate computerized models with Pert Charts and unending meetings of all parties to the opposite extreme involving essentially no structured system at all. When ask why a particular approach was used, most participants have no idea. They certainly don't indicate that an approach was chosen because it was considered to be the best among a variety of methodologies considered.
While organizations take great pains to decide on the best system, most spend precious little effort determining the best approach to decision making. Deciding how to decide often removes the user too far from the "real problem" of looking at systems.
Major surprises about what a system does or how it performs can usually be traced back to a rushed or otherwise inadequate evaluation
Lack of attention to the process to be used prior to initiating work is not a trivial issue considering that many user complaints are related directly to how the system was chosen rather than the product itself. This problem is not easy to see at first since complaints are often leveled at "the system" or "the company." Consider a few examples:
When we installed the computer I thought we would be able to ... but I guess we can't.
I don't like the new system because the reports are confusing and hard to read.
I like it but you can never get on the terminals; they're always busy.
It's too slow.
Another user told us we would be able to ... but apparently we can't.
I don't like it because it takes a lot more time to ...
They never told us we would have to ...
Most complaints of this nature should have been dealt with while the system was being selected. If there are insufficient terminals or response time is poor, the RFP/Proposal/Contracting process was probably faulty. Users who incorrectly thought a system would have certain capabilities did not get their information from the correct source - the proposal written by the company and the contract signed by both organizations.
With software that is essentially complete - i.e. little customization will be performed - learning the capabilities and limitations of a system is relatively straightforward. It is time consuming if done right but it is still possible to do. Major surprises about what a system does or how it performs after it is installed can usually be traced back to a rushed or otherwise inadequate evaluation.
The tasks described below are not meant to be a formula for success. They represent an example of one way to do they job and they should be examined with the same degree of thoroughness as any other approach. The main reason these tasks are included is to show how a logical plan can be constructed that makes sense to users - under the right circumstances - and incorporates the all-important aspects of user participation and responsibility that are central to quality improvement in any of its forms.
A good project plan is more than a list of tasks, completion dates and assigned responsibilities. Too often simple project schedules are drawn up by one or two people who would like the effort to be conducted in a certain fashion (usually quick and cheap) with little regard for the real problems they will encounter. This results in a plan that may be completely unrealistic and is therefore abandoned or significantly modified almost every week.
Key individuals in all affected departments must understand the plan and believe it is the best way to make the decision
More sophisticated plans can suffer from too much complexity. A manager with a new "project planning program" can be a dangerous person. The human issues of selecting and installing a new system are often ignored as reams of paper are generated to explain how a computer will be selected. Ironically, a major goal of the new computer will be to reduce paper!
There are two simple but critical criteria for a good plan: 1) Key individuals in all affected departments must understand the approach outlined in the plan and believe it is the best way to make the decision. 2) They must also agree to provide the resources required of their unit to conduct the project in the time frame indicated.
To begin a system selection without such an understanding will almost guarantee the project cannot be executed as planned or the installation will be a disappointment to many users.
To develop a good plan one should start with example task lists obtained from a variety of sources. From these a set of tasks should be selected that make sense to the users, are in a logical order and appear to cover all the bases. No plan should be used just because "we've always done it this way" or "we read it in a book."
Doing something because an "expert" said it was right is a poor excuse for managing. Order of execution is just as important as the tasks themselves. Sending out a Request for Proposal (RFP) at the beginning of a project may result in 30 detailed proposals to review. Screening vendors first based on major criteria and sending RFP's to a few finalists will drastically simplify the evaluation task and result in a better evaluation since effort can be concentrated on the most likely candidates.
After observing many selection projects, a number of questions come to mind.
Why does a well qualified vendor sometimes decline to bid - often to the dismay of the purchasing organization?
What value are site visits when participants often cannot even remember which system they saw let alone many of its important features?
Why is it that procurements seem to go smoothly until proposals are received and it is time to make a decision?
Who do complaints arise about how a decision was made after the selection is finished?
After considering these and numerous other questions the author developed a list of tasks aimed at addressing such "process" questions. Because the environment has changed - users buy "packages" not "development" - it was appropriate to start over in preparing a task schedule. The tasks can be grouped into three stages as shown as shown in the Figure above.
Since understanding what was available - not writing specifications for a system - was the primary concern, education formed the core of the new methodology. Part of the education was structured through classes and textbooks while other elements were informal. These consisted of vendor presentations, site visits and RFP questions all aimed at learning what systems could do and - just as important - could not do.
The question of when and how to justify a computer system plagues every organization. One cause of this difficulty is that justification is seen as a "one shot" process with a straight forward yes or no answer. Many people simply want to know whether a computer system can be justified and they would like to know the answer before they waste money evaluating systems.
The problem is that much of the information needed to do a thorough justification is not available until a significant portion of the evaluation is complete. This can be seen when the "justification question" is rephrased in a more meaningful fashion.
"Is it in the best interest of the organization to invest this amount of money on this product at this time?"
It is obvious from such a statement that a detailed justification cannot be accomplished before proposals are received. That's when system costs will be known. Justification at the outset of a project should be stated differently
A preliminary justification
has the benefit of being "honest"
"Based on our limited knowledge of the subject matter does the likelihood we can eventually justify a computer appear high enough to warrant researching the subject matter in depth?"
While it would be nice to completely justify a purchase before doing any work, this is an unrealistic goal. The best an organization can do at the beginning is to justify spending resources on an investigation.
This might appear to be a half-hearted approach since there is no guarantee a system will be purchased even after all the research is done. It is, however, no more half-hearted than discovering at the conclusion of the evaluation that the original "justification" was faulty and no system is installed as a consequence.
A "preliminary justification" has the benefit of being honest. An administration will gain a great deal of respect with the following statement. "We won't know whether or not we will be able to afford a system until we learn more about the capabilities and costs of the available products. We do, however, believe the prospects are good enough that we should investigate the possibilities in more detail."
Most project teams are too large. They also suffer from a lack of structure. A core project team of 4-8 people should be adequate to conduct most evaluations. As mentioned earlier, it is important to separate technical evaluation from management guidance and approval. If this is done properly, the project team's responsibility will be to evaluate the systems in depth and make recommendations to management at each major decision point.
Consequently, most of the work can be handled by the technical experts and manager(s) of the effected department with proper coordination with others throughout the organization. A few people can be designated as "ad hoc" members and brought in to the process when their specific input is required. Contributions from other staff members will be solicited by the team at numerous points in the process.
Once a team is selected, it is vital that all members understand the importance of their contribution. If a team is too large, it is easy for one person to miss a meeting. A small dedicated team where each member has a well-understood assignment is a much more effective approach.
An initial screening should be conducted quickly to eliminate vendors that are least appropriate for the organization. Vendors that are new to the market and have limited experience would be eliminated by organizations that prefer low risk approaches.
Vendors with a poor track record in support would not be considered by organizations that typically rely heavily on such service. Published lists of vendors as well as experience of peers can form the basis for initial screening.
This task should conclude with a list of vendors eliminated and the reason(s) each one was excluded. A second list should include the names of vendors selected for evaluation along with a brief description of each.
Although it is tempting to start a search by talking to vendors and seeing installed systems, this is a mistake. Team members that are inadequately prepared waste their time as well as the time of vendor representatives and other uses when they ask the wrong types of questions.
Reading standard material provided by a vendor before the representative comes in for a presentation will make each meeting much more productive. Some understanding of terminology and distinguishing features of a system prior to meeting with the vendor will add significantly to the value of the presentation.
Vendor visits must be structured by the user. Otherwise, each one discusses different aspects of a system and no comparison can be made. Certainly each representative should be encouraged to point out unique features of his or her product, but the overall format should be similar. While it is common practice to have products demonstrated - through terminal connection or PC software - their are drawbacks to demonstrations at this stage.
First and foremost, it is best to review systems in layers - from the top down. A major part of an introductory presentation is to learn about the company, their clients and their approach to the business. If too much time is spent on details of terminal operation, data entry, report generation, etc., the highest layer will be omitted. Details can be examined later when the team is better prepared and fewer products are involved.
There is always a trade-off between the number of companies under consideration and the depth to which each product can be investigated. The number of companies involved at the beginning precludes a detailed investigation of each one. As the field is narrowed and decisions must be made on an increasingly detailed level, terminal demonstrations of selected systems can provide a certain amount of that detail.
The second problem with early demonstrations is that few details will be remembered for the duration of a project. Information that is important about exactly how a system operates will be lost by the time an in-depth comparison is conducted.
Even though a final configuration cannot be determined until later in the project, it is possible to obtain pricing information on a tentative configuration. Most companies can produce a "budgetary proposal" with a small effort and skeletal information about transaction volumes. It is very important that the user suggest the number of terminals, printers, etc. so that all companies will propose roughly comparable systems.
While some - particularly first time users - will find it difficult to guess at a configuration, it is better than letting the vendor guess. The vendor is generally biased towards underconfiguring at this stage. They do not want to look high priced so if they believe the user will need 20-30 devices, they will propose 20 - or less sometimes.
The results of this budgetary proposal should be used to determine whether or not the preliminary justification was on track. If prices, are much higher than suggested earlier, it would be legitimate to ask whether the original analysis was still valid.
In guessing at the initial configuration, it is better to err on the high side rather than to underconfigure. Now one has ever gotten into trouble for saying "we have decided to spend less money by removing some of the terminals from the original configuration". On the other hand, how many people have ever said they made a mistake and bought too many terminals?
The purpose of a site visit
is NOT to learn what a system does.
Concerning software, no decisions should be made at this stage about whether or not a particular optional package will be used. All software modules should be priced and included in the preliminary proposal.
Contrary to what many people think, the purpose of a site visit is not to learn what a system does. Only the vendor can tell you that - preferably in writing in a contract. While the average user you meet during a site visit has some information about the capabilities of the system, he or she may be completely wrong in important aspects.
They know how they use the system. They may not know about capabilities it has which their organization choose not to install - maybe even before they were hired. They do not know about capabilities the vendor is now marketing which don't happen to be in the version they installed. They may not know that a particular feature of the system which they like very much was custom developed for their organization and is not available to the visitor's facility.
The main purpose of a site visit is to determine one user's reaction to the system and to see how it operates in a roughly comparable environment. A second purpose is to discuss issues of installation and operation which will be valuable later on. In these areas, users are the experts not the vendors. Many of these issues will be common to all vendors so it should not be considered a waste of time to visit an installation of a vendor that is subsequently eliminated.
Sending out Request for Proposal's (RFP) and reading the responses is a time consuming job. As a result, the number of vendor's receiving an RFP should be strictly limited. It is difficult to do justice to detailed proposals from more than four our five vendors. Most organizations do not find it difficult to reduce the number of vendors to a manageable number before sending out RFP's. If a vendor is going to be eliminated because they do not have a strong record, for example, this can be done without the effort of proposal preparation and review.
Finalists can be chosen through a simple matrix scoring scheme using high level functionality and vendor strength criteria. Only vendors that score highest at this level should be considered for detailed evaluation.
A major flaw in many system selection projects is that no formal evaluation system is developed until very late in the game. In particular, many organizations do not develop the criteria and methodology for evaluation until after proposals are received. It is at this point they realize how difficult is to make a decision where many people with different needs are involved.
Pricing should be excluded from the evaluation until a ranking based on system capability and corporate strength has been completed.
They also realize that one system will not be judged best by all participants so a method of compromise must be developed. This method usually involves an assignment of weights and scores to various system features which are then totaled to come up with the final vendor ranking.
The problem with doing this after proposals are received is that RFP probably did not request the appropriate information in the best format for evaluation. How could it, when the evaluation system was not prepared at the time the RFP was written?
While it is not possible to describe a complete evaluation system in this document, a few critical points will be mentioned.
A hierarchical approach works best when assigning weights. This allows weights to be changed in various components without changing them throughout the entire system.
Not all participants should evaluate all elements. Individuals were selected because of their expertise in specific areas. They and they alone should judge the system in the areas were they have the most experience.
Pricing should be excluded from the evaluation until a ranking based on system capability and corporate strength has been completed. After this ranking is done and comparable proposals have been received, it is possible to make a decision concerning the difference in capability vs. the difference in price among all the vendors.
Mandatory requirements should be kept to a minimum for two reasons. First, a large number of mandatory requirements will more than likely eliminate every vendor since they only have to miss one to be disqualified - if they are indeed mandatory requirements. Also, mandatory requirements are often dropped when the price to meet them is determined. So, in fact, they weren't mandatory at all. They are like all other "requirements" - desirable if they are affordable.
There are on-going debates about whether or not a Request for Proposal is required. In the strictest sense it is not. What is required is a detailed understanding of what the proposed systems will do and what the obligations of the vendors will be. This material should be in writing because it will eventually form the basis for a contract with the chosen vendor. One way - certainly not the only way - to get this information is to request it in writing with an RFP. Some organizations would refer to this as a questionnaire, a specification, etc. but in any event it is a written request for specific information. For purposes of discussion, this will be referred to as an RFP in this document.
A Request for Proposal should not be a System Specification. Specifically, it should not tell the vendors exactly what their proposed system should do. This approach worked fine when any new system involved a significant amount of reprogramming. Customization was then the rule, not the exception.
The high cost of customization and support of custom software has changed this situation completely. Most users try to use standard packages to the greatest extent possible. Vendors are certainly aware of customization problems as well. The only ones who boast of their willingness to customize are the ones who have partial systems and are looking for someone to help them complete the development effort. With packaged systems it makes more sense to ask the vendor what the system does and how it works rather than telling them what you would like it to do.
Asking questions has one other advantage. It allows the vendor to offer suggestions and demonstrate methods of operation that you may not have thought of during earlier stages of the process. The evaluating team has only to compare answers to the RFP and judge which ones they like best. They do not have to try to "invent" a best answer at the RFP stage and they are open to new ideas right up to the time of final comparison.
This approach is in distinct contrast to closeting a team early in the process so they can document their needs and describe the "optimum system" without the supposedly confusing influence of vendor claims and counter claims. Systems are too complex today to believe that a typical user can describe how a system should operate in detail without an intense interaction with a variety of vendors and users to learn about all the possibilities.
The more exposure key individuals have before a system is purchased, the less confusion there will be during the installation.
Vendors should not be asked to rush their responses unless there is good cause. One to two months for a detailed response for a complex system is not unreasonable. A busy vendor - the type you would like to work with - has many prospects in the sales cycle. They all have limited resources and the hospital that realizes this and works with the vendors to make the most of their time will come out best in the long run.
It is now time to bring in terminals or small systems for a live demonstration. The field has been narrowed so adequate time can be devoted to each system. With fewer vendors, the confusion about which system does what will be minimized. With a good background in the general capabilities of each system, team members are in a much better position to probe more deeply into how each system works than they would have been during initial presentations. Finally, It is close to the time for a detailed evaluation so the knowledge gained through very recent on-site demonstrations is more likely to be of benefit in this process.
As with all aspects of the selection, demonstrations should be structured so each vendor addresses roughly the same topics. Because each representative will be on-site for a day or more, it is possible to expose a significant number of people - not just the project team - to the intricacies of each product. If certain aspects of a system are to be demonstrated at a particular time, individuals who have knowledge in these areas should be brought in to observe.
The more exposure key individuals get before a system is installed, the less confusion there will be during the installation.
More evaluations grind to a halt during this task than at any other point in the process. The reason is simple. As mentioned earlier most organizations do not give adequate attention to the evaluation criteria and methodology before attempting to perform the evaluation. As a result, they realize at this point how difficult it will be to make a decision that is best for the organization but that may not be considered best by one or more individuals or units. The simple decision making approach that works for small systems that affect few people cannot be "scaled up" to handle complex procurements.
At this critical juncture, the unprepared organization is forced to confront the issue of "exactly how are we going to decide" for the first time. The lucky ones struggle with this problem and finally determine how the evaluation will be done and then realize they did not collect the correct information during previous tasks. So they go back for with a revised RFP or a second round of questions. The unlucky ones are not able to devise a workable methodology and the process slows to a crawl or stops altogether.
If, however, the organization worked on the evaluation process ahead of time it is only a matter of working through the steps that were agreed to before RFP's were sent out. Very little additional material should be needed precisely because the RFP questions were developed to collect the information the organization felt would be useful in comparing systems.
Without this core technical content all the legal language in the world will not form the basis for a satisfactory installation.
At the conclusion of this task, the team should have a tentative ranking of vendors along with a list of reasons why each was ranked as it was. The ranking is a numeric score and the reasons justify each score.
User contacts to this point have been limited to site visits - generally one per vendor. After the detailed evaluation, the team will undoubtedly have questions remaining. Some of these can best be answered by talking to other users. Another set of questions - relating to each vendor's recent installation experience - can only be addressed by the newest clients. A structured survey - where at least five recent clients of each of the top ranked vendors are called - is appropriate at this time. Each site should be ask similar questions except for items that are vendor specific.
A third set of questions have little to do with vendor selection but are helpful in any case. Questions about the "installation experience" in general will help the organization avoid some of the problems faced by others. Open ended questions like "What should we watch out for?" and "What would you do different next time?" will allow others to share their experience with your team right before the installation is initiated.
If a major capability of a leading system - a specific HIS interface for example - was not observed during the initial round of visits, it might be necessary to see it at this time. Such a visit would not involve the entire team - only those who are concerned with the item in question.
No major purchase should be made based solely on contact with a sales representative and one or two marketing support analysts. Once a system is selected an entirely different set of people will take over including installers, support personnel and developers of future enhancements.
Before a contract is signed, it is helpful if client managers and executives can meet managers in the vendor organization. A one day visit - possibly to two or more leading vendors - can uncover information that is impossible to find in volumes of printed material.
If necessary, a detailed cost-benefit analysis can be performed at this time. All of the information including exact system costs and complete descriptions of capability are available. In addition, the experience of other users - in terms of installation and operational costs - can be factored in to provide an accurate life cycle projection.
Justification is often difficult and may be done in steps. The aspects of justification that can be accomplished during each phase of evaluation are shown in the accompanying Figure.
With all of the homework accomplished, the team is in a position to recommend the top ranked vendor for negotiation. The reasons the system was selected are completely documented. The justification for a new system is finished. The major effort remaining is to summarize the work to-date so it can be presented to busy executives and board members.
Contract negotiations almost always take longer than anticipated. The only way to make this task go quickly is to accept the standard contract with few changes. Significant changes must be reviewed and agreed to by executives and lawyers from both sides. This process takes time.
A contract, however, should not be looked at strictly as the purview of attorneys. First and foremost it must describe - in user terms - exactly what the vendor is proposing to install. Without this core technical content all the legal language in the world will not form the basis for a satisfactory installation.
A poorly run selection process cannot be salvaged at the last minute by bringing in a shrewed contract negotiator. A good contract is the culmination of a sequence of well-planned evaluation and selection tasks. It is the formalization of the understanding between two parties that takes months to develop.
Some organizations rank vendors and negotiate with the top ranked company. Others negotiate with the top two or three. In either case, lower ranked vendors should not be ruled out until an agreement is signed.
The final presentation to the Board of Directors is straight forward at this point since the necessary homework has been done. All questions about system capability, cost, benefits, installation effort and impact on the organization have been addressed. A simple overview of the process and the resulting recommendation is backed up by volumes of technical detail.
Since board members have limited understanding of many applications, their questions are more likely to address process issues. Did you consider ...? Why did you take this particular step. Etc. Because the process was well-structured and all parties agreed it was the best approach, there should be little difficulty in answering questions of this nature.
A complex system selection process should not be initiated without careful planning. Not just any plan will work. Before any evaluation activity is undertaken the following questions must be considered:
Has adequate effort gone into developing a comprehensive project plan?
Does the plan make sense in content and structure to all parties?
Do they truly believe it is the best approach?
Does it involve key individuals and units throughout the organization?
Does it rely on internal expertise to the maximum extent possible?
Is the approach in line with the modern concept of packaged software as opposed to an earlier model based on custom programming?
Have all parties who will be relied on for significant contributions agreed to commit the time and energy?
Has the approach worked well in our organization before? In other organizations? If not, what assurance do we have that it will be successful here?