Return to Braley Consulting Services, Inc. Home Page
"It seems like a good system but they certainly weren't using much of its capability."
That's a frequent comment heard when a project team visits an installation while evaluating computer systems. Is it true that systems are not utilized to their fullest potential? Based on 23 years experience in healthcare systems I am convinced that underutilization of complex information systems is an increasingly serious problem.
Of course there is a great variation from site to site. Some hospitals use all the advertised capabilities and create many new features on their own. That's why vendors steer prospects to specific facilities for site visits. "They really know how to use the system."
It is noticeable in many cases where systems are not being fully utilized that the enthusiasm of the staff is low as well. While enthusiasm is difficult to measure, one often gets the impression that trying to get more out of a system is too much work. "It's just not worth my time or energy".
This paper will consider some of the reasons why systems are underutilized and expectations at the time of purchase are frequently not met.
"How much was paid for capabilities for which little or no benefit was received?" Why is this an important issue? "What difference does it make if we don't use parts of a system?" It is true that every system will have features that a given user does not need. You can expect some percentage of any system to be ignored. The important questions are:
How do you measure percentage of utilization in complex systems? One approach would be to survey users and managers at all levels and have them estimate the percentage utilization in areas where they are knowledgeable. A laboratory manager who knew that quality control, workload recording, ad hoc reporting and reflex testing were not used in a laboratory system could provide one estimate. A tech at the bench, after reading through the list of available commands and options, could indicate the percentage which he or she understands and uses. While admittedly an unscientific approach, such a simple survey would at least point out parts of a system which were not used. It could also identify areas where, for example, management thought a system was being used but in fact it was not. This is another reason for confusion on the part of the site visitors mentioned earlier. Different people will say entirely different - and sometimes conflicting - things about a system.
Of course many times there are honest differences of opinion - one person likes a particular feature or says that response time is adequate - another may disagree with both statements. The first is likely to use the feature because it is beneficial and she does not waste time waiting on the system.
The second does not see the benefit or would claim the system is to slow and it is easier to do it the "old way".
Other statements about usability are important to consider, however. A manager believes the on-line interface to an instrument works just fine while the bench tech bypasses it because it is too cumbersome. A program that was installed initially has been abandoned by office personnel; they have reverted to a manual log.
These situations make measurement of system use difficult. That's the reason that opinions must be collected from many individuals at all levels in the organization. Only then can a true picture emerge.
To use an example that almost anyone would relate to, one might survey WordPerfect users to determine the utilization of the program "everybody" uses. How many WordPerfect users use Style Sheets, Macros, Imported Graphics, Tables, Indexing, Table of Contents, Multiple Columns or the Thesaurus?
It is likely that a large percentage of WordPerfect users use the program for memos, letters and other short, simple documents. Almost any word processor - some costing less than $50 - could do this work. Undoubtedly the same thing would apply to the spreadsheet Lotus 1-2-3.
The chart below illustrates the results of a hypothetical survey as described above. In this example, likely estimates from many facilities are plotted and the results range from systems that were so underutilized as to be deemed "failures" to the opposite extreme - a small number of facilities classified as "super users".
Utilization refers to the percentage of the advertised, applicable features used by a particular hospital. "Advertised" means that the vendor specifically promotes a feature while "applicable" indicates the feature is appropriate for this user. The small category of "super users" include those elite facilities that not only use most of the advertised features but have created features of their own.
These are the show places for the vendor - hospitals that are at the cutting edge and envision uses the vendor has not even imagined. It is easy to see why some users - i.e. those who spend the time and energy on the process - can out think the vendor. The vendor does not actually use the system on a regular basis while the users do.
Some vendors will say "you can visit any of our sites - they are all satisfied clients". Others will be very selective and provide a short list of chosen references because they want to make sure the prospect visits a show place. Both vendors miss the point. The first ignores the fact that some sites are just better than others and serious prospects should be guided to those facilities. Some sites - although happy with the product - may be using only a limited portion of its capability.
In the second case, the vendor takes too much responsibility for the site and does not want to acknowledge that some users get more out of a system than others. The ideal situation would be for vendors to provide a complete user list and indicate which clients were using which components as well as which sites would provide the most insight into the benefits of a system. All parties should acknowledge that even a good system can be used poorly. The success of a system depends as much - if not more - on the institution than it does on the product installed.
"While initial training may be excellent, follow-up sessions …are even more critical."
Of course some vendors who do not provide a complete users list are trying to cover up sites where they have serious problems. If this is the reason for listing only selected references, most prospects would want to avoid these companies. All vendors have problems from time to time. The least they can do is discuss them openly and explain what they are doing to solve them. Conversations with representatives from those troubled sites can confirm that the vendor is indeed working on solutions even if a site visit is not appropriate.
Why aren't applicable features used? After all they've been bought and paid for. Isn't the user missing out on some of the advantages the system has to offer? Many reasons are given:
Given the fact that complex systems have thousands of pages of documentation and hundreds of commands available, is it any wonder that users are unaware of many capabilities. For whatever reason, they just do not know about very useful features of the system.
Three training related items are possible causes. If the employee is new or the product recently installed, it is possible that some training has not been completed. However, given the scheduling problems faced by many users, it is also likely that some employees will miss some classes. The result could easily be capabilities that go unused. Poor training can have the same results.
Attending class however does not guarantee that a user will take full advantage of a product's capabilities. Even the most serious student being trained under ideal conditions can be overwhelmed by the complexity of a new system. While initial training may be excellent, follow-up sessions that review advanced features are even more critical.
They are important because basic capabilities taught in beginning classes will most certainly be learned sooner or later no matter how poor the initial instruction. Advanced features - which might be used by fewer employees and are not as well-known - could be completely neglected no matter how beneficial they would be.
More advanced classes and other types of follow-up can concentrate on these areas once users are comfortable with the basic parts of the system. The problem at many sites is that once the initial flurry of installation and training activities are completed, they just don't get around to the follow-up that can make the difference in a mediocre installation and an outstanding one. Poor documentation is another cause of underutilization. Often sophisticated systems are crippled by inadequate documentation. Documentation can take a variety of forms including reference manuals, tutorial guides, quick lists of common commands, product overviews and on-line help screens. No matter what form it is in, documentation must be accessible by many types of users under a variety of conditions.
"No amount of user-friendliness will help if the user does not know a feature exists …."
Certainly an "intuitive" or "user-friendly" system would require less step-by-step written instruction but any system must have reference materials that naturally lead the user to all features. No amount of user-friendliness will help if the user does not know a feature exists or what its purpose is. While a user may be aware of a command and have some understanding of its purpose, the depth of understanding is often limited. Without knowing the benefit to himself, his department, some other individual or department, or the institution as a whole, most individuals would not exert the effort required to change. "It's not worth it since I don't know what good it's going to do." Given that a user knows about a capability and is aware of a benefit, he may still say "that's nice but we just haven't had time to install it or train the staff". This is understandable in the first few months of a project. But when a year or more has elapsed and major benefits are still not being realized, it is a serious problem.
Often this situation arises because the purchaser devoted a great amount of time to learning about the features of a system and its costs during the evaluation process and very little time understanding what it would take to make it work. Staff projections for the installation and in subsequent years are often pulled out of the air.
An organization that is unfamiliar with a certain computer application will generally make optimistic estimates of capabilities, installation schedules, costs and staffing. Often these estimates are suggested by vendors - who are generally not rewarded for pessimistic attitudes. The penalty for underestimating staff time is eventually paid when important parts of a system are not used due to lack of time to set up the files, test the system and train all users. All human beings have a reluctance to change. Some are more rigid than others. "This is the way we've always done it". Most people must feel some pressure in order to make changes. This pressure might come, for example, from an understanding that the current heavy workload would be reduced by implementing new systems or procedures.
At other times pressure must come from other departments or administration. This pressure cannot be in the form of heavy handed threats or demands. It must be applied with the utmost care and patience. In the most successful installations users are encouraged to change not forced against their will.
A serious problem arises when an individual is required to change for the ultimate benefit of another person or department. The team approach described below will address this issue.
Numerous trends will contribute to the problem of underutilization including
In the continuing battle for leadership, companies will add more and more features to their systems. This increasing complexity results in not only more commands to learn but also increases the difficulty of using the original more basic commands. Using twenty commands is easier if they are not buried in a set of sixty commands - many of which are unfamiliar and unused.
New hardware and software can be expected to arrive at an ever increasing rate. Powerful low priced processors and sophisticated software development tools will greatly accelerate the product cycle.
Traditionally users been required to learn one primary system. In part this was due to the fact that systems were isolated and interfacing was difficult. With increasing efforts to integrate systems more users will have access to a variety of programs. In many cases the human interface with each system will be different. Until consistency can be developed - most likely through front end "shells" - the user will be required to learn a variety of different commands and data entry techniques.
Many people believe that skill levels of new employees and dedication to the job on the part of some employees are decreasing. It is certainly true that as computer applications spread, more individuals - including those whose positions have traditionally required less skill - will be brought into the process. This results in a catch 22. Increasingly complex systems will be used by employees with decreasing skill levels.
Finally, as organizations fight to trim costs, training is one of the first areas considered for reduction. The impact of these cuts is not obvious without close scrutiny so it is easy to sacrifice in this area. Systems become more complex as time and resources to learn them decrease - another catch 22.
Systems that are not fully utilized are expensive to operate. Costs are increased across the board including:
Underutilization results in the purchase of system components which are never used - the most obvious waste. In addition systems with features that go unused are more complex than simpler systems and have correspondingly high maintenance costs both in terms of vendor charges and user time. Finally, training and operational costs tend to increase when features are added - even though some are not used.
Managers and staff alike must keep in mind that buying a good system is not sufficient. Even spending a lot of money will not guarantee success. The objective of all parties who are concerned with the best use of resources should be to increase utilization of systems so the organization will maximize value received over the life of the product.
Improving utilization begins before a system is installed and extends throughout its lifetime. Areas to consider include:
Choice of Systems Frequently systems are selected based on a list of features. The one with the most features is considered best. That, of course, makes the selection process relatively easy. A wiser choice would take into account practical considerations as to whether or not major components would be used.
A phone survey of users of similar size, for example, could uncover parts of a system that are not widely used. That does not mean the organization selecting the system would not use the features. It does mean that these areas should be scrutinized carefully if functionality that appears so useful is - in fact - not used by many people.
Selection Process It is possible that the entire selection process may need to be overhauled. If the organization has not reviewed the evaluation methodology in detail to make sure it is appropriate to the systems being evaluated, the process may be the root of the problem.
While everyone understands that equipment and systems need to be replaced from time to time, it is just as important to determine whether or not procedures and processes have become obsolete before they are used again.
A selection process is often used because
None of these are valid reasons to use a particular approach. While they may be a good source of ideas about what to do there is only one final conclusion that is legitimate. "We are using this approach because we have reviewed it in detail and it meets the objectives of our organization regarding the evaluation and implementation of new technology".
This statement implies that a set of objectives have been developed. If this is not the case, a list of objectives should be drawn up which are broader than merely specifying an ideal system. These objectives might include:
While it is outside the scope of this paper to discuss the selection process in detail, it is important to point out that the objectives should look beyond the evaluation of products and prepare the organization to get the utmost benefit from whatever product is chosen. The first objective listed above - Select the Best System - should be met by choosing a system with the best combination of "usable" features not just the longest list. "… another schedule plucked from the air!" Probably no single aspect of system selection causes more problems than development of the evaluation criteria. How do you decide what is "best"? Hours are spent on site visits, vendor presentations and preparation of a Request for Proposals. After all the material is collected, the difficulty in actually making a decision hits home when conflicting needs and concerns begin to surface. It is particularly difficult to select a system that will satisfy many individuals and departments because the benefits of systems are not often distributed uniformly. The tendency is to try to be all things to all people with the emphasis being on which system offers the most features.
This aspect can consume so much time and energy that it is hard to consider greater concerns relating to the real usefulness of a system - particularly for first time users.
Regarding the experience of users, it is easy to assume that the replacement of an existing computer system will be noticeably easier than the installation of the first system. Very often this is not the case. The major issue is change. There is a reluctance to give up a "comfortable" old system in spite of its short comings - whether or not that old system is manual or computerized.
In any case the evaluation criteria should be considered much earlier in the process - before everyone gets bogged down in the details. If a broader view is taken initially, priorities can be established and the details fall into place more easily.
The traditional "specification" approach to soliciting proposals is not a good solution for this problem. In this approach, vendors would be told exactly what to provide - ideally to meet the needs of all users - and they would bid systems according to this specification. This approach was developed many years ago - at a time when vendors did a large amount of custom programming and were not reluctant to modify their system for each site.
The complexity of modern systems and the costs of maintaining multiple versions of software have changed all that. For the most part it is not possible to tell a vendor what a system should do in hopes of meeting all of the criteria established by an interdisciplinary committee.
It is best to ask what each product can do - with little if any custom programming - and then determine which system scores highest using an evaluation criteria which all parties have agreed to ahead of time.
The second objective - Develop Creative Users - is intended to help users get the most out of whichever system is chosen. Early and thorough user involvement including numerous contacts with current users are critical to meeting this objective.
The potential benefits of a very good system can be offset by a bad installation. Selecting the right system does not guarantee a smooth installation. Therefore, one goal of the selection process should be to prepare users for the implementation.
Finally, costs must be kept under control. It is particularly difficult to get numerous users in a large facility involved in a selection process without wasting hundreds of hours of staff time in meetings. Waste must be avoided but involvement is mandatory. A well-structured team approach can be used to focus effort through key individuals while accommodating input from all concerned parties.
The diagram on the following page shows how a project team can interact with other individuals and units in the institution who must participate if the effort is to succeed. An important part of the process is to divide up responsibilities between staff members, the project team, a management committee that must oversee the work and the various departments involved.
Poor scheduling is the cause of many problems. Too often inexperienced teams have no idea how long a project will take. The tendency is to be optimistic. No one wants to think an evaluation might take six months, a year or more. Even though receiving approval to proceed on a project may take years, once it is received the pressure is on to do things quickly. "You have approval to pick out a system and we want it done in six months" - another schedule plucked from the air! A realistic schedule can be prepared following a few simple simple rules:
It is amazing how rigid the expected completion date can be when all other milestones slip and slide all over the place. The classic case involves a project that is scheduled to start in June and finish in December. Frequently the starting date is postponed by one or two months but the scheduled completion date does not change.
"Users can develop usable but awkward procedures for even the best new system." Computer projects are notorious for missing deadlines. A large part of the problem is that the target deadlines are unrealistic in the first place. This is not a computer or programming problem. It is a problem of poor planning by management.
Enthusiasm and creativity should be planned into the process. These characteristics are not accidental. Employees must have time to figure out innovative ways to use advanced technology in their specific settings. If they have little or no contact with a product prior to installation, they will be so overwhelmed with learning the "keystrokes" and performing their primary assignments, they will have no time to be creative. They will be relieved to survive each day.
This situation can be self-reinforcing. Employees can develop usable but awkward procedures with even the best new system. They do this because they are not familiar with better solutions and they have to get by somehow. Once created these procedures become part of the "system" and a layer of resistance to change is built around them.
Implementation During implementation it is important to monitor all aspects of the project - the product and personnel included. With regard to the staff, the following suggestions will help:
Comments by users are too often handled as "complaints" or "problems" to be solved. Many are dismissed with a "you know how nurses are" type of response. Most comments should not be classified as complaints. Instead, they should be looked at as opportunities to improve the operation or utilization of an expensive new system.
How much feedback users supply will depend greatly on how this feedback is handled. If one or more users let others know that management ignored or rejected their ideas, you can be certain that others will not bother to make suggestions. What a terrible waste! The best source of detailed information about the operation and effectiveness of a system is the staff that uses it day in and day out.
For everyone willing to speak up, there may be five or more with good ideas who are reluctant to come forward. Management should work to find ways to get the others to speak out and not try to silence the few who do.
Too often milestones are thought of as endpoints. "Well it's over - the system is installed and running". That may be the end of the formal installation schedule but it should be the beginning of a refinement process that lasts for the lifetime of the system. A large percentage of the benefits of a complex system will not be realized at the time the system is initially installed.
If the installation has gone well, the system will perform the basic tasks for which it was purchased. More sophisticated management reports, Q.C. analysis, forecasting tools, and other higher level functionality my not be used initially. Whether or not such capabilities are eventually used makes the difference between users who are getting their money's worth and those who are not.
Training and Documentation These are inseparable topics. They can reinforce or detract from each other depending on the quality of each. A tops down approach where the big picture is presented first helps users see how they fit in. Each module and its specific commands can then be covered in detail.
Training should be planned in stages. Too much material presented at one time can be overwhelming. Three or four classes spread out over six months are much better than one lengthy class at the beginning of the implementation.
The presentation of materials in the classroom should evolve with experience. The second set of students should not see the same material as the first set. If they do, it means the trainers did not learn anything from the first class.
In-house documentation and training are generally superior to vendor supplied materials. Although time consuming to prepare, they can be tailored to a specific facility and they will be available over the lifetime of the product for refresher courses and new employee programs.
A follow-up plan of action must be in place once the initial installation is complete. This plan should begin with an understanding that initial training is barely adequate. It should incorporate the following ideas:
Regularly scheduled and productive meetings are part of any good installation process. These meetings should not stop just because the system has become operational. The meetings may change in nature; schedules, agendas and attendees may vary. However, periodic, well-planned sessions to review the status of a new system are every bit as important as the meetings that were held during the installation phase.
From the initial conception of the idea for a system and throughout the selection, implementation and useful life, maximizing utilization should be a primary goal.
It is not sufficient to select a good computer or even to pay a great deal of money for the "Cadillac" version. How a system operates once it is installed and how much users benefit from it will vary greatly from site to site. To maximize utilization of a system users must