1 Dr Ian Purves, 2 Mike Sowerby, 1 Robin Beaumont, 1 Bob Sugden
1 Sowerby Unit for Primary Care Informatics, Newcastle University
2 NHS Executive Primary Care PC1 (Prescribing
Branch), Quarry House, Leeds
This paper describes the methodology of a large NHS IT
R&D project in general practice: Prodigy. This description of the Prodigy methodology
is contrasted with the PRINCE methodology and the real life methodology of NHS IT
projects. The lessons from Prodigy, the PRINCE manuals and a discussion of the usage of
PRINCE in real projects are highlighted. The paper aims to open a debate on NHS IT project
management, user involvement, supplier involvement and evaluation.
In order to provide a credible test of the concept of
computerised decision support, the first requirement was the development of approximately
600 prescribing recommen-dations, each of which contains appropriate drug choices. They
were developed by a small professional group from sources including the British National
Formulary, the Drug and Therapeutics Bulletin, and existing guidelines. To ensure that the
clinical advice and therapeutic recommendations have professional credibility, an expert
group nominated by the RCGP, RCP, GMSC and RPS was established to validate them. The
advice and recommendations will be further developed, primarily in the way they are
structured on screen, during the trial in response to feedback from the doctors
participating.
The second requirement was to develop software packages
which could be integrated into existing UK 'host' computer systems. Five software
companies agreed to collaborate with this trial, producing software suitable for seven of
the systems commonly used by GPs. Again, this software will be developed in the light of
comments from participating GPs as the trial proceeds. If the trial is successful, one of
the outputs will be a system specification which could be included in future versions of
the Requirements for Accreditation (RFA) for NHS GP computer systems.
The project consists effectively of three successive cycles
of development, testing and evaluation (see below). Each phase of the project will assess
(i) how far the system and clinical recommendations are acceptable to GPs and patients at
the point of consultation, and (ii) the impact on their prescribing decisions. The
timescale for this phase of the project is too short to test health outcomes. However, the
clinical recommendations are, wherever possible, evidence-based (or where true
evidence-based guidelines are not available they are based on a consensus of expert
opinion). They will be further developed by the GPs in the trial: it is therefore likely
that modification of prescribing behaviour, in the direction of the recommendations, will
result in better health outcomes.
The NHS is one of most computerised health systems in the
world; it is particularly strong in clinical computing. More than 95% of general
practitioners are now computerised with a central NHS expenditure of £100 million which
makes up 50% of general practice
computing expenditure. There is an increasing growth of
computing requirements in the secondary sector with a current annual expenditure of £220
million, or 1.8% of revenue. However the more ambitious hospitals are spending 4% and it
is predicted most hospitals will move towards this figure. Further, only 50% of hospitals
are experimenting with clinical computing as opposed to administrative support and this is
embryonic. On average, 20% of hospital expenditure on IT comes from central sources, the
rest from local budgets.
Much of this dynamic activity is being driven by central strategic projects. However, despite being dynamic there are a lot of complaints being floated about:-
In the process of writing this paper we decided to poll
people's views. We presented some common statements about IT projects to the Internet
listserver GP-UK and asked for responses. We have appended these responses as an appendix
as we are anxious not to over-present this background information. However, we will use it
in discussion later.
The problems highlighted in the bullet points above have
meant that often individuals have been picked out within the press as 'scapegoats' or the
'responsible manager'. However, in the clinical community there are a lot of complaints
about the quality of deliverables, the running of projects, and the preponderance of
management consultants. PRINCE is the Project Management Standard for government IT
departments and it is PRINCE which is often criticised. However, PRINCE methodology is
for project management not for system development. This distinction is often
lost, and some of the confusion seems to emanate from the PRINCE documentation, in that
although it is a project management method it puts forward core components of products and
activities. These components are obviously appropriate but are not a detailed part of
PRINCE.
"In PRINCE, a project is regarded as
having a defined and unique set of products, a corresponding set of activities to
construct the products, appropriate resources to undertake the activities, a finite
life-span and an organisational structure with defined responsibilities.
A project is divided into a number of stages,
each forming a distinct unit for management purposes. Like the project, a stage has a
defined set of products and activities, a finite life-span and an organisational
structure. The production of these products, to the agreed quality standards, marks the
completion of a stage.
PRINCE defines the organisational structure of
the project and its stages, the structure and content of the project plans, and a
set of controls which ensure that the project is working to plan. These three,
together with the products of the project and the activities which produce
them, comprise the PRINCE components."
It is within this context that we started the Prodigy
project.
As a final part of the Introduction, we would like to
comment that the purpose of this paper is not to hit at sacred cows (PRINCE (?)). Nor are
we presenting a new NHS IT project methodology for general practice. What we have to say
relates to lessons we have learned while running a large NHS IT project which has
delivered its products in a novel way (for the NHS). In the process of running this
project we have had a lot of favourable comments from those we respect with regard to our
methodology and have had a lot of negative comments on PRINCE. We feel that we are in a
position to put a balanced view on this 'debate'. We hope to leave you and those that
commission NHS IM&T projects with a fresh perspective and hopefully a new debate on
the way forward.
From the outset we firmly believed that if we were to deliver the same product using several suppliers and deliver something that was usable, we would need both the suppliers and users at the centre of the project methodology. We also had very tight schedules, and having had experience of PRINCE projects, we decided not to follow the normal NHS IT project route. Of course this flexibility was possible because Prodigy is a Research and Development project. However, retrospective analysis shows that we still used the PRINCE structure but adapted it to suit our means. It has also become clear where some of the conceptual problems have arisen using PRINCE in the past. To enable people to see the issues and lessons we have learned we are deliberately going to use the headings of PRINCE components to present the Prodigy methodology. The PRINCE components are as follows:-
In the following sections of this paper we present PRINCE
in italics and the PRODIGY methodology within the same context in plain text.
The management style of the project has been a mixture of
formal and informal reactive communication. Its aim has been to be pragmatic and bottom-up
while having a top-down resource control. The management of timescales and risks has been
equally pragmatic and very effective. The project started with extremely tight schedules
and despite being complex has managed to deliver with very little slippage. The
organisational structure is shown in Figure 1:
Figure 1: The Prodigy Management Structure
PRINCE: The Project Board has overall control of a PRINCE project and has at least the below members:-
PRODIGY: The project management group has met formally roughly bi-annually. However the project manager has informally updated each member of this group on an 'as required' basis. The group consists of the following members:-
PRODIGY Tasks: Project Management Group
PRINCE: Assumes day-to-day responsibility for management
of the project throughout its stages.
PRODIGY: The project manager (MS) has taken responsibility
for day-to-day decision making and documentation of those decisions, supported by his NHS
Executive support team and members within the project team. Communication between the
project manager and the project team has been on a daily basis by telephone and internet
email. The email communication has also consisted of attached files. The project manager
and project team have a two-weekly face-to-face meeting. The project manager also took the
responsibility of negotiating with the suppliers to come to an agreement on work to be
performed, time-scales and payment. These agreements were monitored by the GP practices
and the project team and were reported to the project manager if difficulties were not
resolved by simple negotiation. Any problems arising were managed by the project manager.
PRODIGY Tasks: Project Manager
Project Team Leader (Stage Manager)
PRINCE: Has technical responsibility of ensuring that
stage products are produced on schedule, to high quality standards, and within budget. The
project manager and stage manager may be the same person.
PRODIGY: The project team is overall led by a project team
leader (IP) who is directly responsible to the project manager and is the individual most
in contact on a day-to-day basis with the project manager.
PRINCE: The stage team are responsible for conducting
the activities and producing the products of the stage.
PRODIGY: The work within the project team is managed by two
individuals: one (RB) from the evaluation side of the project and the other (BS) from
software specification side and project team budgets. However the team is a horizontally
managed structure and communicates frequently via email and includes all team members in
communications. The team also meets on a weekly basis to monitor progress and problems.
Clearly ad hoc meetings occur on a daily basis as individuals share rooms and the
same building.
The great strength of this project management style is that
those who are developing and using the prototype system are in control of the
deliverables. The methods of communicating with these groups will be mentioned in the
activities section below but it is worth re-phrasing them:-
GP Practices (n=137)
|
GP Suppliers (n=5)
|
The GP practices also have an algorithm to help them decide
how they should respond to any problems (see Figure 2). This algorithm gives the GP
practices access to the supplier, the Project Team and the Project Manager (MS).
Figure 2: The Prodigy practice communication algorithm
To date the project methodology has worked very well. The
only exception to this is that we underestimated the required resources needed to
communicate with 137 GP practices. This led to the practices not being as informed as they
should have been. We have now corrected this fault by adding a full-time communications
officer to the project team whose task is to ensure effective production of newsletters,
regular telephone (or email where available) contact with GP Practices and suppliers and
site visits.
PRODIGY Tasks: Project Team
PRODIGY Tasks: System Suppliers
PRODIGY Tasks: GP Practices
PRINCE: Maintains the continuity of project development activities and product integrity.
PRODIGY: We have developed an innovative way of dealing
with quality assurance that stems from being a heavily bottom-weighted project.
The role of BAC and TAC are within one member of the
project team (BS); however the quality assurance takes place by tightly negotiated
contracts with payment on delivery, monitoring of timescales by the Project Manager and
technical overseeing by the suppliers and NHS Executive IMC. We believe that the BAC and
TAC should be one person as it is easier to understand the separate roles. We also believe
that they should be part of the project team with BAC and TAC being part of their role to
ensure that this activity is core and continual. Of course adequate control and monitoring
should be in place.
The UAC is again a member of the project team (RB); however
the quality assurance is undertaken by the users involved in the project.
Clearly the Prodigy Team synthesises the opinions and views
of the suppliers and GP practices into a digestible form. This synthesis protects
confidentiality of those involved in the study as well pooling information. However
information is fed back to the suppliers and GP practices and over time project
deliverables will be scrutinised, therefore it is important for all to realise that
opinions and views should be faithfully reported. It is therefore inherent in the project
method that we have Quality Assurance group which is also the driving force behind the
project direction. The size of these groups means that any bias is diluted to become
irrelevant and that we have a truly bottom-up directed and quality assured project. It is
also vital to understand that the inherent involvement within the project of the suppliers
and users means that quality assurance is a major part of the project rather than
'cursory' reading of reports and meeting with individuals with 'day jobs' and otherwise
little involvement with the project.
PRINCE: A central Project Support Office within a
department may provide staff for the Project Assurance Team, and other support services
for projects.
PRODIGY: The project team takes on the role of the support
office although some tasks are delegated to the NHS Executive, to the Project Manager's
support team.
PRINCE: Technical plans are concerned with the products
to be delivered and with the activities necessary to ensure that the products emerge on
time and to the required quality standards.
PRODIGY: Plans were generated and monitored on Microsoft
Project.
PRINCE: Resource plans are concerned with managing the
finances and resources of the project. They are derived from the corresponding Technical
Plan.
PRODIGY: Resource plans were created in Microsoft Excel and
monitored against budget headings.
PRINCE: Action must be taken at the planning stage to
ensure that the project can deliver products of the desired quality. Testable quality must
be defined, agreed and recorded on the relevant Product Descriptions, and a testing
strategy adopted; quality review procedures must be established and review activities must
be defined and resourced.
PRODIGY: The project bottom-based structure ensures quality
assurance as mentioned under PAT. The Software Requirement Specification had within it
appropriate metrics. Interestingly it has been felt that the Software Requirement
Specification has stifled some innovation in the suppliers and we have the interesting
problem of writing the phase 2 specifications in a less concrete way but still enable
evaluation of deliverables to specification (perhaps a future paper). The details of the
plans are contained in this paper, the important issue is that quality assurance is
bottom-based, at the heart of the project and co-ordinated by funded daily activity.
PRINCE: The Project Board sets tolerances for Project
Plans and Stage Plans. These tolerances define the limits and cost within which the
project can operate without further reference to the Project Board or to the IT Executive
Committee.
PRODIGY: The Project Manager sets tolerances for Project
Plans.
PRINCE: Regular and formal monitoring of actual progress
against plan is essential to ensure the timeliness, cost control and quality of the system
under development.
PRODIGY: As PRINCE.
PRINCE: Controls are provided to be applied throughout
the project. These controls cover all aspects of project activity and, at the highest
level, allow senior management to assess project status prior to committing further
expenditure. Controls are applied via meetings of project management and project staff,
with each meeting producing a set of pre-defined documents.
PRODIGY: As PRINCE.
PRINCE: Quality and technical controls are applied to
specific products rather than to the overall output of a stage or project. The aim is to
identify and correct errors as early as possible in the development process.
PRODIGY: We monitored products at the end of each task.
Those products produced by the project team were also monitored on an ongoing basis.
However we quality-assured the computer suppliers' products at the end of the development
cycle. In retrospect this should have been on an ongoing basis with both project team and
experimental practice sites, as some suppliers had difficulties and others felt the
Software Requirement Specification limited their implementation. Future plans are to have
a looser specification, early 'story-boarding', focus groups of users and suppliers,
regular meetings, and conformance testing in both user sites and in the 'laboratory'.
PRINCE: Identifies a distinction between Management
Products (in the form of plans, reports and other control documents) and Technical
Products which are required by the end user and are defined at the start of the project by
the Project Board.
PRODIGY: The management products have already been discussed; we took the approach that these took up minimal time (approximately 5-10%). The main thrust of the project methodologies are to deliver the technical products; as follows:-
PRINCE: Again identifies a split between Management
Activity and Technical Activity which relate to delivering their respective products,
PRINCE highlights that a standard development method, such as SSADM, is used as they
contain structured breakdown of activities and products.
PRODIGY: Management Activity: The method used is very
similar to PRINCE and has been highlighted in detail in the above sections.
LITERATURE: Technical Activity: The development of software systems has changed over the years. Mumford1 described the different types of participation philosophies:-
An analysis of these philosophies highlighted a preferred
method of representative participation. In order to engage and inform the users so they
could participate, the method of rapid prototyping and joint requirement planning (JRD)
with joint application development (JAD) sessions were introduced2. Others
including Alavi & Wetherbe3 highlighted a need for the use of a mixture of
prototyping and modelling. Modelling has itself developed a lot since SSADM and object
methods such as Rumbaugh4. OMT are now leading the field. Connell & Shafer5
have linked rapid prototyping with object modelling techniques. Clearly there is also the
important issues of identifying user requirements and analysing whether or not the
prototypes are delivering what is required; the field of Human-Computer Interaction is
rapidly developing and the book by Preece et al6 gives a good summary.
In the domain of health care computing relating to decision support, Heathfield & Wyatt7 have published a recent outline of important issues to be considered while developing systems such as Prodigy. These issues are listed:-
More recently Lock8 has lamented the need for
evaluation of systems:
"Recent thinking suggests that this is
insufficient, and that a more comprehensive assessment should consider whether information
technology supports the commercial and strategic objectives of the organisation. For the
NHS, the impact of computer systems on patient care as well as on the 'business'
objectives of hospitals should be considered."
Donaldson9, in his editorial to Lock's paper,
emphasised these sentiments:
"Formal evaluation of major information technology
investments in the public sector, including the NHS, has tended to focus on criticism of
the implementation process - factors such as lack of design planning, poor project
management, inadequate staffing, and cost overruns - rather than on health care benefits
or their absence."
It is with this background that Prodigy designed its
technical activity.
PRODIGY Technical Activity: The methodology used a
combination of standard health service research and software development methodologies. It
is a 'user participative iterative design methodology', with an ongoing dialogue between
users and the development team leading to a series of successively refined prototypes. We
have constructed the project to actively involve medical practitioners at all stages,
while patient responses are also being highly valued, and will be incorporated in the
final results. The project clearly benefits from having stimulated this user-driven
aspect. We employ a wide variety of quantitative and qualitative evaluation techniques.
Although it has proved to be very successful in other sectors, this methodology is unique
in UK health care computing. Figure 3 shows the methodology used, with the shaded ellipses
identifying our current stage:
Figure 3: The Prodigy iterative methodology
The Prodigy sites were generally selected using a
purposively sampling method with the following criteria:-
Exclusion Criteria
However, one supplier gave a full user list and the 30
sites were selected by randomised mailings; we needed to mail 500 users to get the
required number of users and then had to use a follow-up purposively sampling method to
get adequate stratification (discussed later).
The comprehensive range of evaluation methodologies employed by the team is listed below:-
The experiences of the above methods will the subject of
future papers.
Project budgets follow surprisingly different patterns; the
classification of expenditure is equally variable to the extent that our chosen
classification proved impossible to map to other projects. However one classification
entity is generally mappable: that is, project management. In two other large, current,
general practice IT projects, the percentage expenditure is 50% of the total budget - all
of this is consumed by management consultants! Generally other projects spend very little
on involvement of users and suppliers; furthermore they do very little prototyping and
evaluation.
The relative expenditure in the Prodigy project is
presented below in table 1.
Table 1: The Prodigy relative expenditure
| Project management | 4.1% |
| Knowledge base development and maintenance | 2.9% |
| Software Requirement Specification and modelling | 2.2% |
| Involvement of users | 6.7% |
| User hardware upgrades | 17.7% |
Involvement of suppliers
|
22.1% 11.0% 8.3% 2.8% |
| Conformance testing | 0.4% |
| Rigorous evaluation | 19.7% |
The first thing of note is that project management ran to
6.9% of the budget, clearly nearer the accepted standard of 10% than most NHS IT projects.
The highest expenditures, both roughly 20%, were 'iterative software prototyping' and
'rigorous evaluation'. As important runners-up came 'User hardware upgrades', 'Software
support & training' and 'Involvement of users'. In this pilot phase 'Knowledge base
development and maintenance' was a low expenditure but would be clearly the major expense
if Prodigy proved a worthwhile product to the users.
It might be said that all we have done is apply
commonly-used project management and software development in practice. So what are the
lessons? Given the context of the current environment and the fact we are unusual in the
NHS domain we felt able to comment from this perspective.
One major issue is: What is the aim of the project? We suggest that most projects in the initial phase could use the following outline:-
Other lessons are now listed under their own headings.
Over the years IT projects have learnt that products should
be iteratively developed using both rapid prototyping and modelling. Modelling has moved
on from SSADM, and object modelling predominates; we prefer to use Rumbaugh with a CASE
tool - System Architect. However we have learned that the suppliers find penetrating
object models very difficult and therefore we have created simpler
"non-standard" ways of diagramatically presenting models for them.
It is clear that involving users and suppliers in IT
projects ensures that project delivers a product which is required and iteratively
developed by the suppliers who will be putting the system into the clinical workplace. The
lesson we have learned is that this is practical (in large numbers) and welcomed by almost
everyone.
At each stage of iteratively developing the prototypical system there is a need to evaluate whether the system is achieving its aims. The evaluation needs are different at different stages. In the early stages the following questions should be answered:-
Method: Focus groups / Questionnaires
Method: Focus groups / Questionnaires
Method: Story-boarding focus groups / Laboratory tests / Video ethnographic observation (pre/post)
Method: System log files / external process indicators
(pre/post) - these measures are strengthened by using time series methodologies with
population controls
General Methodological Note: The ideal is that initially
the study population is randomised; even if large numbers of users elect not to
participate the bias is still reduced. It should also be recognised that purposively
sampling may still be necessary to include the correct stratification within the user
base. If purposively sampling is undertaken then clear inclusion and exclusion criteria
are required. We used the two approaches of purposively sampling and randomisation. For
one supplier we sent out a questionnaire to identify inclusion and exclusion criteria and
to offer the practice to be involved. We sent out 300 mails to randomly selected practices
and got 30 responses but we then had to go on to purposively sampling. It is not clear how
pragmatic randomisation is at the moment for these types of studies but it is still to be
preferred until more data is available on the cost benefit of the strategy. If purposively
sampling is undertaken a questionnaire should be used to identify relevant bias within the
study group (other papers from Prodigy will identify these potential biases). These kinds
of interventions can not be blinded at the user level but evaluators should be blinded as
to suppliers and user sites until after the analysis. A positive Hawthorn effect will
clearly be present in the user sites; nothing can be done about this in this sort of study
design and should be recognised in reports. However, if controls are used for external
process indicators using controlled time series analysis then blinding the control group
will stop any potential negative Hawthorn effect.
Once the product has finished its prototype stages and
become a delivered product, then it is appropriate to study the intervention in a much
more rigorous way, probably with patient health outcomes and cost utility analyses. There
is a political (small and potentially large "p") and professional issue on
whether general usage of the product should now take place with a parallel effectiveness
trial, or an effectiveness trial should take place before general usage. The preceding
development methodologies do not give rigorous answers to the effectiveness question but
it may be felt to be 'politically' and professionally imprudent not to put a system in
general usage; in this situation those making the decision should be aware that later
effectiveness studies may be either positive or negative. Whatever decision is made we
propose a system similar to the CSM black triangle system is used for newly- designed
clinical computer interventions and they should be evaluated further.
General Methodological Note: Effectiveness studies
require that the study population should be randomised. Potential confounders should be
stratified (e.g. system suppliers). Again these kinds of interventions cannot be blinded
but evaluators should be blinded as to suppliers and user sites until after the analysis.
The positive Hawthorn effect will clearly be present. Using methods such as balanced
incomplete block designs can equalise and minimise for the Hawthorn effect.
We decided to evaluate and monitor training delivered by
the system suppliers. This was a useful lesson as we found a wide variability in quality.
For future stages we intend to design the training and evaluate its effectiveness in
collaboration with the suppliers. This gives the added advantage that should the
prototypes become a usable product then the training has been developed and iteratively
improved along with the system.
We are at a pivotal point in implementing evidence-based
practice / decision support. As such there are lots of lessons to learn for the future.
Specifics of the technicalities of knowledge base change control will be covered in future
papers. Generally speaking modelling comes into its element here - a good model is vital.
One of the biggest lessons is that the interactions between the knowledge authors and the
system specifiers need to be very close and interactive. Furthermore the team should
always be looking to the future and to mechanisms for continual development and
maintenance should the system become a practical product. Clearly in the earlier stages of
the project informal consensus methods10 are appropriate but more rigorous
methods11 should be used if the product becomes more generally available.
In review we have found PRINCE to be very similar to our management methodology. The only difference is in the way we have performed our quality assurance. We feel that the PAT is probably one of the weakest points of PRINCE as it generally uses a consensus or consultative participation method rather than the representative participation method we used (see above). However one point that is regularly made is that most problems with PRINCE relate to important issues:-
Project Communication
Projects such as Prodigy are dependent on constant and free
communication which is mainly informal. Formal communication and documentation have a role
but it should be secondary to general informal communication. Asynchronous techniques such
as email are vital in busy lives and all projects should depend on freely copied email.
Quality assurance should be a core activity to any project.
A common problem with Project Assurance Teams are that they are poorly funded and the
individuals involved end up have large piles of papers dumped on their desk some weeks
before a PAT meeting. The quality of their assurance is the best that can be achieved in
these situations. We feel that by having quality assurance co-ordination as a key part of
the project team's work and a large user and supplier project base is to be recommended.
It means that quality is daily activity, it is adequately funded and the quality assurance
is being done by those who will be using the system rather than their representatives. The
risk of self-interest within the project team is minimised by the size of the quality
assurance and the fact those who are quality assuring the project are also participants.
As can seen in the Appendix, PATs can be used in heavily managed PRINCE projects to keep
things on the rails and deliver 'high quality' products; which in reality should have been
stopped or need reworking before implementation.
In the first iteration of Prodigy we had a reasonably tight
System Requirement Specification. This enabled reasonable bench testing and fairly
rigorous conformance testing. For phase 2 we are going to loosen the specification but
tighten standards with the hope that this will encourage innovation. Furthermore we
discovered that in phase 1 the bench testing was very useful but did not always take into
account how the system was used in practice. Therefore the lessons we have learned are
that for the future it is going to be harder to bench test the systems in a laboratory so
we are to undertake the conformance testing in GP surgeries and in video focus group
interaction with the systems in a laboratory. Clearly one man, one day (often a couple of
hours), a bit of paper with weak metrics and a computer is worthless for conformance
testing.
As worthwhile comment, Ewan Davies in July 1996 posted the
following on the GP-UK listserver:-
"I'm not sure there is a place for the RFA in the future. If there is it needs to limit its scope to three areas - gross functionality, standards and safety:-
Whatever is in the RFA there MUST be a business case
that justifies its cost to the NHS and suppliers. You can have any feature you can think
of in your system as long as you are willing to pay for it. The consultation process needs
to be more transparent. RFA has no place as a wish list. It should not be used as a tool
to create homogenous systems: UK GP computing has gain greatly from diversity and
competition."
We generally agree with Ewan's sentiments and have
presented them as we feel they add to the debate, but there are issues with which we don't
fully agree and need further discussion. Clearly we need to discuss the whole role of NHS
IT Projects, prioritisation of user requirements, conformance testing, commercialisation,
and the Requirements for Accreditation for the future.
The above points are the lessons we have learned in Prodigy. As part of writing this paper we consulted widely over the internet about PRINCE and the appendix was the result. The only outstanding issues raised as lessons from the internet discussion which we did not identify within Prodigy, but clearly others have within their projects, are as follows:-
We have presented to you the workings of a precocious child
and compared them with those of an aristocrat. As in life the child still needs to develop
and the aristocrat is mature but can still learn. Interestingly the sacred cow is not the
aristocrat as it has a different spiritual meaning for everyone; in fact it is not clear
what the sacred cow really is! We believe that the precocious child will develop into a
fuller being than the aristocrat whose role is still there but perhaps better understood
with a clearer defined role.
We hope the learning and debate will continue.
1 Mumford E. Designing participatively. Wright Ltd, Sandbach, England, 1979
2 Martin J. Design and Construction Vol III (Information Engineering). Prentice Hall, Englewood Cliffs, NJ, 1990
3 Alavi M, Wetherbe JC. Mixing prototyping and data modelling for information system design. IEEE Software 1991 (May):88-91
4 Rumbaugh J, Balaha M, Premerlani W, Eddy F, Lorensen W. Object-Oriented Modeling and Design. Prentice-Hall International Inc, Englewood Cliffs, NJ, 1991
5 Connell J, Shafer L. Object-orientated rapid prototyping. Prentice Hall, Englewood Cliffs, NJ, 1995
6 Preece J, Rogers Y, Benyon D, Holland S, Carey T. Human-Computer Interaction. Addison-Wesley Publishing Company, Wokingham, England, 1994
7 Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision support systems. Meth Inf Med 1993; 32:1-8
8 Lock C. What value do computers provide to NHS hospitals? Br Med J 1996; 312:1407-1410
9 Donaldson LJ. From black bag to black box: will computers improve the NHS? Br Med J 1996; 312:1371-1372
10 Grimshaw J, Eccles M, Russell IT. Developing clinically valid practice guidelines. J Eval Clin Prac 1995; 1:37-48
11 Eccles M, Clapp Z, Grimshaw J, Adams PC, Higgins B,
Purves I, Russell IT. Developing valid guidelines: methodological and procedural issues
from the North of England Evidence Based Guideline Development Project. Quality in
Health Care 1996; 5:44-50
Many individuals helped put this paper together: they all
go unrecognised as we were not able to get a reply from them all in time for the deadline.
Some were willing to be attributed but as they had not seen the paper, and they were in
the minority, we felt it unreasonable to name them. Thanks one and all.
While writing this paper we posted an e-mail to GP-UK
explaining what we were doing and asked several questions (those highlighted in bold
below). The response to the questions and other comments are listed below. There were 12
respondents.
"Yes, I have seen this all too frequently. Not through
lack of will to do the coordination, but through the lack of means and competence. The
Health Care Modelling team have been asked to help mitigate this problem (although they
can only be part of the solution)."
"I have seen this happen, and it's not a fault of the
methodology. The Service is (of necessity and funding) tactically oriented, so when there
is a need to be strategically or holistically focused to include a number of inter-related
projects, it isn't able to take the wider picture - the classic case being the NHS
IM&T national infrastructure projects, which are run separately, although
"regular co-ordination meetings take place" ;-))"
"Unfortunately this does tend to happen. The
difficulty is that projects can be committed to delivering specific products to external
timescales set without due regard to the complexities (a) of building the products (b) of
interdependencies. If the PID has been drawn up properly, interdependencies and risks
should have been identified at the outset, and appropriate control measures put in place.
To be effective, these *must* be monitored and updated, and where they change proper
impact analyses done with exception plans to update project plans. A weakness is that
changes in risks and interdependencies may not be (perhaps cannot be) allowed to change
the delivery timetable."
"Try replacing "suppliers who have to
deliver" with "users and clients who separate the project from the dynamic of
their business". I have, on many occasions, had to tell Project Boards of ancillary
issues which need to be addressed - but as it was all too difficult, and the situation was
not addressed, making it difficult for the supplier to deliver. My worry is that it's
often seen as them and us - when it should be (and is in the most successful projects) we.
An interesting comment made by one Chief Exec to me once was that when suppliers say
"we want this to be a partnership", ask them if they're prepared to share the
risks, as well as the rewards, which is what a true partnership means - but the
client/user also has to meet their end of the bargain, which usually means making a far
higher level of commitment than most Health organisations can, or are prepared, to
do."
"Supplier involvement seems to vary enormously. From
'our' side of the fence I would say that suppliers often fail to take into account the
goodies that projects are offering and would prefer to go with existing proprietary
products that have been slightly tweaked in the direction of the project products."
"This can be due to confusion or lack of definition
over what precisely the project is supposed to be delivering. Specification projects (are
you perhaps thinking of RFA?) can get bogged down in technical details relating to
possible solutions when they should be concentrating on what are the user and business
functions to be satisfied. It should be left open to suppliers to propose the most
appropriate (and pragmatic) solution that meets those functional requirements. Specific
technical standards should only be included where there is a clearly identified reason for
doing so."
"CISP and DISS and DISP all produced working systems.
Many others produced paper products. Horses for courses."
"If this is the case, then surely caveat emptor
applies?"
"Yep"
"In the context of IMG projects this is almost
invariably the case. I think this shows up another weakness of PRINCE in this context; it
was primarily designed to cope with the design of operational systems. It can be applied
effectively to specification projects, but it needs to be closely coupled with a
well-defined and well-controlled configuration management system so that once a
specification product is baselined it does not change without a proper impact analysis
involving input from *all* affected parties being done."
"The CBS development projects, DIS (Data Interchange
Standards), Clinical Terms and AR all had (or have) very heavy user involvement. I would
say that user input is usually pretty good and is listened to."
"I think the problem is often twofold - the
methodology runs the project, instead of being its servant, and the client staff involved
have insufficient experience of their role and associated responsibilities. The most
successful project I managed had a Board Chairman who said "I'm not interested in the
process - that's for the auditors, and the front-line staff; I work on an exception basis,
and like to keep things simple, so in our monthly meetings I want five things - are we
ahead or behind schedule; are we under or over budget; what are the issues in order of
criticality; what do *we* have to do to fix the problems; and I want this documented on
one page." This approach provided a very sharp focus, and kept things
simple....."
"The problem here is defining the users properly, and
defining a proper mechanism for capturing their input. As referred to above, I think there
is a tendency to do this by having "user" representatives on project boards.
This I think does *not* work because the project board is too senior and not close enough
to the detail project work to be able to work with the project effectively. In the case of
IMG projects, these tend to be strategic and can be somewhat remote from the real
end-users in the NHS, concentrating perhaps more on what the centre would like to see
happen rather than on what the people doing the job in the NHS want to happen. This can
create a tension in project activity. Users can (and should) play an important part in the
QA process as reviewers. An important part of the UAC's role is to ensure that this
happens; UACs should be backed up by the user member(s) on the board to ensure that this
is done. The PRINCE methodology does not really make adequate provision for user
representative panels, partly because of course it's designed for developing operational
systems."
"IMG seems unable to escape its dependency on
consultancy firms for doing much of its work. Use of consultants has, for senior managers,
the personal advantage of distancing themselves from difficult decisions, but in the EDI
area it brings some big disadvantages to the service as a whole: the big-firm consultants
usually don't understand either the clinical business or the technical issues of EDI very
well, but are always very hot on project management-speak.
"So they spend a lot of time with the people in the
service who do know, and regurgitate the pickings to the [project] manager. Something is
always lost in the listening/regurgitating through, and the report needs correcting.
Nevertheless the consultant seems to have (with 2 days' study) a higher credibility than
the person who's made the matter a lifetime's work. The net effect of this is to make
expertise held within the service invisible to those at the top of it, and blunts the
decision-making abilities accordingly. In addition consultants need (to safeguard their
position) to interpose themselves between key participants in any project, and sometimes
this corrupts the dialogue.
"Some quite big problems have occurred from the
activities of consultants in the 'Dataflows for Fundholders' and 'NHS-Wide Networking
Projects', and would also have arisen in the 'GP-Provider Links Project' without a
concerted effort by most of the Project Board. MIG needs to consider how IMG might escape
this destructive pattern of behaviour."
"The methodology is good at forcing
exceptions/highlights & reports to appear but awful at telling you what's actually
happening in the project. The project leader & board chairman usually run things &
everybody else has peripheral involvement only. This means megabucks get spent & the
board meets 2-4 times per year to rubber-stamp decisions already made (or at least
directions already charted). Too much of the work is left to external consultants who make
loadsa money by constantly reinventing the wheel. Very frustrating. PRINCE does not force
a strategic perspective with other similar linked projects with the result that there are
several overlapping IMG projects I've been involved in - where I seemed to be the only
person around who knew about the other projects … So as a mechanism for
accountability, reporting & project planning, it's OK, BUT a huge amount of time goes
into producing all the different types of reports the methodology requires. PRINCE is not
democratic - it allows too much power to the project chairman & leader & lacks any
strategic input or influence."
"Those that do the jobs well usually contribute well
to successful projects & vice versa (i.e. I suspect that project success varies more
with the quality of people assigned to it than with the project management
"methodology" adopted)."
"Oddly, I've come to the conclusion that methods are
necessary but not in themselves sufficient to rise above the 'personality' factor. A
blinkered set of principals makes for a blinkered project etc."
"My view is that the single biggest problem we have is a lack of people with the skills to manage these projects. There are three main things required:
No-one can be all three of the above. However, the manager
of the project should have 1. plus either 2. or 3. and should be able to liaise very, very
well with whoever fulfils the one of 2. or 3. that they don't have."
"I believe that used properly and applied
intelligently PRINCE is a very powerful methodology. I think that it doesn't always work
as well as it should partly because the key people involved in PRINCE projects do not
always understand their roles, and partly because PRINCE methodology is often perceived as
a bureaucratic overhead, that by writing a PID and the odd project manager's report you
are somehow magically complying with PRINCE."
"My initial thoughts are that if PRINCE goes wrong, it's because:
(a) someone is using it slavishly, hoping the method alone will be enough, and not paying enough attention to the contents / products of the project:
(b) within IMG, there is little or no effective control above Project Board level, and the business case, if one is prepared, is weak. This is where the relationship between the project and its context should be examined and taken into account. Neither is control usually exercised by the ultimate purchasers/users of the system, because they do not usually fund development (altho' they increasingly do have to fund the implementation).
(c) the initial definition of the project and its products are poor, often because the project is initiated by bureaucrats, not users, who are involved later (sometimes, it seems, only because PRINCE requires it). The 'users' are often also merely people working in the domain, not people committed to using/paying for the product.
(d) quality assurance is not done thoroughly. Not enough time is allowed for it, and QA comments are not really resolved.
(e) the project owners (especially civil servants) are more interested in avoiding failure rather than achieving success. Thus exception conditions are hushed-up or down-played, rather than dealt with honestly. The criteria for 'success' are therefore primarily based on completing the project to time and to budget, rather than on the quality of the products themselves.
(f) yes, the project products are defined in isolation, without sufficient awareness of related work going on or completed elsewhere. This has definitely been true within IMG, but is a general problem - communications within large organisations are difficult, and there may be few / no mechanisms for taking an overall view. That's why generalised business modelling is important - it provides a backdrop for doing this. Also if project owners are mesmerised by budgets and timescales, there is little chance of being able to apply mid-course corrections if they are discovered to be necessary. This was what happened in the AR project.
People have said that the method itself is cumbersome, but
I would not agree. Used with a light touch, and without Cecil B de Mille-sized project
management staffing, the results can be good, viz AAGP and the CBS COSS study. PRINCE is
nothing more than codified common sense. But the documentation about it is largish, and
the people who use it often have no specific training in it. So first-time users tend to
be in awe of it, and hope that by going through the PRINCE motions, all will be well. It
won't."
"The good thing about PRINCE is that it can be
tailored to meet your needs, and can be as simple or comprehensive as you need. I guess
the single most important question should always be "What are we trying to do
here?", and "How will we know we've been successful?", with all that means
in terms of its answer."
"Creative ways should be devised to persuade suppliers
to write software now and to reduce the subsequent support costs so that the total public
expenditure is at least no higher. Inconsistencies already exist. For example the
Executive's funding of (a) software development for GP Fundholding (b) GP-FHSA links, (c)
Prodigy, and most damaging, (d) EDI for contracting data show that the principle [of not
paying the suppliers] has not always been adhered to. A further reason for applying some
funding (not necessarily direct to the suppliers) at the time of software development is
that it allows a greater lever for professionally-led quality control of the software
produced. There is some evidence that the suppliers would welcome this as it is cheaper to
rectify problems at an early stage in design/development than when a product is found
faulty in its use. Such user-led review would also bring them closer to their
customers."
"PRINCE does involve an overhead - a very necessary
management overhead. What it can bring is a comprehensive check-list of all the things you
have to consider in delivering a successful project, together with procedures for ensuring
issues and interdependencies are properly addressed.
It has to be applied intelligently. Full PRINCE procedures
(with project boards, PATs, product descriptions, exception reports and the rest) may not
be appropriate for very small projects (e.g. with budgets of less than £100K). This
doesn't mean there isn't benefit from applying the philosophy of PRINCE though. It doesn't
always seem to work very well with "soft" deliverables, where the end-product is
(for example) a strategy document or consultative report. An important element of PRINCE
is obtaining a clear sign-off of the end products by the "user" against the
pre-defined product descriptions. If the project is unclear or vague about who the
"users" of the products are, or what the products are intended for, then you
have problems in deciding when the project actually ends.
"Related to this is the problem of applying PRINCE
methodology to extensive programmes of work, consisting of several complex projects each
operating under PRINCE methodology. This tend[s] to manifest itself by inflated
project/programme boards with a range of user representatives for the diverse interests
involved. To be effective the project board needs to be small enough to take proper
executive decisions; if it gets above more than about six in number this can get very
difficult. The project board is not the place for user input and consultation; that is
more properly done through the PAT. The role of the PAT tends to be understated. The PAT
has a very important role in PRINCE; its role is to check finished products against their
product descriptions and report to the project board. A weakness is that it if PRINCE is
not applied properly it degenerates into an advisory body. Formal QA meetings are an
important mechanism at which problems should be formally logged and equally formally
signed off as and when resolved.
"Understanding of project roles is important for those
taking part in boards and PATs. The board member's role is to direct the project;
membership of the board should therefore be restricted to those with direct responsibility
for the implementation of the project. The co-ordinator's role is to ensure that all user,
technical and business issues affecting the project are brought out. The coordinators are
responsible to the project board for ensuring (a) that product descriptions are complete
and contain proper quality criteria; and (b) that products are completed to those
descriptions in line with the criteria. Co-ordinators should liaise closely with their
respective board members over project issues; unfortunately there is a tendency (because
of work/travel commitments) for them only to meet at formal board meetings."
"IMG is stuck with its "I've started so I'll
finish" attitude. By the way, this is not a flaw of the project management method,
but of the people doing the managing. How many IMG projects do you know that have been
'pulled'?"