"Performance indicators: do they improve the effectiveness of development assistance?"

09/11/2009

Performance indicators are increasingly used to measure the effectiveness of public policies. Inspired by benchmarking techniques currently used in the private sector, they stem from the idea that to survive in a competitive context, each country must continuously test its organization by the importation of "best practices" currently used by its "competitors" (what Professor Alain Supiot calls "the dogma of universal competition"). Thus, countries are ranked through a battery of quantitative indicators, refering to a normative implicit reality that takes the form of targets in the context of Results Based Management (RBM), which imposes on the " less efficient" continuous improvement of their scores. These indicators are increasingly used by funding agencies to serve as a basis for providing development assistance through a change of strategy regarding "Project support" towards supporting budget in accordance with the Paris Declaration on the effectiveness of development assistance from March 2, 2005.

What are the objectives of such measures of performance? What is the relevance of,such indicators considering the results on the field? We try to answer these questionsthrough three lines of analysis, namely the challenge of appropriation of budget aid (I), the achievements and limitations in the use of indicators (II), and the problem of the autonomy of countries receiving assistance in the conduct of their policies (III). We llustrate the analysis with examples drawn mainly from the Strategic Framework for Growth and Reduction of Poverty (CSCRP) of Mali.

I. The challenge of appropriation of budget support

The objectives for budgetary support are designed to enhance efficiency through
appropriation and more responsibility of recipient governments, as well as alignment with national procedures and the mobilization of resources at lower cost. In addition, the system of budget support is supposed to help not only to disburse large amounts with less bureaucratic constraints, but also to strengthen the coherence of government policies in recipient countries. In practice, there are overhangs (A), but also room for improvement (B).

A. Progress in the implementation of budget support
In Mali, the budget supports are taking place under the Framework Agreement (GBS) signed between the Government and funders in March 2006, and through specific arrangements in areas of education, health and decentralization / institutional development (sectoral budget support). These, respectively, signed July 18, 2006, July 19, 2006 and September 11, 2009, condition disbursement of funds in the sectors concerned to achieve quantified indicators.
This means that the Malian experience in this area is very recent compared to the
countries of Southern and Eastern Africa (Uganda, Tanzania, Mozambique) who are the
pioneers on the continent. Despite this insufficient hindsight, the Malian experience is
viewed with interest thanks to its efforts of alignment with national procedures
(programming assistance based on CSCRP), but above all because of the appropriation
and mobilization of resources, particularly those of the European Development Fund
(EDF)6. There is a consensus that the process of managing Project support is largely
controlled by the funding agencies. In addition, it removes responsibility from the
recipient governments, increases the mobilization of resources and increases
transaction costs.
However, if we put aside the fact that funding through budget support is criticized by
non-state actors, as far as their own activities are concerned, because it goes through
procedures administered by national governments,7 there is a consensus to recognize
its advantages, especially its ability to mobilize maximum resources and a beginning of
alignment with national procedures8.
Nevertheless, despite these advances, there are some margins for improvement.
B. Margins of improvement in the implementation of budget support
In terms of harmonization, it is true that there are important gains in the internal
coordination of funding agencies, including through the thematic groups, and much work
remains to be done on a national level to coordinate action internally in order to set
priorities in a dialogue with funding agencies. This is explained by the weak institutional
capacities of administration that - it must be said - does not meet the conditions of
management by results. In this regard, one can only regret the fact that in the 1980s,
these countries, under the Structural Adjustment Programs (SAPs), had been led up to
"cut back" so severely public administrations, which in fact were already flimsy, and
5 On the choice of indicators in budgetary support, see: Introduction of performance indicators in supporting the
Structural Adjustment Programs in ACP Countries, Study commissioned by the European Commission, Summary
Report, Vol.1, CERDI, June 2002
6 On the evaluation of development assistance by civil society in Mali visit: Federation of NGO community in Mali,
Workshop Report: Official Development Assistance, an engaged civil society, Bamako, July 2007.
7 The fact that national procedures for managing budget support are controlled by the governments of recipient
countries, is seen by NGOs as an obstacle to their independence.
8 Not only many funding agencys continue to use project support through their own management units, but even
among those who opted for budget support, many are spending part of the assistance through institutional support,
managed by their own procedures, which increases transaction costs.
Traduction libre Constance Cournède IEA 3
multiply the units of management programs, which not surprisingly has not helped create
a stable institutional environment. In Mali, for example, any administrative reform during
those years had as guiding principle "the control of structures and staffing," culture that
still marks the Malian government. During this period several cells of Planning and
Statistics (CPS) within ministries were closed, thereby weakening the ability of countries
to develop sound policies based on reliable statistics, all things that the current
Government, encouraged by the funding agencies, are correcting in recreating these
services which are now playing a central role in supplying indicators. Despite this and
despite the progressive introduction of Medium Term Expenditure Framework (MTEF),
most governments can not make firm commitments beyond the budget cycle, because
the planning and budgeting are separate processes under different administrations with
often divergent interests.
Furthermore, if there is a beginning of alignment of some funding agencies on national
procedures, the coexistence of both types of aid, sometimes within the same programs
with the same funding agency, imposes to national administration heavy constraints in
terms of reports, reducing the time spent on providing services to people, which is, after
all, the purpose of this aid.
II. The achievements and limitations of performance indicators
Results based management is centered around setting targets for national policies and
measuring their results. It incorporates principles of so-called "good governance",
including clear and measurable objectives, a decision-making process based on
evidence, transparency and continuous improvement.
Proponents of this approach see it as a way to bring the governments of developing
countries accountable to their citizens and to the funding agencies who fund some of
their programs. To link actions to results and make more systematic monitoring of
performance, they establish chains of results through logical framework linking
objectives, activities developed in order to achieve them, the resources mobilized, the
results achieved (compared to what was expected) and the impact obtained.
This is where the famous indicators designed to measure objectives in relation to results
step in. In fact are they relevant?
The analysis of the situation on the ground shows that there are achievements (A), but
also limits (B) in the use of indicators.
A. The gains in the use of performance indicators
The analysis of the situation on the ground shows that this exercise is of some use in the
conduct of public policy, at least for some indicators. This is true for example for several
indicators (reforested areas, percentage of population which have access to drinking
water9, immunization coverage of children under one year, rate of assisted delivery, rate
of prevalence of HIV-AIDS, rate of budgetary devolution, rate of staff decentralization,
9 In the Global Report on Human Development 2007/2008 UNDP, the Human Poverty Index (IPH1) evaluated for Mali
in 2001 to 58.2% decreased, thanks to achievements in the access to basic social services ,especially access to
water having improved a lot.
Traduction libre Constance Cournède IEA 4
etc. ..). Indicators of devolution (budget and staff) that measure the efforts of
Governments in their commitment to devote a growing share of the State Budget to
regional and subregional service, and to assign more personnel (thus closer to the
people ) seem particularly relevant when we know that in 2002, in Mali, on average
nearly 80% of the state budget was spent in the capital city and on average nearly 50%
of country officials worked there10. The use of these indicators has led governments to
provide significant effort of budgetary devolution in recent years, the devolution of
human resources appearing to be much more problematic because of the reluctance of
officials to serve in the region, particularly in those of Timbuktu, Gao and Kidal,
considered as difficult. However, the tree should not hide the forest. You can not
manage all aspects of public policies with indicators.
B. The limits of performance indicators
The analysis of the situation on the field shows that there are some limitations related to
the adverse effects of certain indicators (1), difficulties of interpretation (2) and
measurement of values (3), the issue of a harmonized methodology for collecting and
processing data (4), and the choice of priorities and target values (5).
1. The perverse effects of performance indicators
With a closer analysis, we realize that using multiple indicators produces effects that
sometimes go against the stated objectives. This is so with all the indicators designed to
measure the budgetary effort for certain priority sectors. Thus, although the indicators
that measure the share of national budget indeed devoted to education (25.7% in 2008)
and health (11.6% in 2008)11 encourage a further concentration of budget at the level of
these sectors, the fact remains that they have the disadvantage of encouraging the
concerned administrations to spend budget allocations in activities which contribution to
improving the academic status or health of populations could seem questionable.
In the field of justice, if the indicators on the number of judges per capita and the number
of justice infrastructure (courthouses, prisons, homes of judges, etc..) built or
rehabilitated, give an idea of the budgetary efforts for the sector, it remains clear that
they do not inform on the quality of justice, its independence, still less on the satisfaction
of litigants. It must be said that the combined use of these two indicators is likely to push
governments to concentrate their efforts on the recruitment of magistrates and the
construction of courthouses, to the detriment of all essential issues related to fair
earnings for judicial staff and to difficult reforms for the raising of moral standard in the
judiciary system.
10 Commissioner for Institutional Development (CID), Institutional Development Program, 2002, P10.
See also: Mission Decentralization and Institutional Reform (MDRI), Administrative Decentralization in Mali: overview
and definition of a national program to strengthen devolution and support decentralization, 2000
11 Ministry of Economy and Finance, 2008 Report of implementation of CSCRP, May 2008, P11.
Traduction libre Constance Cournède IEA 5
In addition to the issue of absorption capacity and quality of public expenditure recorded
in these sectors (which can be described as concentration of Budget: Education,
Health), the use of this indicator leads each ministerial department to analyze its budget,
not in terms of its operating needs, but now in terms of percentage of the national
budget, which, and it is already evident, create bidding wars without end around
budgetary decisions. In this regard, we observe in the latest generation of Documents
about the Strategy in the Reduction of Poverty (DSRP), a shift in priorities towards the
infrastructure sector, and since the vertiginous rising of the prices of grain on the world
market , agriculture, yesterday abandoned, begins to go high on the agenda. If, in this
case, we can only welcome this development, in general, the permanent changes of
priorities, illustrate the fragility of choice when not worn by the peoples concerned. The
succession of reforms sometimes abandoned before being completed, often depending
on the international agenda, without consultation with interested countries, constitutes a
real source of instability of public policy in developing countries, Africa appearing in this
regard, as the field of all experiments.
In the field of education, although the indicators in relation with the enrolment gross rate
in fundamental cycle 1 (80% in 2008)12, and the completion rate of the fundamental
cycle 1 (53.3% in 2008)13, have been vectors of universal schooling, the fact remains
that each has adverse effects. Indeed, given the practice on the field, it is reasonable to
think that the idea to raise the enrolment rate made administrations less fussy in the
supervision of schools like community koranic schools and madrasahs which also
contribute to the recovery of this indicator (up to 11% in 2008)14. This same concern has
led almost all African countries since 1990 to recruit teachers overwhelmingly lowskilled,
with serious consequences on educational systems, especially since most of
these non-professional teachers have great difficulty in acquiring the ethics of the
profession. But the paradox of this situation lies in the fact that it was indeed the
structural adjustment programs that had led the authorities in some countries, including
Mali, to close most schools for the training of teachers and to conduct a program of
voluntary departures proposing early retirement in the 1980s, creating a sudden
shortage of qualified staff in the education sector15. When in the 1990s, priority was
given to social sectors, not only was there more than enough professional teachers
available on the labor market, but it encouraged the recruitment of contractual staff
underpaid. Moreover, in order to control the wage bill, in many African countries classes
are functionning with over 100 students, or with so-called dual systems division and
double shifts16. Today, the consequences of these contradictory policies on the quality of
education are such that everywhere in Africa these policies should be questioned again
12 Ministry of Economy and Finance, 2008 Report of implementation of CSCRP, May 2008, P33.
13 Ibid.
14 Ibid.
15 Certainly, in principle, teachers and health workers were not eligible for the program to preserve these social
sectors, but in fact they were the most affected because their personnel lived under relatively more difficult conditions
than other officials.
16 It was a system that due to shortage of staff was operating two classes held by the same teacher at the same time
in two rows in the same room (double division) and two cohorts of the same class held alternately in the morning and
the afternoon by the same teacher (double shifts).
Traduction libre Constance Cournède IEA 6
in order to give a minimum of coherence. The projections announce a workforce of
around 180 000 students for the University of Bamako in 2015, when higher education is
by no means the priorities followed by indicators!
It is also the use of completion rates in fundamental cycle 1 who has led the Ministry of
Education to issue directives to schools to enable a maximum of pupils to moving into
the upper classes to complete this cycle even when they were well below the necessary
level, which to many observers is a major weakness in the Malian education system.
2. The challenge of interpreting the indicators
Furthermore, some indicators should be handled with care and should be analyzed by
crossing them with some other data. Thus, even if the pupil-teacher ratio of fundamental
cycle 1 is relevant in principle, in that it gives an indication of the level of supervision of
students, the fact remains that it is still relative since in schools that concentrate the
most teachers (in fact in the cities) many teachers are used more in school
administration than in the classroom, or are overstaffed. It is also not surprising that this
indicator is not indicated in the reports on the implementation of CSCRP.
In health sector, the indicator on the percentage of population living within 5 km of a
functional health centre (58.2 in 200817) doesn’t give enough information on the actual
accessibility of health care. It is well known that because of the destitution of centers in
drugs and often in medical personnel (as soon as we are away from major cities),
people see only a very limited interest to attending, preferring, particularly for the
poorest, to use traditional medicine (when there are still trees!), or worse go to "street
pharmacists" real quacks who prescribe counterfeit drugs. It is clear from the report of
implementation of the PRSP in 2007 and 2008, that barely one in three residents
attended a health center18, even if it is rare for a person living in Mali not to suffer at
least once in a year from health problems, especially malaria, a disease that is prevalent
at an endemic level. In truth, health in developing countries who do not have a system of
compulsory health insurance, with the poor population in majority (human poverty index
from 56.4% in 2008 in Mali19), the real challenge remains affordability.
3. The difficulties in measuring indicators
Although relevant, some indicators are not easily measurable. For example, in the field
of social security, the percentage of population covered by a mutual insurance company
and that of the poor supported medically. It is a real concern since the safeguards of
traditional social protection are being weakened, and thus play less and less the role of
social shock absorber for the poorest. However, it is difficult to gather reliable
information on mutual insurance companies, their members, and the actual benefits
offered because it is an informal sector which operates in a legal and institutional
inappropriate frame. In such a context, how to build a new social contract imposing
17 Ministry of Economy and Finance, op. cit. p. 35
18 Ministry of Economy and Finance, op. cit. p. 36
19Global Report on Human Development 2007/2008 UNDP.
Traduction libre Constance Cournède IEA 7
reciprocal obligations on families, communities, local governments, civil society and the
state to ensure access of the poor to social protection? What efforts for dialogue,
reconciliation and openness between the actors in order to renovate the approaches
and means of action against the realities of the sector? Indicators do not give
information to all these questions.
4. The risks associated with international comparability of indicators
Sources of information are based on methodologies whose international comparability
(which is underlying the choice of these indicators) is sometimes far from assured. Thus,
the 2008 Report of implementation of CSCRP indicates an unemployment rate of
7.7%20, which does not fail to surprise, when compared with the situation in
industrialized countries. But when you consider that this figure21 takes into account
people of working age to meet simultaneously the following two conditions (not having
economic activity during the reference period or not having kept a formal link to
employment and being available for employment), we understand that it hides a massive
underemployment of over 80% in most African countries. The vast majority of people
regarded as not affected by unemployment are indeed active in traditional agriculture or
in the informal sector and work only a few months a year. In reality the expansion of the
informal economy is one of the most significant changes in the labor market in Africa
over the past two decades. The employment structure has been significantly
transformed by shifting dynamics of formal sector employment into the informal sector,
which has become the reservoir of the urban workforce. The ability to adapt to the
realities of the labor market and the ease of entry and exit made it attractive for
entrepreneurs in particular as regards sensitive groups like women.
5. A difficult consensus for building around indicators
One of the challenges of results based management, is certainly the definition of
relevant and legitimate objectives. The consensus in this matter is a challenge because
of the difficulty of defining priorities. If, in a company, a consensus can be obtained
without much difficulty over the choice of goals (cost reduction, productivity, profit, etc..),
the issue is much more difficult in regard to public policy. Because the goals of public
policy are manifold and often contradictory, the same problem can be solved in different
ways, the consensus can be extremely laborious.
Moreover, if in the spirit of the Paris Declaration indicators should be chosen in the
matrix of PRSP or logical frameworks of sectoral programs, in fact, we must recognize
the weight of the funding agencies in the process of choosing some indicators and
setting targets. These words of a representative donor illustrate the situation: "The
Government of Mali led the aircraft, we ensure that there is a good flight plan, the
altitude is well controlled, if the needles (of the gauge) are not good, we are providing
20 Ministry of Economy and Finance, op. cit. p. 41
21This statistic is derived from the Permanent Survey of Households (EPAM) conducted in 2007.
Traduction libre Constance Cournède IEA 8
better ones"22. This might surprise observers, but we know that taken by the throat
states are sometimes forced to accept harsh conditions, hoping to gain later margins of
maneuver. In fact, in order to regain margins of maneuver, the real challenge now for
developing countries is to rebuild their capabilities undermined by decades of structural
adjustment, isn’t it?
III. Strenghtening the autonomy of developing countries in the conduct of public
policy
While managing public resources wisely and being accountable to citizens is a matter of
common sense, which justifies in principle the results based management, we must also
agree on the meaning and content to be given to this concept, since everything cannot
be quite quantified. The culture of results is also not the same everywhere. Each country
must develop its public policy from its own realities, without being imposed an external
model. A country can draw on successful experiences elsewhere, it is a matter of
common sense in an open world, but it must be intentional and thus with the necessary
adaptations. The issue is about respecting the dignity of states, including the most
vulnerable among them, allowing them to afford a real autonomy in the conduct of their
policies.
This movement must be carried by democratic government enjoying legitimacy and
popular support for real, with the full involvement of civil society and parliament. For, it
must not be mistaken, the choice of indicators is not politically neutral. In a country like
Mali, heir to an ancient culture, with buildings classified as World Heritage of Humanity
in the old city of Djenne, Timbuktu and Gao, and thousands of ancient manuscripts, how
can we understand that in the CSCRP there is no indicator to safeguard these
treasures?
RBM should not be seen as a new ideology or a tool in the hands of a few technocrats,
who would decide what is good for everybody, and thereby impose changes on society.
It can be legitimate only as an inclusive approach to support the nation in a collaborative
research to make an optimum use (intelligent, balanced and fair) of public resources. In
a system of good public policy, indicators should be designed primarily to the
implementation of national policies, and then has to reward the achievements through
concerted management of aid
In this approach, politicians, parliaments and citizens organized into associations must
take their place. In order to help people to appropriate this debate, we must give correct
information on these issues, spaces for free expression, and reasonable consideration
of different opinions, especially those of weaker groups.
22 As reported by Isaline Bergamashi in: Strategic Framework to fight against poverty, budget support and policy
dialogue in Mali. Budget support in question: How effective, for whom? GRET Collection debates and controversies,
n2, pp 27 and following.