Terms of Reference for External Consultant/Firm
Consultant for Project Final Evaluation
Solar Photovoltaic Electricity for Karampuang Island
|Type of evaluation||Endline Survey and Summative/Final Evaluation|
|Expected evaluation methodologies||Mix method, endline survey (quantitative) with key informant interviews and focus group discussions (qualitative)|
|Consultant involvement||· Submit Project Final Evaluation Proposal|
· Conduct document review
· Develop data collection tools
· Submit Inception Report
· Train enumerators
· Perform data collection, management, quality assurance, and analysis
· Present initial findings
· Produce and present Project Final Evaluation Report
· Produce 3-5 Stories of change based on findings from interviews and/or discussions
|Location of work||Jakarta with travel to project site (Karampuang Island, Mamuju, West Sulawesi)|
|Expected start/ end dates, number of work days||8 January 2017 – 21 February 2017 (maximum 45 days)|
- Project Description
- Project background and objectives
The majority of rural communities in Indonesia live in poverty with low electrification ratios. Traditionally, costly and polluting diesel-fired power generation has been the main source of power for rural areas. In addition, many remote communities still lack access to any power at all with little expectation of being supplied on-grid power by the state-owned electricity company (PLN) in the foreseeable future. At the isolated community level, a properly developed, constructed and sustainable Community-based Renewable Energy (RE) power is one of the solutions to raise the quality of life in these communities. These plants should be owned, managed and maintained by the rural communities, but they lack sufficient capital to build them and thus need funding and technical support from third parties to achieve realization.
- SKY Energy (SKY) is a grantee of Millennium Challenge Account Indonesia (MCA-I), on a renewable energy and community development project titled ‘Solar Photovoltaic Electricity for Karampuang Island’ (SPV project). The project is aimed at establishing a 598.64 kWp off-grid electricity service with solar photovoltaic (PV) generation and distribution at 4-areas in Karampuang Island, West Sulawesi Province, Indonesia. The project has established Special Purpose Vehicle (SPV), i.e. PT. Karampuang Multi Daya (KMD), which will act as utility company owned by the community at the project site.
The overall objective of the project is the “Increase of household income and reduction of GHG emissions due to improved access to and utilization of electricity generated from renewable sources”. To achieve this objective, the project plans to deliver the following outcomes:
- Improved awareness among communities and stakeholders on efficient use of electricity, safety of Solar PV facilities, and Natural Resources Management.
- Households (HHs) including women-headed HHs and vulnerable groups have reliable and sustainable access to electricity from renewable sources.
- SPV has capacity to operate and manage the solar PV system.
- Increased local economic opportunities from the utilization of electricity generated from renewable resources.
The SPV project is run by a consortium consisted of SKY and PT. Inti Karya Persada Tehnik (IKPT) company. Responsible for the construction component, IKPT works together with two (2) contractors, i.e. PT. Kontrol Power Utama (KPU) and CV Alif Karampuang (AK). While for the crosscutting (community development) component, SKY employs a group of specialist (Crosscutting Team) and sub-contracts Yayasan Karampuang (YK) as a local partner to implement one of the crosscutting components. The project officially started on 10 February 2017 and will end on 31 March 2018. To date, the project has implemented awareness raising activities aimed to reach the total village population of 3,317 people; constructed four (4) solar power plants to reach the overall 784 households and ensured equal access to the marginalized groups; prepared PT. KMD to fully run the solar PV services after the project ends; and improved economic opportunities among villagers from the new access to electricity.
To measure the successes achieved and answer the predetermined success indicators in the project plans and collect materials to contribute to MCA-I’s impact assessment, SKY plans to conduct a Project Final Evaluation.
- Project areas
The SPV project takes place in Karampuang Village located in Karampuang Island, Mamuju, West Sulawesi Province, with total population of 3,317 people as the main beneficiaries.
- Project stakeholders
- MCA-I: Donor Funding (Including TOPE and PMC)
- SKY Energy: Grantee/Main Implementer (in Mamuju and Jakarta offices)
- IKPT: Consortium Partner/Engineering, Procurement and Construction
- KPU: Sub-contractor for Electrical Work
- CV Alif Karampuang: Sub-contractor for Civil Work
- Crosscutting Team: Consultant team for Crosscutting Component
- Yayasan Karampuang: Sub-contractor for Crosscutting Component
- Relevant government officials at different levels (village, district and province)
- Previous evaluation activities
Prior the implementation of the project, under direction from MCA-I, SKY has conducted several assessments, i.e. Detailed Feasibility Study (DFS) in 2015 and Detailed Engineering Design (DED) in 2016, and Landscape and Life Analysis (LLA) in 2016. Throughout the implementation, SKY has also conducted consistent monitoring & evaluation practices, with the following reports as the result:
- Baseline Report
- Monthly and Quarterly Project Reports since March 2017
- Project Indicator Tracking Table
- Quarterly Stories of Change
- Evaluation Overview (Scope and Coverage)
2.1 Objectives of final evaluation
The objectives of the evaluation are to:
- Measure the achievement of the project objectives and outcomesto examine the changes (positive or negative, intended or unintended) as results and or contributed by the project.
- Determine the relevance, effectiveness, level of coordination in project management and implementation among related stakeholders, and examine the project sustainabilityin the future.
- Document lessons learned, best practices, and stories of changeto inform future project/program development.
- Provide recommendationsfor further innovation or potential scale up project in the future
2.2 Evaluation criteria and questions
The main evaluation question is: To what extent and in what ways the Community based Solar PV project contributed to increased household income and reduced GHG emissions due to improved access to and utilization of electricity generated from renewable sources?
In order to answer the main evaluation question and meet the objectives of evaluation, evaluation criteria with main evaluation questions and technical questions are summarized as followed:
|Criteria||Main evaluation questions||Sub-questions|
|1. Project Design and relevance||To what extent the project design has corresponded with the needs and priorities of local communities and stakeholders?||1.1. Did the project conduct need assessment? Were the assessment results utilized to inform the project design?|
1.2. Was there a clear theory of change/logical framework shaping the project? If yes, were project plan and activities aligned to the logical framework?
|2. Effectiveness||To what extent the project has met its stated objectives?|
How well were project activities planned and implemented?
|2.1. Did the project achieve the intended outcomes and outputs as determined in the logical framework and M&E Plan?|
2.2. What factors contributed to or prevented the achievement of intended outcomes and outputs?
2.3. To what degree the project has complied with the specified standards and requirements (e.g. IFC performance standards)?
2.4. Did the project reach the people it intended to? If not, why?
2.5. How did the project ensure that women and vulnerable groups benefit from the project?
What were main observable changes reported and or observed in target communities (at individual, household, institution, or community levels) as results of the project intervention (Intended, unintended, both positive and negative)?
2.6. In what ways the project has contributed to broader community development (economic and social) in target areas?
2.7. Were the project monitoring systems effective in providing information to program staff to ensure program decisions and course correction (if needed) were made?
|3. Sustainability||To what extent, the project has demonstrated the ability to sustain positive results beyond Compact period?||3.1. Were there any project activities or component that will most likely continue beyond the project end date? If so, what are they and why?|
3.2. To what extent the electricity provision and operations of the SPV can be sustained?
3.3. Why are some components of the project more sustainable than others?
|To what extent the coordination between the project implementer and partners contribute to the project’s achievement?|
|4.1. Was there any clear and appropriate division of tasks, roles, and responsibilities in existing working relationships (SKY, IKPT, KPU, AK, Crosscutting Team, YK, local government, and community)?|
4.2. Was there adequate support received from different units in MCA-Indonesia to facilitate the project implementation? Which type of supports (e.g. Document review, coaching, workshop/training, guideline revision, etc.) were found most useful and why?
4.3. What worked well in the relationship with MCA-I and what could be improved?
|5. Lessons learnt||Are there any lessons on how the project was designed and implemented that would be useful for similar projects in the future?|
|5.1. What worked well and what need to be improved in the project design, implementation and exit/ sustainability strategy, in the following areas?|
· Crosscutting (environmental safeguard, gender and social inclusion, and community development)?
· Awareness raising and community outreach
· SPV (Village Power Utility) set up and sustainability
· Economic opportunity
· Linkage and synergy between project activity with village development plan
5.2. What are key challenges encountered by Sky Energy and Partners in implementing community based RE project?
|6. Replicability||To what extent the project design is replicable in other contexts in Indonesia?||6.1. Is the project design (approaches and strategies including partnership model between private sector, government & community) replicable in other communities in Indonesia? What make it worked and what were the challenges encountered?|
6.2. To what extent and in what ways the partnership with private sector (i.e. Sky Energy) has contributed to the project effectiveness?
6.3. What are pre-conditions needed to ensure successful community-based renewable energy project?
2.3 Evaluation Methodology
The evaluation will apply mix methodologies (quantitative and qualitative). For the quantitative component, a baseline survey has been conducted prior to the implementation of the project. Endline survey would be carried using the questionnaires developed from the previous baseline questionnaire with further development under consultation with SKY and Crosscutting Team. The consultant will conduct an endline survey to seek the changes brought by the project. For the qualitative component, interviews and FGDs would be performed with the data collection protocols developed by the consultant under consultation with SKY and Crosscutting Team. Further explanations toward the use of the mix methodologies are described as follow.
2.3.1 Document Review
Document review would cover reviewing all essential project documents to answer the project indicators and the required evaluation questions. There are 27 indicators from the Monitoring & Evaluation (M&E) Plan, 16 indicators from the Environmental and Social Management System and Plan (ESMS/ESMP), and 21 indicators from the Project Social and Gender Integration Plan (P-SGIP). The project indicators would be shared during initial briefing and consultant should provide the indicators’ final scores based on the survey and document review results. Apart from the project documents, it is highly desired to also refer to previous related research findings that would be helpful for recommendation.
2.3.2 Quantitative Component
Objective of Endline Survey
The endline survey is intended to look at the emerging changes in community members’ awareness, knowledge and behavior on efficient use of electricity, safety measures of solar PV, and natural resources management (NRM) contributed by the project based on the project’s indicators and possible other outcomes that could be addressed by this project as follow:
Based on Project’s indicators
- Number and percentage of sampled people interviewed who are able to name at least 3 key measures of efficient use of electricity
- Number and percentage of sampled of people who are able to name at least 3 key safety measures of SPV facilities
- Number and percentage of sampled of people who are able to name at least 3 key measures for sustainable NRM
- Number and percentage of HHs who are satisfied with the reliability of electricity service
- Number and % of community members trained in economic opportunity skills who have started productive activities by applying knowledge and skills gained from the trainings supported by project
Geographic scope & target population
The survey will be carried out in the targeted village (Karampuang Village), with the following information of population:
|8||Ujung Bulo Wisata||76|
Data source: DFS Study (2015)
Planned analysis of survey data
- Descriptive (means, confidence intervals, etc.)
- Analytical (hypothesis testing, regression analysis, etc.)
The unit of analysis would be based on Households, and a standard level of required precision would be for a value of the relative standard error (or coefficient of variation) of ≤ 0.05 key variables.
2.3.3 Qualitative Component
The qualitative methodology will be used to enrich findings from the endline survey and to give thorough explanations for the evaluation questions, such as household electricity profile (e.g., the utilization of solar PV generated electricity, electricity expenditure, electricity appliance ownership, access for poor households), mechanism for electricity subsidy for poor households, community participation in the solar PV facility development, local community capacity to manage, operate and maintain SPV, and profile of business enterprises/entrepreneurs supported by the project (e.g., its business, product, income, challenges, potentials). Key Informant/In-Depth Interview and Focus Group Discussion will be performed with key informants and discussion participants selected based on determined criteria. Our target of participants comprises of the target community members, local government (village, district, province), consortium partners and sub-contractors, and MCA-I at National and Sub-National levels.
2.4 Main audience of evaluation
The main audience for this evaluation is the grantee’s consortium partners and MCA-I at national and sub-national level, and government.
III. Working Arrangement
3.1 Scope of work
The consultant will be responsible to ensure that the Project Final Evaluation uses scientific research methodologies that meet standard research and evaluation. The scope of work must include:
- Providing a complete Inception Report (final proposal and tools). The inception report will contain the evaluation background, purpose, methodologies, procedure for data collection, management, and analysis, reporting, timeframe, consultant team, and budget. For the survey, consultant should provide a sample design (calculation, allocation, and selection), justification of sample weight (if needed), and response rate. The Inception Report will be made by the selected consultant(s) based on the results of initial briefing with SKY’s Crosscutting team.
- Developing questionnaires for household survey based on the previous baseline questionnaires. This process will be done in close consultation with the Crosscutting team in development, with questionnaire pre-testing and final approval from the Crosscutting team.
- Developing data quality control measures for the data entry. The consultant should propose quality control methods to ensure high quality data collection and data inputs (i.e. field check, double checking for data entry, procedure for data cleaning).
- Training data collection team (enumerators, field supervisor). The consultant should prepare training materials, such as survey background, methodology and sampling design, questionnaires, interview technique, introduction of ethical guidelines, and field practices.
- Leading and coordinating data collection. The consultant will coordinate closely with the Crosscutting team, particularly in selecting the enumerators and field supervisors as well as in designing and overseeing operations of the survey. The enumerator criteria will be discussed during the initial briefing. The consultant should conduct and oversee data collection in the field, evaluate data collection in daily basis and provide daily progress report, including emerging challenges issues in the field and the actions taken to address the issues.
- Conducting data management and analysis. The consultant should report progress of data collection in daily basis. Upon the completion of data collection at field, the consultant has to conduct data cleaning using standard statistical package software, preferably SPSS, merged with the baseline data for analysis. The consultant should ensure that the analysis would take into account the use of appropriate variance estimation technique given the sample design, and methods to reduce potential bias.
- Writing and presenting Project Final Evaluation Report, including 3-5 stories of change. The consultant will develop a Project Final Evaluation Report and organize a dissemination workshop at the local level to present the key finding results to and gather feedback from relevant local stakeholders, consortium partners, and MCA-I.
3.2 Logistic and Administrative support
The consultant is expected to use her/his own computer. Approved administrative and logistic costs will be reimbursed by the Crosscutting Team. The consultant will be able to work remotely, after approval of the work plan.
3.3 Reporting relationship
The consultancy service agreement would be facilitated by SKY in consultation with the Crosscutting Team. The consultant will report to Yoyo Sudaryo, Project Manager from SKY and Adisti Ikayanti, ME Specialist from the Crosscutting Team. Final report acceptance will be subject to approval from MCA-Indonesia.
3.4 Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American Evaluation Association’s Guiding Principles for Evaluators. Below is the summary of the ethical guideline (taken from www.eval.org/Publications/GuidingPrinciplesPrintable.asp)
- Informed Consent: All participants are expected to provide informed consent following standard and pre-agreed upon consent protocols.
- Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
- Competence: Evaluators provide competent performance to stakeholders.
- Integrity/Honesty: Evaluators display honest and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.
- Respect for People: Evaluators respect the security, dignity, and self-worth of respondents, program participants, clients, and other evaluation stakeholders. It is expected that the evaluators will obtain the informed consent of participants to ensure that they can decide in a conscious, deliberate way they want to participate.
- Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation
3.5 Future use of data
All collected data will be the sole property of SKY Energy and MCA-I. The consultant shall submit all raw data and interview records/summary to the SKY Energy. The consultant may not use the data for their own research purposes, nor license the data to be used by others, without the written consent of the SKY Energy and/or MCA-I.
3.6 Expected activities and deliverables
- Initial briefing and development of Inception Report (final proposal and tools) prior to data collection
- Document review
- Development and finalization of survey questionnaire and structured questions for Key Informant/In-depth Interviews and Focus Group Discussions (in English and Indonesian)
- Development of Training Manual for field personnel (in Indonesian)
- Training of enumerators] Data quality control (field check, double checking of data entry)
- Data entry, cleaning and analysis in SPSS and Ms. Excel (raw data sets, merged & modified data set, log files, code book)
- Progress report of quantitative data collection at daily basis (target vs achievement, response rate, main challenges faced and actions taken)
- Development of brief report of preliminary results (Initial Report)
- Draft Baseline Survey and Evaluation Report
- Presentation in a dissemination workshop with local stakeholders with the necessary presentation materials
- Final Baseline Survey and Evaluation Report (in English and Indonesian). The Final Report should include contents as follow:
- Glossary and Acronyms
- Table of contents
- Executive summary
- Introduction, covering purpose of report, purpose of evaluation, detailed methodology, actions to reduce potential bias
- Study limitation
- Key findings against evaluation questions and comparison of endline vs baseline scores from the Project’s Logical Framework (along with narrative description)
- Lessons learned
- Annex including photos taken during data collection
- Stories of change based on evaluation findings
- Summary of evaluation findings for dissemination to stakeholders (2-4 pages)
- Handover of all data collected and used for analysis and report writing (raw data, clean data set, records, analysis working paper/matrix, sealed verbatim transcript, etc.)
3.7 Duration of consultancy
The consultancy will be provided within 45 days at maximum, starting from 8 January 2018 to 21 February 2018.
- Required qualifications
Required qualifications for a single consultant or consulting team are:
- Seven (7) years of experiences conducting and leading both quantitative and qualitative researches, three (3) of which should be with international organizations
- Excellent knowledge and skills in mix method researches (quantitative & qualitative). Able to show experiences in designing, managing, and analyzing data, particularly for regression analysis (pooled cross-section regression). Able to show experiences in designing and analyzing qualitative data.
- Experiences in leading evaluation, preferably for evaluating projects related to renewable energy, natural resources management, community and local economic development.
- Demonstrated experiences in questionnaire development.
- Demonstrated experiences in training enumerators and data collection.
- Application materials and procedures
5.1 Application materials
The proposal should not exceed 5 (five) pages and include the following:
- Consultant’s experience/detailed CVs of all professionals who will work on the Project Final Evaluation. This section should highlight past experience of the consultant in conducting survey, key informant/in depth interview, and focus group discussions in Indonesia. This section should mention names, qualifications and experiences of all persons who would be involved in various aspects of conducting the quantitative and qualitative data collection, along with the level of efforts.
- Methodology. Provide the Final Evaluation methodologies, analysis plan, and sample size including the formula used for the sample calculation, sample allocation, and strategy to minimize bias.
- Work plan. The proposal should clearly mention details of each and every activity including kind of preparatory work, training, house listing, sampling and data collection work, interviews, discussions, data entry, data processing and analysis; results and report writing, and presentation of final report. The time line and person(s) responsible for each activity need to be clearly mentioned.
- Consultant team. If there is more than one consultant on the proposed team, please provide the number of persons needed for both for quantitative and qualitative methodologies. Attach a table describing the level of effort (in number of days) of each team member in each of the evaluation activities.
|Name of person||Level of effort (number of days)|
|Tools (questionnaire, KII & FGD protocols) development||Enumerator Training||Data collection||Data management & quality assurance||Data analysis||Report writing||Presentation of findings|
- Training. State who will be responsible for training of data collectors and include information on prior experience. Describe how training will be done, the topics covered, expected duration and logistic and administrative support needed from SKY Energy and the Crosscutting team.
- Quality control mechanisms. Provide a section detailing the mechanisms to ensure data quality by clearly specifying steps for data validation. This section may also include supervisory mechanism for data quality and the role of field supervisors.
- Data entry and processing plan. This section should clearly state details preparation and mechanism for data entry, validation checks, and data processing activities.
- Data analysis. Provide details on the analyses that will be carried out, and on the person/people responsible for data analysis (including prior experiences).
- Progress updates: This section should clearly indicate the mechanism that would be used to communicate with SKY Energy and the Crosscutting team within the reporting relationships on regular update about the field activities, coverage rate, data entry status etc.
- Proposed Budget with clear description and total amount in IDR and USD.
- Professional References. Provide three references from your previous clients.
- Writing Sample. Provide one sample of a recent Project Final Evaluation Report completed for another client where you are the lead evaluator as well as author and principal researcher.
5.2 Application procedures