APPENDIX I

QUALITY ASSURANCE SURVEILLANCE PLAN

Introduction

This Performance-Based Quality Assurance Surveillance Plan (QASP) sets forth procedures and guidelines that the Department of Education will use in evaluating the technical performance of the Contractor. A copy of this plan will be furnished to the contractor so that the contractor will be aware of the methods that the Government will employ in evaluating performance on this contract and address any concerns that the contractor may have prior to initiating work.

Purpose of the QASP

The QASP is intended to accomplish the following:

Define the roles and responsibilities of participating Government officials and outside experts;

Define the key deliverables which will be assessed;

Describe the rating elements and standards of performance against which the contractor's performance will be assessed for each key deliverable;

Describe the process of quality assurance assessment; and

Provide copies of the quality assurance monitoring forms that will be used by the Government in documenting and evaluating the contractor's performance.

Each of these purposes is discussed in detail below.

Roles and Responsibilities of Participating Government Officials

The following Government Officials will participate in assessing the quality of the contractor's performance. Their roles and responsibilities are described as follows:

Contracting Officer's Technical Representative (COTR). COTR will be responsible for monitoring, assessing, recording, and reporting on the technical performance of the contractor on a day-to-day basis. The COTR will also be responsible completing the Quality Assurance Monitoring Forms (described in greater detail below and provided in Exhibits A, B, and C which will be used to document the inspection and evaluation of the contractor's work performance on three key deliverables in Phase I, and three key deliverables in Phase II.

It is extremely important for the COTR to establish and maintain a team-oriented line of communication with the Contractor's Project Manager (PM) and the PM's office staff in order to perform monitoring functions. The COTR, Contracting Officer (CO), and PM must work together as a team to ensure that required work is accomplished in an efficient and proper manner. Meetings should be held on a regular basis in order to resolve serious problems. Less serious problems should be discussed and resolved on an impromptu basis.

The Contract Officer (CO) will have overall responsibility for overseeing the contractor's performance. The Contract Specialist (CS) will be responsible for the day-to-day monitoring of the contractor's performance in the areas of contract compliance, contract administration, cost control and property control; reviewing the COTR's assessment of the contractor's performance; and resolving all differences between the COTR's version and the contractor's version. The CO may call upon the expertise of other Government individuals as required.

The Contracting Officer's (CO) procurement authorities include the following:

SOLE authority for any decisions which produce an increase or decrease in the scope of the contract;

SOLE authority for any actions subject to the "Changes" clause;

SOLE authority for any decision to be rendered under the "Disputes" clause;

SOLE authority for negotiation and determination of indirect rates to be applied to the contract;

SOLE authority to approve the substitution or replacement of the Project Manager and other key personnel;

SOLE authority to approve the contractor's invoices for payment, subject to the Limitation of Costs clause and the Limitation of Funds clause;

SOLE authority to monitor and enforce Department of Labor promulgated labor requirements;

Authority to arrange for and supervise Quality Assurance activities under this contract;

SOLE authority to approve the contractor's Quality Control Program;

To approve all contractor purchases of equipment, supplies, and materials exceeding $2,500 are encouraged even though not required by FAR 13.106; and

Signatory authority for the issuance of all modifications to the contract.

Key Deliverables to be Assessed

Even though the Government through its COTR will be monitoring the contractor's performance on a continuing basis, the volume of tasks performed by the contractor makes technical inspections of every task and step impractical. Accordingly, the Department of Education will use a quality- assurance review process to monitor the contractor's performance under this contract. Specifically, the COTR will assess the contractor's performance across a set of tailored rating elements for each of six key deliverables:

Refined study design for Phases I and II;

OMB clearance process for Phases I and II;

Draft final reports for Phases I and II.

Rating Elements and Standards of Performance for Key Deliverables

The contractor's performance shall be evaluated in Phase I and Phase II by assessing the three key deliverables described above. Tailored rating elements for each key deliverable have been developed and incorporated into the Quality Assurance Rating Forms (see Exhibits A, B, and C). The rating elements and acceptable standards of performance for each key deliverable are described below:

Refine Study Design:

(1) Quality of data collection plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to collecting data and incorporating secondary data sources (when applicable);

(2)	Quality of data analysis plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to analyzing quantitative and qualitative data, adequately addressing key research questions, providing a clear conceptual model for analysis;

(3)	Quality of dissemination plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders; and

(4)	Comprehensiveness, clarity, and organization of design.

...where acceptable performance would include complete, clear, efficient approaches to addressing the research questions, clear writing style, proper grammar/spelling, clearly organized document format.

OMB Clearance Process:

(1)	Comprehensiveness, clarity, and organization of draft package.

...where acceptable performance would include a complete OMB clearance package, clear writing style, proper grammar/spelling, well-organized document format, accurate and complete descriptions of data collection and data analysis plans;

(2)	Quality of draft data collection instruments.

...where acceptable performance would include complete, clear, straightforward data collection instruments, sufficient instrumentation to support data collection analysis plans without being unnecessarily burdensome to respondents; and

(3)	Quality of support during clearance process.

...where acceptable performance would include timely, relevant, complete and continuous support during the clearance process, including responding to OMB and/or public questions within three days of each request.

Draft Reports:

(1)	Accuracy and relevance of information provided.

...where acceptable performance would include complete, clear, logical, appropriate, accurate reporting on data analysis results for key research questions, appropriate context for interpreting results;

(2)	Usefulness for target audiences.

...where acceptable performance would include clear, tailored language and results for targeted audiences; and

(3)	Comprehensiveness, clarity, and organization of report.

...where acceptable performance would include comprehensive description of key results, clear writing style, proper grammar/spelling, well-organized document format.

Process of Quality Assurance Assessment

While quality assurance is closely tied to these performance standards for deliverable content, timeliness and cost are also important considerations in the assessment of contractor performance. Using monthly progress report information submitted by the contractor (see Section III, B for details) on costs, the contract specialist will verify that the deliverable is at or under cost and on time. The government will evaluate "Target Cost" and "Target Fee" according to contract clause B.2 FAR 52.216-10, INCENTIVE FEE (Mar 1997).

In the event of an excusable delay (as defined in FAR 52.249-14, Excusable Delays and EDAR 3452.242-71, Notice to the Government of Delays), the Department and the contractor shall work together to modify the contract with regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance, shall be measured by the date agreed upon in the modification.

The contractor's performance will be evaluated by the COTR using the appropriate key deliverable evaluation forms (Exhibits A: Draft Evaluation Design, B: OMB Clearance Process, and C: Draft Report). These forms will serve to document and evaluate the contractor's performance for each of the key deliverables under this contract. Each form will be completed by the COTR and the rating element scores will be averaged to generate the final evaluation score for that key deliverable. This final evaluation score will document the COTR's understanding of contractor performance for that key deliverable.

The COTR will evaluate each rating element for each key deliverable in accordance with the following definitions of contractor performance:

•	Unacceptable. Level of performance which is not acceptable and which fails to meet the minimum standards of performance.

•	Acceptable. Level of performance which meets the minimum standards of performance.

•	Superior. Level of performance which exceeds the minimum standards of performance.

The COTR will substantiate, in narrative form, all individual scores judged to be indicative of "superior" or "unacceptable" performance. At a minimum, performance at the "acceptable" level is expected from the contractor.

The COTR will forward copies of all completed AQ monitoring forms and a report of average scores to the CO and contractor according to the following schedule:

Refined study design assessment: submitted by the close of business 10 work days from the date it was received by the COTR.

OMB clearance process assessment: submitted by the close of business 15 work days from the date it was received by the COTR.

.

Final report assessment: submitted by the close of business 15 work days from the date it was received by the COTR.

The contractor shall respond in writing to any "unacceptable" final average evaluation scores within 5 working days after receipt of the form(s).

The CO will review each key deliverable evaluation form prepared by the COTR. When appropriate, the CO may investigate the event further to determine if all the facts and circumstances surrounding the event were considered in the COTR opinions outlined on the forms. The CO will immediately discuss every event receiving an "unacceptable" rating with the contractor to assure that corrective action is promptly initiated.

Incentive Fee Plan

Performance incentive fees will be awarded for those key deliverables that are judged by the COTR to be superior. If the contractor's performance is unacceptable, the government will deduct a fee from the target fee, and if the contractor's performance is acceptable, there will be no incentive fee awarded for the deliverable.

The tasks that will be used to evaluate contractor performance and the bonuses tied to each are listed below. In order to be considered for an incentive fee, the product must be delivered no later than the date specified in the contractor's technical proposal or in the schedule of deliverables in the statement of work. (See FAR 52.249-14, Excusable Delays and EDAR 3452.242-71, Notice to the Government of Delays.)

Task/Deliverable	Unacceptable	Superior

Performance	Performance

--deduction--	--bonus--

Refine Study Design

Task 4	$	600.00	$	1,500.00

Task 11	600.00	1,500.00

OMB Clearance Package

Task 6	300.00	900.00

Task 13	300.00	900.00

Final Reports

Task 10	1,200.00	2,500.00

Task 15	1,200.00	2,500.00

Total Possible Deduction/Bonus	$4,200.00	$9,800.00

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT A: DRAFT EVALUATION DESIGN EVALUATION FORM

Rating Element 1: Quality of Data Collection Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to collecting data lacking a strategy for the incorporation of secondary data sources (when applicable), unnecessarily burdening respondents.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to collecting data and incorporating secondary data sources (when applicable).

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches and/or methods to collecting data, incorporating secondary data sources (when applicable), and reducing respondent burden.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Quality of Data Analysis Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to analyzing quantitative and qualitative data, lack a theory or conceptual model for analysis.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to analyzing quantitative and qualitative data, adequately addressing key research questions, providing a clear conceptual model for analysis.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches and/or methods to analyzing quantitative and qualitative data.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 3: Quality of Dissemination Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful, multi-media approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 4: Comprehensiveness, Clarity and Organization of Design

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, unclear, inefficient approaches to addressing the research questions, unclear writing style, poor grammar/spelling, disorganized document format.

Acceptable performance (5-7) would include complete, clear, efficient approaches to addressing the research questions, clear writing style, proper grammar/spelling, clearly organized document format.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches to addressing the research questions.

Supporting comments (required for unacceptable or superior performance ratings):

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT B: OMB CLEARANCE PROCESS EVALUATION FORM

Rating Element 1: Comprehensiveness, Clarity and Organization of Draft Package

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing OMB clearance package requirements, unclear writing style, poor grammar/spelling, disorganized document format, inaccurate or incomplete descriptions of data collection and data analysis plans, and require major corrections/edits for submission to OMB.

Acceptable performance (5-7) would include a complete OMB clearance package, clear writing style, proper grammar/spelling, well-organized document format, accurate and complete descriptions of data collection and data analysis plans and could be submitted for clearance with only minor corrections/edits.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include submission of the final package to OMB at least one week ahead of schedule for the Year 1 clearance process and at least two weeks ahead of schedule for the Years 2 and 3 clearance process.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Quality of Draft Data Collection Instruments

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, unclear, confusing data collection instruments, insufficient instrumentation to support data collection and analysis plans.

Acceptable performance (5-7) would include complete, clear, straightforward data collection instruments, sufficient instrumentation to support data collection and analysis plans without being unnecessarily burdensome to respondents.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include submission of package to OMB at least one week ahead of schedule for the Year 1 clearance process and at least two weeks ahead of schedule for the Years 2 and 3 clearance process.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 3: Quality of Support during Clearance Process

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include irrelevant, incomplete, and sporadic or nonexistent support during the clearance process, not responding to public and/or OMB questions or responding more than 3 days after each request.

Acceptable performance (5-7) would include relevant, complete, and continuous support during the clearance process including responding to questions from OMB and/or the public within 3 days of each request.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND provides innovative or exceptionally skillful solutions to problems raised during the clearance process, turns around each request from OMB and/or the public within 2 days of each request.

Supporting comments (required for unacceptable or superior performance ratings):

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT C: DRAFT REPORT EVALUATION FORM

Rating Element 1: Accuracy and Relevance of Information Provided

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include incomplete, illogical, unclear, inappropriate, inaccurate reporting on data analysis results for key research questions, lack appropriate context for interpreting results.

Acceptable performance (5-7) would include complete, logical, clear, appropriate, accurate reporting on data analysis results for key research questions, appropriate context for interpreting results.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful reporting on data analysis results for research questions.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Usefulness for Target Audiences

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include unclear, inappropriate language and results for targeted audiences.

Acceptable performance (5-7) would include clear, tailored language and results for targeted audiences.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches and/or methods to providing tailored information to individual audiences.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 3: Comprehensiveness, Clarity and Organization of Report

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing elements, unclear writing style, poor grammar/spelling, disorganized document format.

Acceptable performance (5-7) would include comprehensive description of key results, clear writing style, proper grammar/spelling, well-organized document format.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include relevant analyses of and reports on extant data sources.

Supporting comments (required for unacceptable or superior performance ratings):