QUALITY ASSURANCE SURVEILLANCE PLAN

Introduction. This Performance-Based Quality Assurance Surveillance Plan (QASP) has been developed pursuant to the requirements of the Performance-Based Statement of Work in Contract No. HS-98-0100. This plan sets forth procedures and guidelines that the U.S. Department of Education will use in evaluating the technical performance of the Contractor. A copy of this plan will be furnished to the Contractor so that the Contractor will be aware of the methods that the Government will employ in evaluating performance on this contract and address any concerns that the Contractor may have prior to initiating work.

Purpose of the QASP. The QASP is intended to accomplish the following:

Define the roles and responsibilities of participating Government officials and outside experts;

Define the essential deliverables that will be assessed;

Describe the rating elements and standards of performance against which the Contractor's performance will be assessed for each key deliverable;

Describe the process of quality assurance assessment; and

Provide copies of the quality assurance monitoring forms that will be used by the Government in documenting and evaluating the Contractor's performance.

Each of these purposes is discussed in detail below.

Roles and Responsibilities of Participating Government Officials and Experts.

The following Government Officials and/or experts will participate in assessing the quality of the Contractor's performance. Their roles and responsibilities are described as follows:

Contracting Officer Technical Representative (COTR). The COTR will be

responsible for monitoring, assessing, recording, and reporting on the technical performance of the Contractor on a day-to-day basis. They will also be responsible for assembling a three member Quality Assurance Review Panel (QARP) to complete the Quality Assurance Monitoring Forms (described in greater detail below and provided in Exhibits A, B, C, and D ) which will be used to document the inspection and evaluation of the Contractor's work performance on four key deliverables in Phases 1 and 2, and two key deliverables in Phase 3.

Two additional ED staff and/or outside experts with knowledge and experience in the areas of special education finance, evaluation, data analysis, and/or contract management will serve as QARP members. For each key deliverable assessment, one of these individuals will be selected, based on time availability and appropriateness to the task, to serve with the COTR (resulting in a two to three-person team for each deliverable assessment) in assessing the quality of a deliverable.

It is extremely important for the COTR to establish and maintain a team-oriented line of communication with the Contractor's Project Manager (PM) and the PM's office staff in order to perform her monitoring functions. The COTR, Contracting Officer (CO), and PM must work together as a team to ensure that required work is accomplished in an efficient and proper manner. Meetings should be held on a regular basis in order to resolve serious problems. Less serious problems should be discussed and resolved on an impromptu basis.

The Contracting Specialist (CS) will have overall responsibility for overseeing the Contractor's performance. The CO will also be responsible for the day-to-day monitoring of the Contractor's performance in the areas of contract compliance, contract administration, cost control and property control; reviewing the COTR's assessment of the Contractor's performance; and resolving all differences between the COTR's version and the Contractor's version. The CO may call upon the expertise of other Government individuals as required.

The Contracting Officer's (CO) procurement authorities include the following:

SOLE authority for any decisions which produce an increase or decrease in the scope of the contract;

SOLE authority for any actions subject to the "Changes" clause;

SOLE authority for any decision to be rendered under the "Disputes" clause;

SOLE authority for negotiation and determination of indirect rates to be applied to the contract;

SOLE authority to approve the substitution or replacement of the Project Manager and other key personnel;

SOLE authority to approve the Contractor's invoices for payment, subject to the Limitation of Costs clause and the Limitation of Funds clause;

SOLE authority to monitor and enforce Department of Labor promulgated labor requirements;

Authority to arrange for and supervise Quality Assurance activities under this contract;

SOLE authority to approve the Contractor's Quality Control Program;

To approve all Contractor purchases of equipment, supplies, and materials exceeding $2,500; and

Signatory authority for the issuance of all modifications to the contract.

Key Deliverables to be assessed: Although the Government through its COTR will be monitoring the Contractor's performance on a continuing basis, the volume of tasks performed by the Contractor makes technical inspections of every task and step impractical. Accordingly, the U.S. Department of Education the U.S. Department of Education will use a quality-assurance review process to monitor the Contractor's performance under this contract. Specifically, the QARP will assess the Contractor's performance across a set of tailored rating elements for each of four key deliverables:

Draft evaluation design;

OMB clearance process;

Draft yearly reports; and

Database.

Rating Elements and Standards of Performance for Key Deliverables. The Contractor's performance shall be evaluated in Phases 1 and 2 by assessing the four key deliverables described above, and in Phase 3 by assessing the yearly report and database only. Tailored rating elements for each key deliverable have been developed and incorporated into the Quality Assurance Rating Forms (see Exhibits A, B, C, and D).

The rating elements and acceptable standards of performance for each key deliverable are described below:

Evaluation Design:

(1) Quality of data collection plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to collecting data and incorporating secondary data sources (when applicable) in the first draft;

(2) Quality of data analysis plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to analyzing quantitative and qualitative data, adequately addressing key research questions, providing a clear conceptual model for analysis in the first draft;

(3) Quality of dissemination plan.

...where acceptable performance would include sound, creditable, comprehensive approaches to disseminating reports and otherwise sharing key research or survey findings with a wide variety of stakeholders in the first draft; and

(4) Comprehensiveness, clarity, and organization of design.

...where acceptable performance would include complete, clear, efficient approaches to addressing the research questions, clear writing style, proper grammar/spelling, clearly organized document format in the first draft.

(5) Responsiveness to ED comments and suggestions.

...where acceptable performance would include thoughtful consideration of ED staff reviewers' comments and suggestions for revisions throughout the drafting process, including written responses to unheeded suggestions to each reviewer should they request it.

(6) Timeliness

...where acceptable performance would include a deliverable that is received on time or within a reasonable amount of time (i.e., within one day of the date that it is due.

(7) Consistency with Individuals with Disabilities Education Act (IDEA)

... where acceptable performance would include clear understanding of the Law

(P.L. 105-17) all deliverables must be consistent with IDEA Part B.

OMB Clearance Process:

(1) Comprehensiveness, clarity, and organization of draft package.

...where acceptable performance would include a complete OMB clearance package, clear writing style, proper grammar/spelling, well-organized document format, accurate and complete descriptions of data collection and data analysis plans;

(2) Quality of draft data collection instruments.

...where acceptable performance would include complete, clear, straightforward data collection instruments, sufficient instrumentation to support data collection analysis plans without being unnecessarily burdensome to respondents; and

(3) Quality of support during clearance process.

...where acceptable performance would include timely, relevant, complete and continuous support during the clearance process, including responding to OMB and/or public questions within three days of each request.

Yearly Reporting:

(1) Accuracy and relevance of information provided.

...where acceptable performance would include complete, clear, logical, appropriate, accurate reporting on data analysis results for key research questions, appropriate context for interpreting results in the first draft;

(2) Usefulness for target audiences.

...where acceptable performance would include clear, tailored language and results for targeted audiences in the first draft; and

(3) Comprehensiveness, clarity, and organization of report.

...where acceptable performance would include comprehensive description of key results, clear writing style, proper grammar/spelling, well-organized document format.

(4) Responsiveness to ED comments and suggestions.

...where acceptable performance would include thoughtful consideration of ED staff reviewers' comments and suggestions for revisions throughout the drafting process, including written responses to unheeded suggestions to each reviewer should they request it.

(5) Timeliness

...where acceptable performance would include a deliverable that is received on time or within a reasonable amount of time (i.e., within one day of the date that it is due.

(6) Consistency with Individuals with Disabilities Act (IDEA)

... where acceptable performance would include clear understanding of the Law

(P.L. 105-17) all deliverables must be consistent with IDEA Part B.

Database:

(1) Cleanliness of data.

...where acceptable performance would include a database where an average of 95% of records pass all edit, consistency and outlier checks; and

(2) User friendliness of documentation and database structure.

...where acceptable performance would include complete, clear, accurate documentation of variable names and labels, value labels, codes for missing values, descriptions of procedures used to compute analysis variables, documentation of all edit and consistency checks used to clean the data, and list of any outliers recoded as part of the cleaning process.

(3) Timeliness

...where acceptable performance would include a deliverable that is received on time or within a reasonable amount of time (i.e., within one day of the date that it is due.

Process of Quality Assurance Assessment

While quality assurance is closely tied to these performance standards for deliverable content, cost is also an important consideration in the assessment of Contractor performance.

The contractor's cost performance will be evaluated by the Department at the end of the contract.

In the event of an excusable delay (as defined in FAR 52.249-14), the Department and the contractor shall work together to modify the contract in regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance, where applicable in this QASP, shall be measured by the date agreed upon in the modification.

The QARP or the COTR will use the appropriate key deliverable evaluation forms (Exhibits A: Evaluation Design; B: OMB Clearance Process; C: Yearly Reporting; and D: Database) to document and evaluate the Contractor's performance for each of the key deliverables under this contract. Each form may be completed independently by each of the QARP members selected for each deliverable assessment, or the deliverable may be evaluated solely by the COTR. If a QARP panel is used, the rating element scores will be averaged for each member to arrive at an "overall" evaluation score. Then, if a QARP is used, an average of the members' overall ratings will generate the final evaluation score for that key deliverable. This final evaluation score will document the QARP's overall evaluation of Contractor performance for that key deliverable. If a QARP panel is not used and only the COTR evaluates the deliverable, the COTR's evaluation of the quality of that deliverable will serve as the overall evaluation score.

Each key deliverable will be evaluated in accordance with the following definitions of contractor performance:

Unacceptable. Level of performance which is not acceptable and which fails to meet the minimum standards of performance, resulting in the contractor receiving a reduction in targeted fee for that deliverable;

Acceptable. Level of performance which meets the minimum standards of performance, resulting in the contractor receiving its targeted fee for that deliverable; or

Superior. Level of performance which exceeds the minimum standards of performance, resulting in a bonus over targeted fee for that deliverable.

Incentive fees for the key deliverables will be assessed as follows:

Evaluation Design

Superior: Target fee plus $5,000 Unacceptable: Target fee minus $8,000

OMB Clearance

Superior: Target fee plus $1,000 Unacceptable: Target fee minus $2,000

Yearly Reporting (per reporting year)

Superior: Target fee plus $2,000 Unacceptable: Target fee minus $3,000

Database

Superior: Target fee plus $5,000 Unacceptable: Target fee minus $9,000

Total dollar amount of fee increase possible due to superior performance (deliverable quality): $27,000.

Total dollar amount of fee decrease possible due to unacceptable performance (deliverable quality): $46,000.

Each review panel member, and/or the COTR, must substantiate, in narrative form, all individual scores which they judge to be indicative of "superior" or "unacceptable" performance. Performance at the "acceptable" level is expected from the Contractor.

The COTR will forward copies of all completed QA monitoring forms (without reviewers' names) and a report of average scores to the CO and Contractor according to the following schedule:

Evaluation design assessment: submitted by the close of business 20 working days from the date the final design was received by the COTR.

OMB clearance process assessment: submitted by the close of business 20 working days from the date OMB clearance is granted.

Yearly reporting assessment: submitted by the close of business 20 working days from the date final reports are received by the COTR.

Database assessment: submitted by the close of business 20 working days from the date it was received by the COTR.

For the purposes of documentation, the Contractor may respond in writing to any "unacceptable" final average evaluation scores within 5 working days after receipt of the form(s); however, this does not mean that the QARP members will change their scores nor does it mean that the average final score will be changed.

The CO will review each key deliverable evaluation form prepared by the QARP and/or the COTR. When appropriate, the CO may investigate the event further to determine if all the facts and circumstances surrounding the event were considered in the QARP opinions outlined on the forms. The CO will immediately discuss every deliverable receiving an "unacceptable" rating with the Contractor to assure that corrective action is promptly initiated. Discussion with the contractor of unacceptable performance or deliverables does not negate the Department's right to terminate the contractor for default for poor performance per FAR 52.249-6, Termination (Cost Reimbursement).

The evaluations of any incentive fees (see (Quality Assurance Surveillance Plan, Attachment A and Section B of the contract) are not subject to the "Disputes" clause (FAR 52.233-1).

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT A: DRAFT EVALUATION DESIGN EVALUATION FORM

Rating Element 1: Quality of Data Collection Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to collecting data lacking a strategy for the incorporation of secondary data sources (when applicable), unnecessarily burdening respondents.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to collecting data and incorporating secondary data sources (when applicable).

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches and/or methods to collecting data, incorporating secondary data sources (when applicable), and reducing respondent burden.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Quality of Data Analysis Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to analyzing quantitative and qualitative data, lack a theory or conceptual model for analysis.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to analyzing quantitative and qualitative data, adequately addressing key research questions, providing a clear conceptual model for analysis.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches and/or methods to analyzing quantitative and qualitative data.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 3: Quality of Dissemination Plan

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, illogical, unclear, inappropriate approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders.

Acceptable performance (5-7) would include sound, creditable, comprehensive approaches to disseminating reports and otherwise sharing key evaluation findings with a variety of stakeholders.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful, multi-media approaches to disseminating reports and otherwise sharing key evaluation findings with a wide variety of stakeholders.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 4: Comprehensiveness, Clarity and Organization of Design

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, unclear, inefficient approaches to addressing the research questions, unclear writing style, poor grammar/spelling, disorganized document format.

Acceptable performance (5-7) would include complete, clear, efficient approaches to addressing the research questions, clear writing style, and proper grammar/spelling, clearly organized document format.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful approaches to addressing the research questions.

Supporting comments (required for unacceptable or superior performance ratings):

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT B: OMB CLEARANCE PROCESS EVALUATION FORM

Rating Element 1: Comprehensiveness, Clarity and Organization of Draft Package

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing OMB clearance package requirements, unclear writing style, poor grammar/spelling, disorganized document format, inaccurate or incomplete descriptions of data collection and data analysis plans, and require major corrections/edits for submission to OMB.

Acceptable performance (5-7) would include a complete OMB clearance package, clear writing style, proper grammar/spelling, well-organized document format, accurate and complete descriptions of data collection and data analysis plans and could be submitted for clearance with only minor corrections/edits.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include submission of the final package to OMB at least one week ahead of schedule for the Year 1 clearance process and at least two weeks ahead of schedule for the Years 2 and 3 clearance process.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Quality of Draft Data Collection Instruments

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include missing, unclear, confusing data collection instruments, insufficient instrumentation to support data collection and analysis plans.

Acceptable performance (5-7) would include complete, clear, straightforward data collection instruments, sufficient instrumentation to support data collection and analysis plans without being unnecessarily burdensome to respondents.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include submission of package to OMB at least one week ahead of schedule for the Year 1 clearance process.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 3: Quality of Support during Clearance Process

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include irrelevant, incomplete, and sporadic or nonexistent support during the clearance process, not responding to public and/or OMB questions or responding more than 3 days after each request.

Acceptable performance (5-7) would include relevant, complete, and continuous support during the clearance process including responding to questions from OMB and/or the public within 3 days of each request.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND provides innovative or exceptionally skillful solutions to problems raised during the clearance process, turns around each request from OMB and/or the public within 2 days of each request.

Supporting comments (required for unacceptable or superior performance ratings):

QUALITY ASSURANCE SURVEILLANCE PLAN

EXHIBIT D: DRAFT REPORT EVALUATION FORM

Rating Element 1: Accuracy and Relevance of Information Provided

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include incomplete, illogical, unclear, inappropriate, inaccurate reporting on data analysis results for key research questions, lack appropriate context for interpreting results.

Acceptable performance (5-7) would include complete, logical, clear, appropriate, accurate reporting on data analysis results for key research questions, appropriate context for interpreting results.

Superior performance (8-10) would meet "acceptable performance" standards for this rating element, AND include innovative, exceptionally skillful reporting on data analysis results for research questions.

Supporting comments (required for unacceptable or superior performance ratings):

Rating Element 2: Usefulness for Target Audiences

Circle the appropriate number for your rating:

1	2	3	4	5	6	7	8	9	10

where:	Unacceptable performance (1-4) would include unclear, inappropriate language and results for targeted audiences.

Supporting comments (required for unacceptable or superior performance ratings):