Today at 10 a.m., the N.C. State Auditor released this audit of the North Carolina Virtual Public School.
The audit states the following as results and conclusions:
“The North Carolina Virtual Public School (NCVPS) is currently offering courses that do not meet the content and design standards established in statute and policy and as advertised in NCVPS’ course guarantee.
Of the 12 NCVPS courses audited, eight courses did not meet required curriculum content standards, and 11 did not meet adopted standards of rigor. All 12 courses are currently offered by NCVPS. In addition, NCVPS did not perform teacher evaluations consistently and in accordance with its policy.
As a result, students taking these courses may not achieve planned learning objectives and be career and college ready.
Lastly, nine of 10 courses tested contained third-party content without proper citation, increasing risk of copyright infringement.”
Here are the findings, recommendations, and responses:
FINDING 1: EIGHT COURSES DID NOT MEET REQUIRED CURRICULUM CONTENT STANDARDS
The North Carolina Virtual Public School (NCVPS) offered eight courses during the 2017-18 and 2018-19 school years that did not meet all applicable curriculum content standards as required by state law and NCVPS policy. As a result, there was an increased risk that students would not be taught all required subject matter. The courses did not meet the curriculum content standards because NCVPS lacked sufficient alignment procedures. State law and NCVPS policy require NCVPS to ensure that its courses meet curriculum content standards.
Curriculum Content Standards Not Met
Of the 12 courses evaluated in this audit and offered during the 2017-18 and 2018-19 school years, eight did not meet all applicable curriculum content standards.
State law required NCVPS honors and general courses to meet the North Carolina Standard Course of Study (NCSCOS) curriculum content standard. Advanced Placement (AP) courses were required to meet the College Board curriculum content standards. Both standards define what students are expected to know and be able to do by the end of each school year or course. The standards also include content requirements for both instruction (teaching) and assessment (testing).
For 2017-18 and 2018-19, NCVPS offered 98 courses including AP courses, honors, and general courses that covered core subjects like English, math, and world languages, plus various electives. Of these 98 courses, auditors selected 10 courses at a higher risk of not meeting the applicable NCSCOS or College Board standards because the revision dates were prior to fiscal year 2017. Auditors also selected two additional courses with more current revision dates of 2017 and 2018.
Subject matter experts evaluated those 12 courses for alignment with NCSCOS and College Board standards. Eight of 12 (67%) courses evaluated did not include all required content for either (1) instruction, (2) assessment, or (3) both. See Table 1 below:
Resulted in Increased Risk That Students Would Not Be Taught All Required Subject Matter
Because the curriculum content for these eight courses did not include all applicable requirements, there was an increased risk that teachers would not cover all of the necessary subject matter.
The NC State Board of Education implemented the NCSCOS curriculum content standards to help ensure that:
All students will graduate from a rigorous, relevant academic program that equips them with the knowledge, skills, and dispositions necessary to succeed in both post-secondary education and 21st Century careers…[Emphasis added]
Additionally, the College Board’s goal for AP courses is to provide “rigorous, college-level classes in a variety of subjects that give students the opportunity to earn college credit while in high school.”
Consequently, failure to align NCVPS courses with NCSCOS or College Board standards could adversely impact college and career preparedness for the students enrolled in these classes.
The eight courses had 4,788 enrollments for the 2017-18 school year and were still being offered as of the 2019-20 school year.
Caused by Insufficient Alignment Procedures
NCVPS lacked sufficient procedures, including documentation requirements, to ensure its courses aligned with state standards.
NCVPS management stated they always include standards alignment in the development process. And the auditors’ subject matter expert noted that “an attempt to include objectives and standards, albeit not always accurate, appears to be part of the design of the NCVPS courses.” However, NCVPS was unable to provide sufficient evidence/documentation of how each course’s curriculum was designed to meet the NCSCOS or College Board standards.
During the audit, NCVPS was asked to demonstrate that the 12 courses being evaluated met the NCSCOS or College Board standards when accurate documentation was not available. NCVPS declined.
State Law and Policy Require NCVPS Curriculum to Meet Content Standards
State law requires NCVPS to ensure that its courses meet curriculum content standards. Specifically, the law requires the Director of NCVPS to ensure that:
All courses offered through NCVPS are aligned to the North Carolina Standard Course of Study (NCSCOS).
Additionally, NCVPS policy requires alignment with the applicable NCSCOS curriculum content standards:
All NCVPS courses will meet the curriculum content standards that are set forth by the NC Department of Public Instruction.
Lastly, the College Board issues the curricular requirements that AP courses must fulfill. The course syllabi for the AP courses reviewed stated, “NCVPS Advanced Placement (AP) courses follow the standards set by College Board.” The requirements outline the required content for each course that colleges and universities typically expect students to master to qualify for college credit and placement.
RECOMMENDATIONS:
NCVPS should revise the curriculum content for the eight courses reviewed so that each course meets the standards of the NCSCOS or College Board, as appropriate.
NCVPS management should develop and implement procedures that ensure that all course curriculum offered meets the applicable NCSCOS or College Board standards. Additionally, documentation of the procedures performed to ensure alignment should be retained.
NCVPS should remove from the NCVPS curriculum all courses that it cannot demonstrate meet the curriculum content standards required by state law and policy.
AGENCY RESPONSE
See page 55 for the Department’s response to this finding.
FINDING 2: NO ASSURANCE THAT ELEVEN COURSES MET ADOPTED STANDARDS FOR RIGOR
The North Carolina Virtual Public School (NCVPS) did not ensure that 11 of the 12 courses evaluated in this audit met standards for cognitive rigor (engaging students in higher-order thinking). As a result, there was an increased risk that students would not master the course material. NCVPS did not ensure the courses met standards for rigor because it did not establish formal methodologies, expectations, or benchmarks to do so. NCVPS policy requires its courses to meet the cognitive rigor standards known as the Revised Bloom’s Taxonomy.
Did Not Ensure Courses Met Standards
NCVPS did not ensure that 11 of the 12 courses evaluated in this audit met its adopted standards for cognitive rigor because they did not establish methodologies to do so. These adopted standards, the Revised Bloom’s Taxonomy (RBT), are designed to create courses that engage students in higher-order thinking and allow them to demonstrate mastery of the subject matter. During the 2017-18 and 2018-19 school years, NCVPS offered 98 courses including Advanced Placement (AP) courses, honors, and general courses that cover core subjects like English, Math, and World Languages, plus various electives. Of these 98 courses, auditors selected 10 courses at a higher risk of not meeting the RBT standards because the revision dates were prior to fiscal year 2017. Auditors also selected two additional courses with more current revision dates of 2017 and 2018.
Subject matter experts evaluated these 12 courses, comparing the curriculum (instruction and assessments) of each course to the RBT. RBT defines the kind of knowledge and type of thinking students are expected to demonstrate based on six levels of rigor.
According to the professional expertise of our subject matter experts, at least half of the course material for 11 of 12 (92%) courses evaluated was found to be in level 1 and 2 of RBT (rigor too low). Level 1 and 2 of RBT is comprised of the least engaging type of instruction (such as lectures or notes) and the lowest level type of assessments (such as online multiple-choice quizzes). For example:
- AP Art History – 100% of the instruction material was low rigor, and 52% of the assessments were low rigor
- AP Environmental Science – 93% of the instruction material was low rigor, and 51% of the assessments were low rigor
- AP Physics – 85% of the instruction material was low rigor, and 29% of the assessments were low rigor
- Latin I – 84% of the instruction material was low rigor, and 43% of the assessments were low rigor
- AP Government and Politics – 82% of the instruction material was low rigor, and 85% of the assessments were low rigor
- Physical Science – 59% of the instruction material was low rigor, and 45% of the assessments were low rigor
NOTE: AP Environmental Science and AP Government and Politics were revised post fiscal year 2017 under the NCVPS’ new course development and revision process. Even though these courses were revised under NCVPS’ new process, the courses still did not meet RBT standards based on the subject matter experts’ testing.
Resulted in Increased Risk That Students Would Not Master Course Material
Courses with instruction and assessments rated as low rigor are at a greater risk of being insufficiently rigorous overall and failing to require complex thinking and application of knowledge.
Consequently, students taking these courses may not have mastered the course material.
The 11 courses that have at least half of course material in level 1 and 2 of RBT (rigor too low) had 5,836 enrollments for the 2017-18 school year and are still being offered as of the 2019-20 school year.
Caused by Not Measuring Results from Course Development
NCVPS did not establish a method for measuring how well the courses met the RBT standards for rigor during the course development process. For example, NCVPS did not establish formal methodologies, frameworks, or benchmarks to measure how well the courses were designed to engage students in higher-order thinking and demonstrate a mastery of the subject matter.
Policy Requires Courses to Meet RBT Cognitive Rigor Standards
NCVPS policy requires its courses to meet the standards of the Revised Bloom’s Taxonomy. Specifically, NCVPS’ course guarantee states that NCVPS will create high quality, engaging content grounded in the Revised Bloom’s Taxonomy and integrated with real-world project-based activities.
RECOMMENDATIONS
NCVPS should ensure the curriculum content for the 11 courses reviewed meets its adopted cognitive rigor framework, the Revised Bloom’s Taxonomy. NCVPS should develop and implement formal procedures for developing courses that measure how well all courses meet its adopted cognitive rigor framework, the Revised Bloom’s Taxonomy. NCVPS should remove from the NCVPS curriculum all courses that it cannot demonstrate meet the adopted rigor standards required by NCVPS policy and as advertised in NCVPS’ course guarantee.
AGENCY RESPONSE
See page 56 for the Department’s response to this finding.
FINDING 3: NCVPS TEACHER EVALUATIONS NOT PERFORMED CONSISTENTLY AND IN ACCORDANCE WITH POLICY
North Carolina Virtual Public School (NCVPS) teacher evaluations were not performed consistently and in accordance with NCVPS policy. The inconsistent evaluations increased the risk that poor performance would not be timely identified and corrected. A lack of monitoring allowed the inconsistent evaluations to occur without correction.
Teacher Evaluations Not Performed Consistently and In Accordance with Policy
NCVPS instructional leaders did not perform teacher evaluations consistently and in accordance with NCVPS policy.
Auditors reviewed 62243 teacher evaluations performed during the 2017-18 school year and found:
- 136 (22%) evaluations contained no evidence of review or were not timely reviewed by the Instructional Director 44
- 120 (19%) evaluations contained no evidence that teacher performance issues were addressed
- 82 (13%) evaluations contained incorrect ratings according to the instructions on the NCVPS evaluation document45
- 14 (2%) evaluations were missing required information such as the teacher’s response
Resulted in Increased Risk of Undetected Poor Performance
Failure to perform and document teacher evaluations consistently and in accordance with NCVPS policy increases the risk that poor performance will not be identified and corrected timely.
Consequently, there was an increased risk that NCVPS students could have received substandard instruction and assessment for an extended period.
Caused by Lack of Monitoring Procedures
NCVPS did not monitor its teacher evaluation process to ensure it was performed consistently and in accordance with its policy.
Specifically, there is no documentation that:
- Describes the procedures used for monitoring the teacher evaluation process
- Shows evidence of NCVPS management monitoring the teacher evaluation process
- Demonstrates correction of issues identified when monitoring the teacher evaluation process
NCVPS Procedures List Teacher Evaluation Requirements
The NCVPS teacher evaluation document includes instructions for completing the evaluations. Another document46 defines the purpose and frequency of teacher evaluations.
Together, these two documents require:
- Instructional leaders to perform teacher evaluations bi-weekly, or monthly depending on the teacher’s tenure.
- Teachers to respond to the evaluations and any needed coaching and action plans.
- Instructional Directors to review, provide comments, and initial the evaluations.
RECOMMENDATIONS
NCVPS should develop and implement procedures to ensure that teacher evaluations are performed consistently and in accordance with NCVPS policy. Monitoring should be performed that is continual and responsive to change.
AGENCY RESPONSE
See page 58 for the Department’s response to this finding.
FINDING 4: NCVPS COURSE CONTENT NOT PROPERLY CITED TO AVOID COPYRIGHT INFRINGEMENT
North Carolina Virtual Public School (NCVPS) did not ensure that all third-party content in its courses was appropriately cited. Lack of proper citation increased the risk of copyright infringement and potential litigation from publishers and authors. The courses included uncited content because NCVPS did not perform quality assurance procedures on all currently offered courses. However, NCVPS policy requires compliance with copyright law and proper citation.
Course Content Not Properly Cited
NCVPS did not ensure that all third-party content in its courses was appropriately cited.
A citation should clearly identify source of third-party content and include at minimum “the author’s last name and the year of publication for the source.”
During the 2017-18 and 2018-19 school years, NCVPS offered 98 Advanced Placement (AP), general, and honors courses that covered core subjects like English, Math, World Language, and electives.
Auditors selected 10 of the 98 (10%) courses that NCVPS offered during the 2017-18 and 2018-19 school years. Auditors used originality detection software and determined that nine of 10 (90%) courses contained about 1% to 15% of third-party content without proper citation as shown in Table 2 below.
Examples of uncited third-party content included:
- Questions on a Physical Science assessment found on a practice test published by a well-known educational publisher.
- A question on a Pre-Calculus assessment found in a lesson published by another educational publisher.
- The introduction to a module in the African American Studies course found in a book about African contributions to American culture
Resulted in Increased Risk of Copyright Infringement
Failure to ensure that third-party content was properly cited increased the risk of copyright infringement and potential litigation from publishers and authors of copyrighted work.
Caused by Lack of Quality Assurance Procedures
NCVPS courses lacked appropriate citation because NCVPS did not implement the necessary quality assurance procedures for all currently offered courses. Specifically, NCVPS did not use originality detection software on all currently delivered courses.
NCVPS began using originality detection software to scan courses developed or revised during the 2017-18 school year. However, NCVPS did not perform the same quality assurance procedures on courses developed before then.
NCVPS Policy Requires Compliance with Copyright Law
NCVPS policy requires that all third-party content in its courses are appropriately cited. Specifically, the NCVPS copyright policy states:
The North Carolina Virtual Public School respects the intellectual property rights of copyright holders and complies with copyright law of the United States. All administrators, staff, instructors, and students are required to comply with copyright law and guidelines.
Additionally, NCVPS’s teacher orientation provides resources on APA (American Psychological Association) citations. The basic format of APA in-text citation requires “the author’s last name and the year of publication for the source” to appear in the text.
RECOMMENDATIONS
NCVPS management should use originality detection software to ensure that all third-party material is properly cited in all NCVPS courses.
AGENCY RESPONSE
See page 59 for the Department’s response to this finding.
In the agency response, the state Department of Public Intsruction disagreed with some of the findings and agreed with others.
On Finding 1, the response states: “DPI disagrees with the finding that eight out of the twelve courses audited did not meet required curriculum content standards. Determination related to content alone meeting a standard is a subjective test, with varying results from one person to another, including the third-party vendor. The role of the teacher and related instruction were not evaluated as apart of the audit.”
On Finding 2, the response states: “NCDPI disagrees with the finding that 11 of 12 NCVPS courses audited do not meet adopted standards for rigor, increasing the risk that students may not master course material and the subsequent recommendations.”
On Finding 3, the response states: “DPI agrees with the finding that NCVPS teacher evaluations were not performed consistently. A previous DPI internal audit delivered to OSA identified the same finding, and DPI has already taken steps to resolve this finding.”
On Finding 4, the response states: “DPI agrees with the finding that some course content was not properly cited. A previous DPI internal audit delivered to OSA identified the same finding, and DPI has already taken steps to resolve this finding.”
Here is the audit in full.
