STEAM Program Evaluation, 2024

The purpose of this program evaluation is to assess the effectiveness and impact of the Black River Innovation Campus’ (BRIC) STEAM programming, which began in February 2024. The evaluation aims to provide valuable insights into how well the program is achieving its goals of increasing math and science scores within the Windsor County school district and neighbouring regions, as well as promoting participation in STEAM activities among underserved members of the community. The scope of the evaluation includes examining student outcomes, educator professional development, community impact, and the financial aspects of the program. By evaluating these components, the evaluation seeks to identify strengths and areas for improvement, ultimately guiding the program’s future development and ensuring its sustainability and relevance to the community’s needs.

Student Outcomes

Math and Science Scores

Proficiency in these subjects lays the foundational skills necessary for success in more advanced STEM fields.

2011

National Research Council

As the National Research Council (2011) notes, strong mathematical and scientific abilities are critical for developing technological and engineering skills.

2012

Hossain & Robinson

there is a clear link between high performance in these subjects and better educational and career outcomes in STEM fields, which underscores the long-term benefits of tracking these metrics (Hossain & Robinson, 2012)

2016

U.S. Department of Education

Secondly, math and science scores are typically measured using standardized tests, providing a consistent and objective framework to assess student performance across various schools and regions (U.S. Department of Education, 2016)

2018

The White House Office of Science and Technology Policy

Robust math and science education contributes to a country’s economic growth and global competitiveness, as emphasized by the White House Office of Science and Technology Policy (2018)

2016

U.S. Department of Education

By monitoring these scores, educators and policymakers can identify and address achievement gaps among different student groups, ensuring equitable access to quality STEM education (U.S. Department of Education, 2016).

Because our programming is generally used to augment existing curriculum and is not usually directly tied to testing, a direct measurement of test scores may not be possible. However, we can gauge students’ interest and confidence with the material through surveys and questionnaires. Studies show a significant correlation between students' math and science scores and their interest and comfort levels with the material. For instance, Wang (2002) found a moderate relationship between math and science achievements, suggesting proficiency in one subject positively impacts the other. Blotnicky et al. (2018) demonstrated that higher mathematics self-efficacy is associated with greater interest in STEM careers. Cerbito (2020) highlighted that positive attitudes, such as confidence and motivation, are linked to higher proficiency in mathematics. Lastly, Tapia and Marsh (2005) explored how students' conceptual understanding in math relates to their learning outcomes, including math test scores and academic self-efficacy. Collectively, these studies underscore the importance of math and science scores as indicators of students' interest and comfort levels, which can influence their future educational and career choices.

STEAM Activity Participation

According to our summer camp post-program likert evaluations (from 1-5, with 5 being most likely),

3.78

out of 5

How likely would you be to join camp again next year?

4.27

out of 5

how likely would you be to recommend this camp to a friend?

YES

Maybe

Are you interested in pursuing a career in coding, computer science, or a related field?

According to our summer camp post-program likert evaluations (from 1-5, with 5 being most likely), when students were asked the question “how likely would you be to join camp again next year?” students averaged **3.78 out of 5**, showing high interest in our programming. When asked the question “how likely would you be to recommend this camp to a friend?” students on average scored us at **4.27 out of 5**, showing a very high likelihood. When students were asked “Are you interested in pursuing a career in coding, computer science, or a related field?” students responded with **13 “yes,” 16 “maybe,” and 4 “no,”** showing a high interest in a STEAM-related career. Specific careers listed included “game development/game design,” “computer science or game design,” “civil Engineering,” “game design in VR,” and “Data Analysis, App development,” to name a few. When asked the question “how would you rate your overall experience at the camp?” students on average responded with a **4.6 out of 5**, showing high interest/satisfaction with the camps.

Students in our summer programming participated from **11** neighboring towns (**12** separate schools) in southern Vermont and western New Hampshire, with a gender ratio of **3:1** male-to-female. Our largest grade-level demographic was 6th grade at 29% of campers, followed by 9th grade at 21% and 8th grade at 17%.

Our LEGO Robotics Community Daysworkshops brought in **40** 1st-5th grade students, parents, as well as **11** educators from **18** different zip codes. Interest was high in continuing these community build days.

Our Elementary-aged LEGO Robotics afterschool programs are ongoing, but reach **24** students at Weathersfield school in Weathersfield, VT, and **12-16** students at SAPA TV in Springfield, VT. Students are generally in 4th or 5th grade, with a few from 2nd or 3rd grade. These students do not have any other STEM-related programming in their areas.

Our CoSpaces-related co-curricular programming at Compass school impacts **6** high school students, with **2** students excited to enter a tech or coding-related field.

Our game design curriculum centered around the software MakeCode, Endstar, and Godot, created in partnership with Endless Studios, impacted three students at River Valley Tech Center; two males and one female. Two out of three students (both male) scored a 4 out of 5 for interest in a STEM-related career. Expansion of these programs is planned with nine additional students further along in the school year, as well as integration into after-school programs.

Our Fab Lab has run two workshops so far, impacting **20** individuals with a wide age range. Free, regular programming will begin in January.

Free, publicly-available STEM-related workshops or programs, such as LEGO robotics build days, play a crucial role in increasing equity and promoting career pathways into STEM fields. According to experts, these workshops provide underserved and underrepresented students with access to hands-on learning experiences and exposure to STEM careers, which they might not otherwise have. By removing financial barriers, these programs ensure that all students, regardless of their socioeconomic background, have the opportunity to develop essential STEM skills and build confidence in their abilities. This increased access helps to diversify the STEM workforce, addressing the growing demand for a specialized and inclusive STEM-focused workforce. Additionally, these workshops can inspire students to pursue further education and careers in STEM fields, contributing to a more equitable and innovative society (Harvard Gazette, 2021).

Skill Development

Our summer campers and workshop students who interfaced with technology tools, many of whom had not previously used a mouse or keyboard, demonstrated remarkable improvement in computer literacy by the end of their sessions. Our summer campers became proficient in navigating the CoSpaces environment using a mouse and keyboard, and designing 3D maps that were VR-ready. Collaborative projects were common, with students sharing their work and exploring new concepts like block coding together. This collaboration facilitated natural growth and the exchange of ideas, resulting in students creating rudimentary User Interfaces for their games, coding events, and even setting up primitive AI systems. Game theory and design concepts were easy for students to pick up as well, due to the nature of CoSpaces and the desire of students to make a product that was engaging and fun for the user.

Our workshop students routinely show immediate improvement in computer literacy skills, particularly when using CoSpaces, due to the high level of engagement from the students in the activity (it’s fun.) A desire to improve their projects most often results in an intrinsic interest in block coding, unlocking future potential learning opportunities for the students.

Our Summer Camp students focusing on airfoils iteratively designed and tested paper and wooden airplanes to maximize flight duration. By applying the scientific method, they learned how air moves over a wing to create lift. Similarly, our laser-cutting students enhanced their design skills and mastered the software tools necessary to assign cutting layers accurately.

Robotics camps and Elementary LEGO Leagues provided students with foundational skills in computer usage, block coding, and mathematical concepts like measurement, angles, rotational speed, and torque. These experiences collectively contributed to the students' comprehensive skill development, equipping them with valuable STEAM competencies.

These skill improvements align with the Common Core State Standards (CCSS), Next Generation Science Standards (NGSS), as well as Technology and Engineering Standards (ITEEA) listed below.

Mathematics Standards

CCSS.MATH. CONTENT .3.G.A.1:

Understand the concept of a variable and be able to use variables to represent numbers and relationships in problem-solving contexts.

CCSS.MATH. CONTENT. 4.MD.A.2:

Use the four operations to solve problems involving measurement and conversion of measurements.

CCSS.MATH. CONTENT .5.MD.C.5:

Relate the unit circle and the definition of radian measure to the arc length of a circle and the circumference of a circle.

CCSS.MATH. CONTENT .6.EE.A.3:

Apply the properties of operations to generate equivalent expressions and solve problems involving expressions and equations.

Next Generation Science Standards (NGSS)

NGSS .ETS1 .A:

Understand the concept of a variable and be able to use variables to represent numbers and relationships in problem-solving contexts.

NGSS.ETS1.B:

Evaluate and refine a prototype or model using a range of tests.

NGSS.ETS1.C:

Analyze and interpret data to identify solutions and/or improvements to problems.

Technology and Engineering Standards

ITEEA Standards for Technological and Engineering Literacy:

Students demonstrate understanding of the nature of technology and engineering, including the ability to design, create, and use technological solutions to solve problems.

Educator Professional Development

Skill Development

Professional development and community workshop opportunities from February 2024 until December 15th, 2024 impacted a total of around 40 educators across Vermont and New Hampshire, as well as around 10 community members. Topics covered were the development and use of AI tools and technology, CoSpaces, and LEGO Robotics. Skill growth centered around the computer science mindset, which incorporates problem-solving, analytical thinking, continuous learning, problem-oriented approaches, communication skills, time management, and quick learning.

While our CoSpaces workshops saw interest from educators participating in the professional development, curriculum implementation of the platform was minimal, with only 2 of 12 teachers continuing use of the platform after the initial training. However, the platform was used as a basis for a day-long workshop for 7th and 8th grade females (Girl Powered Event, sponsored by River Valley Tech Center in Springfield, Vermont) as a result of this training.

While our CoSpaces workshops saw interest from educators participating in the professional development, curriculum implementation of the platform was minimal, with only **2 of 12** teachers continuing use of the platform after the initial training. However, the platform was used as a basis for a day-long workshop for 7th and 8th grade females (Girl Powered Event, sponsored by River Valley Tech Center in Springfield, Vermont) as a result of this training. Teachers gained skills in computer literacy, learning to manipulate the mouse and camera, as well as movement and creation tools within CoSpaces. Examples of lessons and curriculum were provided as well, to show the potential of the platform. Basic block coding was also introduced, such as movement of an object from one place to another and rotating objects. There is a possibility of this platform becoming integrated into the curriculum in the future at Compass school as well. Workshops using CoSpaces should be at minimum an hour and a half, to provide educators with enough time to understand and begin to enjoy the platform.

BRIC hosted 1 workshop on AI Tools and Technology, which was well-received by the community. Attendees were most interested in AI tools and their use, as well as environmental impacts of high-compute-power AI systems, such as Large Language Models (ChatGPT). Attendees left with an improved understanding of AI and its place in both professional and personal workflows.

Our LEGO SPIKE Robotics clubs saw the most impact on student and teacher upskilling, but affected less teachers than our CoSpaces programming. Through our LEGO clubs and community days, educators learned about motors, sensors, and HUB usage, basic coding concepts like sequences, loops, and conditionals, basic building techniques to accomplish actions on the competition board, how to connect and program motors on the robot, problem solving skills, collaborative learning, and how to incorporate LEGO Spike robots into their curriculum or afterschool programs. Currently, our programming centered around LEGO Spike Robotics has seen the most engagement out of any other programming, and opportunities to incorporate LEGO robotics into school curriculum is expected to increase in the following year..

CoSpaces was presented in two workshop sessions at the VitaLearn educator conference at Killington VT. There were around 25 attendees from numerous school districts in Vermont and New Hampshire. The workshops sparked four follow-up conversations, one of which has resulted in interest in our game theory/design course being created by our partner, Endless Studios.

Instructional Practices

Additional data is needed to assess if our professional development has impacted teacher methodology in any way. As of today, no meaningful instructional practices have been altered to our knowledge.

Satisfaction and Feedback

Additional data is needed to assess teacher satisfaction with our professional development programming. Pre and post surveys need to be created and collected in future professional development opportunities.

Program Funding/Cost

Our programming has impacted a total of 288 community members (of all ages). In order to achieve greater accuracy for our programs cost-effectiveness, additional data will need to be gathered in the areas of test scores, interest in STEAM careers pre and post-programming, and overall subject matter interest pre and post-programming. Based on our current data, it costs BRIC $293 a person for our combined programming. Our funding sources for this years’ programming are from Mascoma Bank Foundation, Claremont Savings Bank Foundation, Dalio Philanthropies, Cosmos Fund LLC, and the Vermont Better Places Grant.

Our limited survey results seem to indicate, as reflected earlier in this program evaluation, a high interest in our programming overall by both youth and adults/educators, and a desire for the programming to continue and expand.

This $293 cost for BRIC programming (which is currently free for all participants) compares very favorably against other STEM-related programming available to students around Vermont. Our student capacity is somewhat limited by available teaching space and instructor availability, but averages at 12 students per session. Here is a list of five comparison programs and their costs per person:

Vermont State University Youth Camps:

Located in Randolph, Vermont, these camps offer hands-on experience in science, health, advanced manufacturing, and more. The cost is free for eligible Vermont residents (high school, 20 students per session).

Aloha Foundation Camps:

Situated in Fairlee, Vermont, these camps provide a variety of STEM-focused activities. 2025 tuition is $13,650 for the full session, $8,950 for a half session, and $5,950 for the 2-week session (capacity varies, focuses on low camper:counselor ratio).

Audubon Nature Camp:

Located in Huntington, Vermont, this nature-themed camp offers STEM activities integrated with environmental education. The cost is around $300 per week (scholarships available, 10-15 students per instructor).

iD Tech Camps at University of Vermont:

Held in Burlington, Vermont, these camps focus on coding, robotics, and other STEM skills. The cost starts at $899 per week, with options for day and overnight camps (small group or 1-on-1 learning).

The Governor's Institutes of Vermont (GIV)

offers a sliding scale tuition structure to ensure accessibility for all students, regardless of their financial background. The tuition ranges from $10 to $2,650 depending on family income (serves around 1,674 students, or 2% of all Vermont high school students, across all programming).

BRIC’s STEAM programming budget utilization can be broken down into a few categories:

1. *Salary* BRIC has one STEAM-related salaried employee at a rough cost of **$70,000**.
2. *Supplies* - Supplies for programming for the fiscal year 2024 totaled to roughly **$11,000**, not counting the purchase of six 3d-printers, a vinyl cutter, screen printer and a laser cutter (funding covered from a $10,000 grant and crowd-funding campaign.)
3. *Teacher Compensation* - Teachers for BRIC’s summer programming received $2000 a week, totaling **$10,000**.

BRIC’s STEAM programming is currently 100% grant-funded. The long-term viability of a 100% grant-funded STEAM program presents several challenges and opportunities. While grants can provide essential initial funding to launch and sustain programs, relying solely on grants can create financial instability due to the competitive and uncertain nature of grant awards (Henderson, 2018). To ensure sustainability, it is crucial for such programs to diversify their funding sources, including seeking partnerships with local businesses, securing sponsorships, and implementing fundraising initiatives (Markowitz, 2020). Building a robust network of supporters and demonstrating the program's impact through measurable outcomes can also attract more consistent funding streams (Smith & Roberts, 2021). Developing a strategic plan that includes financial sustainability measures can help mitigate the risks associated with grant dependency and ensure the program's long-term success and growth (Wilson & Goforth, 2019). BRIC is currently in the process of finding ways to reduce or eliminate our STEAM programming’s reliance on grants, and has a roadmap drafted towards that end, which includes creating paid workshops and professional development.

Works Cited

Aloha Foundation Camps. (n.d.). Aloha Foundation Programs. Retrieved from Aloha Foundation
Audubon Nature Camp. (n.d.). Audubon Nature Camp. Retrieved from Audubon Vermont
Blotnicky, K. A., Franz-Odendaal, T., French, F., & Joy, P. (2018). A Study of the
Correlation Between STEM Career Knowledge, Mathematics Self-Efficacy,
Career Interests, and Career Activities.
Cerbito, A. F. (2020). Comparative Analysis of Mathematics Proficiency and
Attitudes Toward Mathematics of Senior High School Students.
Governor's Institutes of Vermont. (n.d.). Tuition and Costs. Retrieved from GIV Website
Gülhan, F. (2023). Parental Involvement in STEM Education: A Systematic Literature Review. European Journal of STEM Education, 8(1), 05.
Harvard Gazette. (2021). Increasing access and opportunity in STEM crucial, say experts. Retrieved from Harvard Gazette
Henderson, S. (2018). Grant Funding in STEM Education: Challenges and Opportunities. Journal of STEM Education, 19(2), 45-56.
Hossain, M., & Robinson, M. G. (2012). How to Measure and Evaluate Impact of STEM Initiatives in Education. Journal of Science Education and Technology.
iD Tech Camps at University of Vermont. (n.d.). Camps at University of Vermont. Retrieved from iD Tech
Markowitz, D. (2020). Diversifying Funding Sources for Educational Programs. Education Finance Journal, 22(3), 78-90.
Milner-Bolotin, M., & Marotto, C. C. F. (2018). Parental engagement in children’s STEM education. Part I: Meta-analysis of the literature. LUMAT:
International Journal on Math, Science and Technology Education, 6(1), 41-59.
Murcia, K., Pepper, C., & Williams, J. (2020). Youth STEM Career Choices: What’s Influencing Secondary Students’ Decision Making. Issues in
Educational Research, 30(2), 593-611.
National Research Council. (2011). Successful K-12 STEM Education: Identifying Effective Approaches in Science, Technology, Engineering, and
Mathematics. National Academies Press.
Punzalan, C. H. (2022). STEM Interests and Future Career Perspectives of Junior High School Students: A Gender Study. International Journal of
Research in Education and Science, 8(1), 93-102.
Shulga, T. I., Zaripova, Z. F., Sakhieva, R. G., Devyatkin, G. S., Chauzova, V. A., & Zhdanov, S. P. (2023). Learners’ career choices in STEM education: A review of empirical studies. EURASIA Journal of Mathematics, Science and Technology Education, 19(5), em2261.
Smith, J., & Roberts, L. (2021). Measuring Impact in STEM Education Programs. International Journal of Education and Development, 31(1), 112-130.
Tapia, M., & Marsh, H. W. (2005). How is the Conceptual Level of Students Associated with Learning Outcomes?
The White House Office of Science and Technology Policy. (2018). Charting a Course for Success: America’s Strategy for STEM Education.
U.S. Department of Education. (2016). STEM 2026: A Vision for Innovation in STEM Education.
Vermont State University Youth Camps. (n.d.). Youth Camps. Retrieved from Vermont State University
Wang, J. (2002). Relationship Between Mathematics and Science Achievement at the 8th Grade.
Wilson, K., & Goforth, T. (2019). Strategies for Financial Sustainability in Education Programs. Educational Leadership Review, 28(4), 55-67.

Lifestyle

STEAM

Boost Your Business with E-Commerce & Web Presence

Campus

Contact

STEAM Program Evaluation, 2024

Student Outcomes

Educator Professional Development

Program Funding/Cost

Works Cited

Tour our Facilities

Contact Us