SIGCSE 2020 Online - Papers


Session 1A: Security A

What Are Cybersecurity Education Papers About? A Systematic Literature Review of SIGCSE and ITiCSE Conferences

Authors: Valdemar Švábenský; Jan Vykopal; Pavel ?eleda


3rd Best Paper, CS Education Research Track

Cybersecurity is now more important than ever, and so is education in this field. However, the cybersecurity domain encompasses an extensive set of concepts, which can be taught in different ways and contexts. To understand the state of the art of cybersecurity education and related research, we examine papers from the ACM SIGCSE and ACM ITiCSE conferences. From 2010 to 2019, a total of 1,748 papers were published at these conferences, and 71 of them focus on cybersecurity education. The papers discuss courses, tools, exercises, and teaching approaches. For each paper, we map the covered topics, teaching context, evaluation methods, impact, and the community of authors. We discovered that the technical topic areas are evenly covered (the most prominent being secure programming, network security, and offensive security), and human aspects, such as privacy and social engineering, are present as well. The interventions described in SIGCSE and ITiCSE papers predominantly focus on tertiary education in the USA. The subsequent evaluation mostly consists of collecting students' subjective perceptions via questionnaires. However, less than a third of the papers provide supplementary materials for other educators, and none of the authors published their dataset. Our results provide orientation in the area, a synthesis of trends, and implications for further research. Therefore, they are relevant for instructors, researchers, and anyone new in the field of cybersecurity education. The information we collected and synthesized from individual papers are organized in a publicly available dataset.

Integration and Evaluation of Spiral Theory based Cybersecurity Modules into core Computer Science and Engineering Courses

Authors: Debarati Basu; Harinni K. Kumar; Vinod K. Lohani; N. Dwight Barnette; Godmar Back; Dave McPherson; Calvin J. Ribbens; Paul Plassmann


Cybersecurity education has been emphasized by several national organizations in the United States, including the National Academy of Engineering, which recognizes securing cyberspace as one of the 14 Engineering Grand Challenges. To prepare students for such challenges and to enhance cybersecurity education opportunities at our large research university, we implemented an NSF-funded cybersecurity education project. This project is a collaborative effort between faculty and graduate students in the Engineering Education, Computer Science (CS) and Computer Engineering (CPE) departments at a major US research university. In this effort, we integrated cybersecurity learning modules into multiple existing core CS and CPE courses following Jerome Bruner's spiral-theory model, which has previously been used to reformulate several academic curricula.

In this paper, we present our cybersecurity curriculum initiative, describe the spiral-theory based process we developed to implement the curriculum and provide an in-depth description of four reusable cybersecurity learning modules that we developed. A core tenet of spiral theory holds to revisit topics as students advance through their curriculum. This work applies this approach to Cybersecurity education by carefully designing the learning objectives of the modules and its contents. For evaluating these learning modules we implemented pre and post-tests to assess students' technical knowledge, their perceptions towards the modules' learning objectives, and how it influenced their motivation to learn cybersecurity. Our findings are overwhelmingly positive and the students' feedback has helped us improve these learning modules. Since its inception, our initiative has educated more than 2,000 students and is currently being used to revise the affected courses' syllabi.

A Case Study of a Cybersecurity Programme: Curriculum Design, Resource Management, and Reflections

Authors: Muhammad Rizwan Asghar; Andrew Luxton-Reilly


Cybersecurity is an area of growing international importance. In response to global shortages of Cybersecurity skills, many universities have introduced degree programmes in Cybersecurity. These programmes aim to prepare students to become Cybersecurity practitioners with advanced skills in a timely manner. Several universities offer Cybersecurity degrees, but these have been developed ad hoc, as there is currently no internationally accepted Cybersecurity curriculum.

Recently, an ITiCSE working group on global perspectives on Cybersecurity education developed a competency-based framework that aims to help institutions to implement Cybersecurity programmes. In this report, we present a case study of a Cybersecurity programme at the University of Auckland. We discuss how the curriculum and resource management of this programme evolved, and we present some challenges for the design and delivery of a Cybersecurity programme in the light of this competency-based framework.

Session 1B: Student Experiences

A Longitudinal View of Gender Balance in a Large Computer Science Program

Authors: Amy Baer; Andrew DeOrio


Computer Science has a persistent lack of women's participation. In order to best effect change, we require a more fine-grain analysis of the gender disparity as it changes throughout an undergraduate Computer Science curriculum. In this paper, we use a quantitative approach to highlight, with greater specificity, the point in an undergraduate career where gender balance changes. We also examine the role of grades in students' decisions to stay in the course sequence. Our goal is to enable targeted interventions that will make Computer Science a more welcoming discipline. Our study examines 30,890 unique student records over ten years at a large, public research institution. The records include students who took a Computer Science course over the past ten years. The dataset contains information about gender, majors, minors, academic level, and GPA. The dataset also includes a record from each course taken by each student and their final grade. We observed a modest increase in women's participation in all Computer Science courses over the past ten years. Despite this increase, the gender disparity is still large. Through our analysis, we found that women consistently choose not to continue through the Computer Science sequence at a higher rate than men. This higher attrition could be linked to women receiving lower grades in most introductory CS courses despite having the same or higher GPAs than men. Our results reveal specific areas where intervention can be the most effective in changing the stubborn gender disparity in Computer Science.

Identifying the Prevalence of the Impostor Phenomenon Among Computer Science Students

Authors: Adam Rosenstein; Aishma Raghu; Leo Porter


The Impostor Phenomenon (IP) is often discussed as a problem in the field of computer science, but there has yet to be an empirical study to establish its prevalence among CS students. One survey by the Blind app found that a high number of software engineers at some of the largest technology companies self-reported feelings of Impostor Syndrome; however, self-reporting of Impostor Syndrome is not the standard diagnostic for identifying whether an individual exhibits feelings of the Impostor Phenomenon. In this work, the established Clance IP Scale is used to identify the prevalence of IP among graduate and undergraduate computer science students at a large research-intensive North American institution. Among this population of over 200 students, 57% were found to exhibit frequent feelings of the Impostor Phenomenon with a larger fraction of women (71%) experiencing frequent feelings of the Imposter Phenomenon than men (52%). Additionally, IP was found to have greater prevalence among computer science students than among students of other populations from comparable studies. Due to the negative impacts associated with feelings of the Impostor Phenomenon, computer science education should work to improve student awareness and help student cope with these feelings.

A Multi-level Analysis of the Relationship between Instructional Practices and Retention in Computer Science

Authors: Markeya Peteranetz; Leen-Kiat Soh


Increasing retention in computer science (CS) courses is a goal of many CS departments. A key step to increasing retention is to understand the factors that impact the likelihood students will continue to enroll in CS courses. Prior research on retention in CS has mostly examined factors such as prior exposure to programming and students' personality characteristics, which are outside the control of undergraduate instructors. This study focuses on factors within the control of instructors, namely, instructional practices that directly impact students' classroom experiences. Participants were recruited from 25 sections of 14 different courses over 4 semesters. A multi-level model tested the effects of individual and class-average perceptions of cooperative learning and teacher directedness on the probability of subsequent enrollment in a CS course, while controlling for students' mastery of CS concepts and status as a CS major. Results indicated that students' individual perceptions of instructional practices were not associated with retention, but the average rating of cooperative learning within a course section was negatively associated with retention. Consistent with prior research, greater mastery of CS concepts and considering or having declared a CS major were associated with a higher probability of taking a future CS courses. Implications for findings are discussed.

Session 1C: Automated Systems

Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style

Authors: Abe Leite; Saúl A. Blanco


The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes.

Automated Assessment of the Visual Design of Android Apps Developed with App Inventor

Authors: Igor Solecki; João Porto; Nathalia Alves; Christiane Gresse von Wangenheim; Jean Hauck; Adriano Ferreti Borgatto


One way to teach computing in K-12 is through the development of mobile applications with App Inventor. Although already broadly used worldwide, there is still a need for support for the assessment of the applications created by the students. Existing rubrics focusing mostly on programming concepts do not cover more comprehensively the performance-based assessment of user interface design concepts, important for the usability and aesthetics of the applications. Thus, in order to support the assessment of the visual design of apps based on its compliance with design theory and guidelines, we developed the CodeMaster UI Design - App Inventor rubric in the context of computing education. In order to facilitate its application in practice, we automated the assessment of applications created with App Inventor through a static code analysis by an online tool. We evaluated the reliability and validity of the rubric based on the automated assessment of 1,775 projects from the App Inventor Gallery. The results indicate that the rubric can be considered reliable (Cronbach's alpha = 0.84). In terms of construct validity, there is also evidence of convergent validity. The results presented in this article can be used to support the assessment of computing education in practice as well as to point out further research opportunities.

Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned

Authors: Hassan Khosravi; Shazia Sadiq; Dragan Gasevic


Adaptive learning systems (ALSs) aim to provide an efficient, effective and customised learning experience for students by dynamically adapting learning content to suit their individual abilities or preferences. Despite consistent evidence of their effectiveness and success in improving student learning over the past three decades, the actual impact and adoption of ALSs in education remain restricted to mostly research projects. In this paper, we provide a brief overview of reflections and lessons learned from developing and piloting an ALS in a course on relational databases. While our focus has been on adaptive learning, many of the presented lessons are also applicable to the development and adoption of educational tools and technologies in general. Our aim is to provide insight for other instructors, educational researchers and developers that are interested in adopting ALSs or are involved in the implementation of educational tools and technologies.

Session 1D: Learning A

Lessons Learned From Introducing Preteens in Parent-Led Homeschooling to Computational Thinking

Authors: Carla Sepúlveda-Díaz; Elson Stuardo Rojas; Jocelyn Simmonds; Francisco J. Gutierrez; Nancy Hitschfeld; Cecilia Casanova; Cecilia Sotomayor


Parents that homeschool their children ignore certain topics when they lack mastery or interest in them. Homeschool groups try to address this issue, cooperatively educating their children. We were contacted by such a group that wanted to introduce their children to computational thinking (CT). These children, aged 7-11, have showed an interest in technology, and use online educational resources. None of the parents felt capable of tutoring the group about CT. They also worried about losing control about how their children interact with technology. We report an intervention over 9 months to introduce eleven young homeschoolers to CT in a suburban environment, describing the impact on parent and children attitudes towards technology and CT. We conclude with three lessons: 1) science-related activities should be used to introduce CT among homeschoolers, 2) "success" is establishing a meaningful relationship with a homeschool group, and 3) activities designed for school children need to be adapted to the homeschooling context.

Understanding Wikipedia as a Resource for Opportunistic Learning of Computing Concepts

Authors: Martin P. Robillard; Christoph Treude


Posts on on-line forums where programmers look for information often include links to Wikipedia when it can be assumed the reader will not be familiar with the linked terms. A Wikipedia article will thus often be the first exposure to a new computing concept for a novice programmer. We conducted an exploratory study with 18 novice programmers by asking them to read a Wikipedia article on a common computing concept that was new to them, while using the think-aloud protocol. We performed a qualitative analysis of the session transcripts to better understand the experience of the novice programmer learning a new computing concept using Wikipedia. We elicited five themes that capture this experience: Concept Confusion, Need for Examples, New Terminology, Trivia Clutter, and Unfamiliar Notation. We conclude that Wikipedia is not well suited as a resource for the opportunistic learning of new computing concepts, and we recommend adapting information sharing practices in on-line programmer communities to better account for the learning needs of the users.

TIPP&SEE: A Learning Strategy to Guide Students through Use ? Modify Scratch Activities

Authors: Jean Salac; Cathy Thomas; Chloe Butler; Ashley Sanchez; Diana Franklin


With the rise of Computational Thinking (CT) instruction at the elementary level, it is imperative that elementary computing instruction support a variety of learners. A popular pedagogical approach for this age group is Use-->Modify-->Create, which introduces a concept through a more scaffolded, guided instruction before culminating in a more open-ended project for student engagement. Yet, there is little research on student learning during the Use-->Modify step, nor strategies to promote learning in this step. This paper introduces TIPP&SEE, a metacognitive learning strategy that further scaffolds student learning during this step. Results from an experimental study show statistically-significant performance gains from students using the TIPP&SEE strategy on nearly all assessment questions of moderate and hard difficulty, suggesting its potential as an effective CS/CT learning strategy.

Session 1E: CS 2

MAADS: Mixed-Methods Approach for the Analysis of Debugging Sequences of Beginner Programmers

Authors: Chaima Jemmali; Erica Kleinman; Sara Bunian; Mia Victoria Almeda; Elizabeth Rowe; Magy Seif El-Nasr


be especially problematic for beginners. While there has been some work trying to understand the difficulties that beginners face with debugging, investigating common mistakes or specific error types they struggle with, there is little work that focuses on in-depth analysis of how novice programmers approach debugging, and how it changes over time. In this paper, we present MAADS (Mixed-Methods Approach for the Analysis of Debugging Sequences), a scalable and generalizable approach that combines quantitative and qualitative methods by using a state/action representation and visualization to gain knowledge about the debugging process through a step by step analysis. To demonstrate the utility of MAADS, we analyzed the debugging processes of middle school students who developed code within May's Journey, a game designed to teach basic programming principles. The approach showed great utility in identifying differences in students' debugging techniques and learning paths.

Analyzing the Effects of Active Learning Classrooms in CS2

Authors: Ayesha Syeda; Rutwa Engineer; Bogdan Simion


Active learning environments have only recently started to be analyzed in the CS discipline, in terms of their effect on student performance. Recent studies in CS1 found contradictory results, in part due to different control on the learning pedagogy used, and issued a call for further investigation. This study evaluates the effects of the learning space on student performance in CS2, as measured by their grades. We use a quasi-experimental setup with 529 participants across five lecture sections over one academic term. All sections employ the same active learning method (inverted classroom), identical lecture materials, and the same number of TAs for in-class support, but differ in terms of classroom type (active learning classroom vs traditional lecture hall), instructor, and lecture time of day. Similarly to a recent study in CS1, we find no significant impact of the learning space in CS2. We also inspect factors not analyzed in previous studies, such as student prior preparation (as measured by prerequisite CS1 grades), course drop rates, and exam failure rates, and find that the CS2 sections are statistically similar.

This work also examines student survey responses, to assess student perception differences on properties of the learning space which may impact their learning experience, such as the use of technology, ability to hear the instructor, ability to get help during lectures, and conduciveness of desk types to group work.

Does Story Programming Prepare for Coding?

Authors: Jennifer Parham-Mocello; Martin Erwig


In this research study, we investigate the impact of using the Story Programming approach to teach CS concepts on student performance in a subsequent C++ class. In particular, we compare how students receiving little or no coding to learn and apply these concepts perform in comparison to students who learn these concepts only in the context of coding. While past research has shown that exposure to programming is not a predictor of success in such courses, these studies are based on a 15-week versus 10-week course and do not control for the CS concepts and programming to which the students have been exposed. Consequently, we hypothesize that students from the Story Programming approach will perform worse in the following C++ class. Surprisingly, we find that this is not true: Students from the Story Programming approach with little to no coding do not significantly differ from their peers receiving a traditional code-focused approach.

Session 1F: Accessibility

A Systematic Analysis of Accessibility in Computing Education Research

Authors: Catherine M. Baker; Yasmine El-Glaly; Kristen Shinohara


Recent interest in accessibility emphasizes including it in computer science curriculum as key to producing effective computing professionals. Despite a general consensus that teaching accessibility in computing curriculum is good, there exist few tools and resources to support instructors in higher education. To better understand the relationship between accessibility in curriculum and research, we conducted a systematic literature review of papers in computing education. We analyzed the papers for the courses accessibility is covered in, the topics that are covered and pedagogies and assessment approaches that are used. Across this body of work, we found a number of key learning objectives commonly covered in computing education research, though it appeared the research did not evenly cover these objectives throughout curricula, nor did the research systematically investigate how learning objectives were integrated. Based on these results, we offer suggestions for future directions of accessibility education research and curriculum building.

Participation at What Cost? Teaching Accessibility Using Participatory Design: An Experience Report

Authors: Julian Brinkley


As institutions respond to market demand in their training of the next generation of technology designers, there is an increasing awareness of the need to add accessibility to computer science and informatics curricula. Advocates have suggested three strategies for including accessibility and discussions of disability in courses: changing a lecture, adding a lecture or adding a new course. In this paper we report on our experiences with the latter; incorporating accessibility within two new graduate and undergraduate inclusive design courses taught concurrently. We found that while the use of participatory design was decidedly effective in supporting student learning and ameliorating ableist attitudes, creating and managing teams comprised of students and visually impaired co-designers proved challenging. Despite these challenges, overall, students demonstrated steady growth in their grasp of inclusive design concepts as they tackled accessibility challenges through a series of mobility-related group projects. Efficiencies were also realized through the concurrent teaching of both courses though the pace of course deliverables proved challenging at times for undergraduates. We argue that a review of our experience may help others interested in teaching accessibility related courses, specifically in course design and execution.

Teaching Accessibility to Software Engineering Students

Authors: Yasmine El-Glaly; Catherine M. Baker


This paper describes the development process of a graduate course on accessibility that is offered as an elective for software engineering students. The paper reports on the three iterations of the course evolution as topics and pedagogy are updated from one iteration to the next. The main motivation of the course updates was to cover the educational needs of software engineering students, which are not the same as HCI or design students. Software engineering students learned better and became more engaged with the topic of accessibility when it was tied to programming and development- based activities. The final form of the course was evaluated using a survey, and the results showed that students found the course beneficial to their education and relevant to their career. We discuss the challenges of creating, teaching, and maintaining a course on accessibility, and we offer insights on what research is needed in this area to support accessibility educators.

Session 1G: Mentoring

The Impact of Undergraduate Mentorship on Student Satisfaction and Engagement, Teamwork Performance, and Team Dysfunction in a Software Engineering Group Project

Authors: Claudia Iacob; Shamal Faily


Mentorship schemes in software engineering education usually involve professional software engineers guiding and advising teams of undergraduate students working collaboratively to develop a software system. With or without mentorship, teams run the risk of experiencing team dysfunction: a situation where lack of engagement, internal conflicts, and/or poor team management lead to different assessment outcomes for individual team members and overall frustration and dissatisfaction within the team. The paper describes a mentorship scheme devised as part of a 33 week software engineering group project course, where the mentors were undergraduate students who had recently completed the course successfully and possessed at least a year's experience as professional software engineers. We measure and discuss the impact the scheme had on: (1) student satisfaction and engagement, (2) team performance, and (3) team dysfunction.

Formal Methods and Computing Identity-based Mentorship for Early Stage Researchers

Authors: Mark Santolucito; Ruzica Piskac


The field of formal methods relies on a large body of background knowledge that can dissuade researchers from engaging with younger students, such as undergraduates or high school students. However, we have found that formal methods can be an excellent entry point to computer science research - especially in the framing of Computing Identity-based Mentorship. We report on our experience in using a cascading mentorship model to involve early stage researchers in formal methods, covering our process with these students from recruitment to publication. We present case studies (N=12) of our cascading mentorship and how we were able to integrate formal methods research with the students' own interests. We outline some key strategies that have led to success and reflect on strategies that have been, in our experience, inefficient.

Undergraduate-Led Survey Class to Improve CS Education for New Students

Authors: Nathan Zhang; Jacky Liang; Amanda Tomlinson; Frank Boensch; Anant Sahai


Many first-year undergraduate students do not have sufficient breadth of technical knowledge about subjects in Electrical Engineering (EE) and Computer Science (CS) to make informed choices toward their education. By the time students are exposed to subjects they may be interested in, the cost of switching areas of focus may be too high. With undergraduate enrollment in CS more than doubling in the past decade, many institutions lack adequate staff and infrastructure to address students' needs. To help newly enrolled students make better decisions with limited departmental resources, we present a first-semester, low-overhead survey course that covers a wide variety of topics. The class has been offered for five consecutive semesters by upper-class undergraduate volunteers with minimal faculty involvement. We report the format, content, and student feedback for the course. Our results suggest that such a class can provide new students with valuable guidance and better prepare them for an education in CS and EE.

Session 1H: Problem Solving

Investigating Novices' In Situ Reflections on Their Programming Process

Authors: Dastyni Loksa; Benjamin Xie; Harrison Kwik; Amy J. Ko


Prior work on novice programmers' self-regulation have shown it to be inconsistent and shallow, but trainable through direct instruction. However, prior work has primarily studied self-regulation retrospectively, which relies on students to remember how they regulated their process, or in laboratory settings, limiting the ecological validity of findings. To address these limitations, we investigated 31 novice programmers' self-regulation in situ over 10 weeks. We had them to keep journals about their work and later had them to reflect on their journaling. Through a series of qualitative analyses of journals and survey responses, we found that all participants monitored their process and evaluated their work, that few interpreted the problems they were solving or adapted prior solutions. We also found that some students self-regulated their programming in many ways, while others in almost none. Students reported many difficulties integrating reflection into their work; some were completely unaware of their process, some struggled to integrate reflection into their process, and others found reflection conflicted with their work. These results suggest that self-regulation during programming is highly variable in practice, and that teaching self-regulation skills to improve programming outcomes may require differentiated instruction based on students self-awareness and existing programming practices.

Can Majoring in Computer Science Improve General Problem-solving Skills?

Authors: Shima Salehi; Karen D. Wang; Ruqayya Toorawa; Carl Wieman


Teaching students to become skillful problem solvers is a goal of university education, but it has been difficult to measure such skill or demonstrate the benefits of particular educational experiences. This paper presents a study of college students solving a problem unrelated to their academic majors. The analysis suggests that the educational experiences of Computer Science (CS) students may better train them in problem-solving than the experiences of other majors. In this study, students from a variety of undergraduate majors and grade levels were given a 15-minute problem-solving task embedded in an interactive science simulation. The complex task calls upon many problem-solving practices needed by scientists and engineers in their professions. Although this task has little resemblance to the problems encountered in a computer science course, CS students performed significantly better than students in any other major. In addition, only for CS students was there an indication of improvement in problem-solving from lower to upper grade levels. We propose that general problem-solving and computational thinking share some common practices, such as problem decomposition and comprehensive data collection. Furthermore, we present preliminary evidence that training in computational thinking is transferable to problem-solving tasks across domains and discuss how the unique features of CS programming assignments could be generalized to other science and engineering courses to foster students' general problem-solving skills.

Investigating Patterns of Study Persistence on Self-Assessment Platform of Programming Problem-Solving

Authors: Cheng-Yu Chung; I-Han Hsiao


A student's short-term study behavior may not necessary infer his/her long-term behavior. It is very common to see a student changes study strategy throughout a semester and adapts to learning condition. For example, a student may work very hard before the first exam but gradually reducing the effort due to several possible reasons, e.g., being overwhelmed by various course work or discouraged by increasing complexity in the subject. Consistency or differences of one student's behavior is more likely to be discovered by multiple granularity of learning analytics. In this study, we investigate students' study persistence on a self-assessment platform and explore how such a behavioral pattern is related to the performance in exams. A probabilistic mixture model trained by response streams of log data is applied to cluster students' behavior into persistence patterns, which are further categorized into "micro" (short-term) and "macro" (long-term) patterns according to the span of time being modeled. We found four types of micro persistence patterns and several macro patterns in the analysis and analyzed their relations with exam performances. The result suggests that the consistency of persistence patterns can be an important factor driving student's overall performance in the semester, and students achieving higher exam scores show relatively persistent behavior compared to students receiving lower scores.

Session 2A: Data Science A

A Data Science Major: Building Skills and Confidence

Authors: Stephanie Rosenthal; Tingting Chung


Data science is a growing field at the intersection of mathematics, computer science, and domain expertise. Like many universities that are building data science degree programs for undergraduates, our small, liberal-arts university saw increasing opportunities in the region and decided to build a data science degree from the ground up, without a pre-existing computer science (CS) department to leverage for courses or culture. We designed and implemented an academically-demanding curriculum that combined mathematics, information systems, and new data science courses, and that also encouraged and supported student success. Each introductory course included active learning design to engage students. To increase retention, all major courses included assignments designed to build skills but also student confidence in their ability to learn challenging technical topics. Outside of the classroom, we created opportunities for professional advancement and developed a technical culture at the university. We will share our approach, course highlights, and lessons learned from building such a curriculum at an institution without a CS department.

Creating a Balanced Data Science Program

Authors: Joel Adams


3rd Best Paper, Curricula Initiatives

As we consider the next fifty years of computing education, a phenomenon that shows no signs of abating is the data deluge, in which commercial companies, the natural sciences, the social sciences, professional sports teams, government agencies, and other institutions are generating ever-increasing quantities of data. To address the challenges posed by the data deluge, the discipline of data science has arisen, and an increasing number of universities are offering undergraduate data science programs. Many of these programs have their origins in a computer science or a statistics department, leading to a data science curriculum that is more heavily weighted toward computing or statistics. By contrast, the data science program described in this paper is a joint endeavor between computer science and statistics that seeks to provide balanced training in both areas. Its broad goals are to produce students who (a) are well-trained in both computer science and statistics, (b) are equipped with specialized data-related skills that are not normally taught in either of those disciplines, and (c) can apply their skills to a domain area. This paper reports on the author's experiences leading the effort to create this program, which has seen good growth, received positive feedback from students, and is successfully preparing students for internships. We offer this report in the hope that it may serve as a model for other institutions considering the addition of an undergraduate data science program.

Insights for Curriculum Development: Identifying Emerging Data Science Topics through Analysis of Q&A Communities

Authors: Habib Karbasian; Aditya Johri


Updating curricula in new computer science domains is a critical challenge faced by many instructors and programs. In this paper we present an approach for identifying emerging topics and issues in Data Science by using Question and Answer (Q&A) sites as a resource. Q&A sites provide a useful online platform for discussion of topics and through the sharing of information they become a valuable corpus of knowledge. We applied latent Dirichlet allocation (LDA), a statistical topic modeling technique, to analyze data science related threads from from two popular Q&A communities "Stack Exchange and Reddit". We uncovered both important topics as well as useful examples that can be incorporated into teaching. In addition to technical topics, our analysis also identified topics related to professional development. We believe that approaches such as these are critical in order to update curriculum and bridge the workplace-school divide in teaching of newer topics such as data science. Given the pace of technical development and frequent changes in the field, this is an inventive and effective method to keep teaching up to date. We also discuss the limitations of this approach whereby topics of importance such as data ethics are largely missing from online discussions.

Session 2B: Diversity

The Intersectional Experiences of Black Women in Computing

Authors: Yolanda Rankin; Jakita O. Thomas


Efforts to broaden participation in computing have led to gender-focused interventions intended to increase the number of women in the field of computing. However, such efforts have failed to significantly increase the percentage of Black women in computing. For example, only 1% of the 28,884 bachelor's degrees in computing were awarded to Black women in 2018. Moreover, too few empirical studies have intentionally explored the lived experiences of Black women, an often overlooked and understudied population in the computing ecosystem. In this paper, we introduce intersectionality - the complex overlap of socially constructed identities such as race, gender, class, sexuality, etc. - as a theoretical framework and springboard for exploring the lived experiences of Black women in computing. We interview 14 Black women in various stages of the computing ecosystem (undergraduate students, graduate students and early career professionals) to understand how intersectionality influences their ability to persist in computing. Preliminary findings from the analysis of the 14 interviews provides insights into how the interlocking systems of oppression (i.e., gendered racism) play out in computing education and negatively impact the recruitment and retention of Black women in the field of computing.

Lessons Learned from Providing Hundreds of Hours of Diversity Training

Authors: Luther Tychonievich; James P. Cohoon


The past thirty years have seen many advances in our understanding of issues that impede diversity, as well as interventions that can remove or mitigate those impediments. The advances are generally presented in psychology journals and are under-consumed by computing educators, leading to the need for diversity training tailored to said educators. We have hundreds of hours of experience providing diversity training to computing educators, and have learned many lessons about how training should and should not be delivered to engage participants in understanding diversity-impeding forces and implementing interventions to improve the attraction and retention of diverse students.

This paper is a collection of those lessons learned. We focus on concrete approaches to presenting material rather than on training organization or content, including a dozen specific practices that have proven particularly effective or ineffective in reaching diverse audiences. Examples include how how to avoid offending participants, how to help experts not lose their audience, how to pace and organize material for maximum impact, how we tie many topics into a single cognitive framework, and how we've learned to handle participant-raised objections. We also include commentary as to why we believe they did or did not work. We also describe possible future research to validate and expand on our observations and a few open questions for diversity training.

When Twice as Good Isn't Enough: The Case for Cultural Competence in Computing

Authors: Nicki Washington


The commonly documented diversity, equity, and inclusion (DEI) issues in the computing workforce are the direct result of corporate cultures that benefit specific groups and marginalize others. This culture usually begins in undergraduate computing departments, where the demographic representation mirrors that of industry. With no formal courses that focus on the non-technical issues affecting marginalized groups and how to address and eradicate them, students are indirectly taught that the current status quo in computing departments and industry is not only acceptable, but also unproblematic. This directly affects students from marginalized groups (as the reasons for attrition are similar in both higher education and industry), as well as faculty (as biased student evaluations directly affect hiring, promotion, and tenure decisions).

This position paper presents the need for cultural competence as a required focus for university computing departments nationwide. By improving these issues before students complete baccalaureate computing degrees, companies will have talent pools that better understand the importance and necessity of DEI and also work to ensure they help foster a more diverse, equitable, and inclusive environment. In addition, more students from marginalized groups will be retained in the major through degree completion.

Session 2C: CS0

Design Principles behind Beauty and Joy of Computing

Authors: Paul Goldenberg; June Mark; Brian Harvey; Al Cuoco; Mary Fries


Best Paper, Curricula Initiatives

This paper shares the design principles of one Advanced Placement Computer Science Principles (AP CSP) course, Beauty and Joy of Computing (BJC), both for schools considering curriculum, and for developers in this still-new field. BJC students not only learn about CS, but do some and analyze its social implications; we feel that the job of enticing students into the field isn't complete until students find programming, itself, something they enjoy and know they can do, and its key ideas accessible. Students must feel invited to use their own creativity and logic, and enjoy the power of their logic and the beauty and elegance of the code by which they express it. All kids need genuine challenge and sensible support so all can have the joy of making-seeing themselves as creators, not just consumers, and seeing that it is their own intellect, not just our instructions, that is the source of that making. Framework standards are woven into a consistent social and intellectual storyline to give the curriculum integrity.

Principles guide even our choice of programming language. Learners should focus on the logic and structure of their thinking, not on misplaced semicolons; attention to such syntactic detail is antithetical to broadening participation. We feature recursion and higher-order functions because they beautifully exemplify abstraction, a key idea in CS and the CSP framework.

BJC also places significant emphasis on the social implications of computing, balancing fundamental optimism about computing technology with a critical view of specific uses of technology.

CS 0: Culture and Coding

Authors: Albert Lionelle; Josette Grinslad; J. Ross Beveridge


In 2018, Colorado State University redesigned their CS-0 course to become a general education requirement for the university within Arts and Humanities, and a guaranteed transfer course across the state for a similar category in other universities. The first CS course in the State to be accepted as a GT-Pathway course. This redesign had to be carefully done due to a need to introduce liberal arts style topics such as CS History, Philosophy and Ethics and Inclusive Design issues, while maintaining current coding and student success standards that were already expected for the CS-0 at the university. We termed this combination as Culture and Coding. In order to add more without reducing retention, the course was redesigned around the Psychology of Learning and spacing of topics in a "Spiral" Manner. Each topic was briefly introduced, and throughout the semester, students would dive deeper into the topics. This allowed for a 50% reduction of time focused on teaching coding topics, with students performing equivalent on exams compared to previous models of the course that focused 100% of the time on coding topics. Furthermore, students taught by the spiral teaching method outperformed students taught using traditional methods in the follow-on course. Our evaluation suggests that the spiral model of teaching computer science may allow for greater retention of topics, allowing classes to either cover additional concepts or go more in depth on current topics.

Introducing Computing to a Cohort of Incarcerated Youth

Authors: Kirsten Mork; Theresa Migler; Zoë Wood


Computer Science and programming are changing the world, but not everyone has equal access to education about this field. In California, Juvenile hall students typically lack opportunities to learn computer programming. In this paper, we present an experience report about the course we created to address the needs of this population. Specifically, we created and taught an introductory computing course focused on engagement. This project-based game design curriculum was launched to a small cohort of the juvenile hall in spring 2019. This course focused on engaging students while introducing computing programming concepts such as variables, logic and function. Student surveys and reports from their teacher showed this class had a positive impact and was well received by students and staff. We hypothesize and show initial positive indication that creative, game-oriented curriculum had a positive impact on the demographic. We also present some of the challenges encountered when working within the juvenile hall system and our solutions and general recommendations for these types of classes.

Session 2D: BPC Initiatives

Algebra I Before High School as a Gatekeeper to Computer Science Participation

Authors: Ryan Torbey; Nicole D. Martin; Jayce R. Warner; Carol Fletcher


A complex web of factors can influence whether students participate in computer science (CS) during high school. In order to increase participation in CS for all students, we need to better understand who is currently participating and what factors might be hindering participation. This study utilized a large-scale, student-level dataset from the Texas Education Research Center to investigate factors that predict high school student participation in CS and advanced CS courses. Our dataset contained information on over 1.1 million Texas high school students from the 2017-2018 school year, allowing us visibility into CS course availability in schools, student course taking, and detailed demographic information. We used multilevel mixed-effects logistic regression models to explore predictive factors of student participation in CS and advanced CS courses, limiting our analysis to students whose schools offered CS. In both models, our results showed that students who took Algebra I before high school had more than double the odds of being enrolled in a CS course. This work supports and extends previous understanding of factors that are predictive of CS participation in high school, contributing to the existing literature by uncovering the importance of Algebra I before high school as a potential gatekeeper to participation in CS.

Promoting and Supporting Computer Science Among Middle School Girls of Color

Authors: Veronica Madrigal; Ryoko Yamaguchi; Adam Hall; Jamika Burge


BRIGHT-CS (Building Student Retention through Individuated Guided coHort Training in Computer Science) is a research and development project that 1) creates a computer science learning ecosystem for middle school Black girls and other girls of color and 2) researches the merits of the ecosystem in supporting persistence in CS to determine best practices for broadening participation to other marginalized student groups in computing. First, this paper describes the BRIGHT-CS program, from the structural, instructional, and curricular designs of the program to partnerships with local and community organizations that make up the ecosystem. Second, it presents the initial findings of research on the program and its impacts on student outcomes such as social-emotional attributes associated with persistence. The study employs a multi-method descriptive design. Data includes student surveys, interviews (from students, parents, instructors, teachers, and mentors), artifact reviews, and student observations. The study includes 46 students across four middle schools in two states. At the start of the program, 37% of the students reported being very interested in CS, and 72% reported being very confident in learning CS. This is much higher than a national benchmark of students. After four months of program implementation, the qualitative results show a more nuanced picture of the value of a learning ecosystem. First, the ecosystem offers implicit messaging about equity and success. Second, the ecosystem offers explicit messaging about personal challenges and improvement. Third, following the implicit and explicit messaging to students, students went from naïve confidence to authentic self-efficacy in CS.

Broadening Participation in Computing: The Role of Upward Transfer

Authors: Jennifer Blaney


Broadening participation in computing (BPC) has received increasing attention within the computing education community. Yet, much of the existing research on this topic focuses primarily on students who follow traditional pathways to four-year computing programs. This multi-institution study addresses this gap in the literature, providing insight into how upward transfer students (i.e., students who transfer from community colleges to four-year computing programs) fit into the larger BPC narrative. Specifically, I use survey data from computing majors across 15 universities to examine how the characteristics and experiences of upward transfer students differ from those of other computing majors. Findings reveal that while upward transfer students tend to be more diverse than other computing majors in terms of race/ethnicity, socioeconomic status, and other background characteristics, women are even more underrepresented among upward transfers compared to computing majors who began their degree at four-year institutions. These and other findings are discussed, focusing on implications for how institutions can support upward transfer students as part of their larger BPC efforts.

Session 2E: Math & CS

Didactic Methods of Integrating Programming in Mathematics in Primary School: Findings from a Swedish National Project

Authors: Gashawa Ahmed; Jalal Nouri; leChen Zhang; Eva Norén


The association between mathematics and programming in an educational context is not new. Today, programming has been introduced into curricula worldwide for younger children. In the Swedish case, primary school teachers are expected to integrate programming in mathematics education from autumn 2018. However, Swedish teachers' knowledge of programming and programming didactics is limited. Meanwhile, there is little research on K-9 programming education. This has led to the dilemma that the mathematics teachers have limited support in didactic knowledge and good examples. This study reports on a teacher professional development project in programming. More specifically, teachers used Lesson Study to plan, execute, and evaluate lessons that integrated programming into various school subjects in elementary school. This study analyzed the didactic strategies developed in 10 lesson studies, as well as mapped the opportunities and challenges of pupils' learning in the mathematics subject. The result was the identification of three didactic strategies, which were analog programming, robot programming and block programming, as well as 11 didactic methods applied within these strategies. The paper contributes with examples of the didactic methods that teachers have developed and evaluated using lesson study. The paper further provides insights on how teachers can take progression into account by applying the three didactic strategies. At last but not least, the study shows a great need for teachers to develop computational thinking abilities.

Comprehending Code: Understanding the Relationship between Reading and Math Proficiency, and 4th-Grade CS Learning Outcomes

Authors: Jean Salac; Cathy Thomas; Bryan “BT” Twarek; Bill Marsland; Diana Franklin


As many school districts nationwide continue to incorporate Computer Science (CS) and Computational Thinking (CT) instruction at the K-8 level, it is crucial that we understand the factors and skills, such as reading and math proficiency, that contribute to the success of younger learners in a computing curriculum and are typically developed at this age. Yet, little is known about the relationship between reading and math proficiency, and the learning of key CS concepts at the elementary level. This study focused on 4th-grade students (ages 9-10) who were taught events, sequence, and repetition through an adaptation of the Creative Computing Curriculum. While all students benefited from access to such a curriculum, there were statistically-significant differences in learning outcomes, especially between students whose reading and math proficiency are below grade-level, and students whose proficiency are at or above grade-level. This performance gap suggests the need for curricular improvement and learning strategies that are CS specific for students who struggle with reading and math.

Motivating Future Adventures in Computing by Unmasking Math Behind Movie Special Effects

Authors: Katie Chea; Christine Moore; William Bares


Children and adolescents consume increasing hours of screen time, the largest share of which goes to viewing movies according to Common Sense Research. Recently, movies with the highest box office revenues typically feature computer-generated imagery (CGI). Despite this substantial consumption of computer-generated imagery, the typical young viewer is unaware of how large a role mathematics plays in creating these movies. Studies have shown that there is a link between proficiency in math and predicted success in computing. In this paper, we describe our efforts to develop and evaluate CS Unplugged and code-based activities that attempt to explicitly connect math concepts with the creation of movie special effects magic. The learning activities have been designed and evaluated with middle school and first-year college students to demonstrate their potential to increase awareness and knowledge of how math is used in movie effects and to increase desire for future study of math. The paper describes the motivating context, pedagogical strategies, math learning outcomes, and CS Unplugged and code-based activities so that others may apply the methods. The paper concludes with a presentation and analysis of results from a five-day STEM summer camp for middle-school children and activities presented in first-year college classrooms and discussion of limitations and directions for future work.

Session 2F: Ethics & Academic Integrity

Teaching Academic Honesty in CS50

Authors: David J. Malan; Brian Yu; Douglas Lloyd


We aspire to teach academic honesty in CS50 at Harvard University not only by addressing academic dishonesty when it occurs but by addressing it before it does. By way of communication, course- and campus-wide awareness of policy, just-in-time prompts, interventional conversations, and problem sets have we tried to preempt submission of plagiarized work. But few interventions have had significant or lasting effects on the number of instances thereof. Most impactful has been the addition of one sentence to the course's syllabus, a "regret clause" that encourages students to come forward within 72 hours of some dishonest act on their part, before the course itself is even aware. While we might zero the work in question in such cases, we commit to not escalating the matter further to the university's honor council, where the outcome might instead be admonishment, probation, or even required withdrawal from the university itself. We instead advise students on how best to move forward and connect them as needed with support structures on campus for academics and mental health. Since 2014 have 89 students invoked the clause, between 1% and 3% of the course's student body each year.

What Do We Teach When We Teach Tech Ethics? A Syllabi Analysis

Authors: Casey Fiesler; Natalie Garrett; Nathan Beard


As issues of technology ethics become more pervasive in the media and public discussions, there is increasing interest in what role ethics should play in computing education. Not only are there more standalone ethics classes being offered at universities, but calls for greater integration of ethics across computer science curriculum mean that a growing number of CS instructors may be including ethics as part of their courses. To both describe current trends in computing ethics coursework and to provide guidance for further ethics inclusion in computing, we present an in-depth qualitative analysis of 115 syllabi from university technology ethics courses. Our analysis contributes a snapshot of the content and goals of tech ethics classes, and recommendations for how these might be integrated across a computing curriculum.

Teaching Computer Ethics: A Deeply Multidisciplinary Approach

Authors: Rob Reich; Mehran Sahami; Jeremy M. Weinstein; Hilary Cohen


We report on a curricular experiment at Stanford University focused on teaching computer ethics. After nearly a year of preparation, we launched a new course at the intersection of ethics, public policy, and technology that deeply marries the humanities, social sciences, and computer science.

While the teaching of computer ethics courses dates back decades, such courses are often taught by a (single) CS faculty member without significant training in ethics, do not include a policy component, and are meant for CS students. By contrast, we take a deeply multidisciplinary approach, where three faculty instructors, from philosophy, political science, and CS, each bring their respective lens to four related course modules: algorithmic decision-making, data privacy and civil liberties, AI and autonomous systems, and the power of platform companies. Panels of guest speakers drawn from academia, industry, civil society, and government provide a practitioner's view of the topics addressed. Additionally, custom case studies were developed under the direction of the course staff. These materials (videos of the speaker panels and the case studies) are freely available for use by the broader community.

We report on the details of the course structure, including how multiple disciplines are integrated throughout the course, including lectures, discussions, and assignments. We discuss aspects of the course that worked well as well as challenges in making the course broadly accessible (beyond just CS majors). Importantly, we also include a discussion of students' response to the course, showing that a deeply multidisciplinary approach resonates strongly with them.

Session 2G: Student Motivation

Influence of Course Design on Student Engagement and Motivation in an Online Course

Authors: Kalpathi Subramanian; Kiran Budhrani


We present a course design model for applying project-based learning to an online undergraduate object oriented systems course. In our model, projects and reflection are central to the curriculum. Our model challenges students through modularized, repetitive project cycles beginning with analysis and design (i.e. using pseudo- code, flowcharts, diagrams) then coding, debugging, testing, and finally, reflection. We analyzed student reflection responses from two semesters to extract major themes and sub-themes, then mapped these to the MUSIC model (eMpowerment, Usefulness, Success, Interest, Caring) to understand our model's influence on student engagement and motivation. We found that a rhythmic project cycle encourages self-regulation in online students to formulate project plans, track their progress, and evaluate their solutions. Online students feel empowered when course projects promote choice, flexibility, creativity, experimentation, and extensions to other applications. Online student success is dependent on the clarity of instructions, course scaffolding, level of challenge, instructor feedback, and opportunities to reflect on personal failure, success, and challenge. Online students are interested in projects that are familiar, real-world, and fun, but expect to be situated in team-based environments. Students appreciate instructors who are caring and accommodating to personal needs. We recommend six salient strategies for improving online course and project design: design a visible, rhythmic structure; set transparent expectations and instructions; encourage design before implementation; connect to real-world applications and tools; experience happy challenges; infuse sustained reflection.

A Project-Based Learning Approach for Enhancing Learning Skills and Motivation in Software Engineering

Authors: Beatriz Pérez; Ãngel L. Rubio


Software engineers must be able to manage complex projects, so that skills such as teamwork, leadership or initiative are critical to their successful development. Because of this, it is fundamental that the learning of software engineering as an academic discipline provides solid links between theory and practice. Educational frameworks such as those derived from the European Higher Education Area state that student-centered approaches are a useful tool for achieving these objectives. In this context, we present a project-based learning (PBL) experience report in a software engineering program of a Spanish university. The experience is based on the formation of small heterogeneous teams, which face the initial phases of a software methodology during the development of a project close to a real one. Through a strategy of role rotation and documentation transfer, all students perform different tasks and face different challenges throughout the project. Summative assessment is also adopted, considering not only teacher ratings but also students' peer assessment. The results prove the positive effect of using PBL to improve the training of students in acquiring different skills as future software engineers.

The Prevalence of Anxiety and Depression Symptoms among Brazilian Computer Science Students

Authors: Lígia Maria Soares Passos; Christian Murphy; Rita Zhen Chen; Marcos Gonçalves de Santana; Giselle Soares Passos


The mental health of undergraduate students has become a focus of attention for health experts and for society. Several studies have considered students from the Health Sciences, predominantly medical students. Lately, discussions about the mental health of Computer Science (CS) students have become increasingly common.

This paper presents a study about the prevalence of anxiety and depression symptoms among CS students. This study was conducted at Universidade Federal Rural do Rio de Janeiro (UFRRJ), a Brazilian public university. The aim of this study was to verify the prevalence and associated factors of anxiety and depression symptoms among Brazilian CS students.

Students were randomly selected (n=131) from the CS program at UFRRJ, aged between 18 and 30 years. The participants completed a sociodemographic questionnaire and two inventories -- the Beck Anxiety Inventory (BAI) and the Beck Depression Inventory (BDI) -- that were used to measure levels of anxiety and depression symptoms. Statistical analysis was performed using the TIBCO Statistica\textsuperscript{\texttrademark} software (version 13.5).

The prevalence of anxiety and depression symptoms among the students were, respectively, 51.9% and 64.9%, which are higher than previous findings in the Brazilian general population and among medical students. Additionally, the BDI and BAI scores were higher among female students, and we observed statistically significant relationships between BAI, BDI, and sleep quality, among others.

This paper describes our study and findings and addresses ways in which CS students may be able to reduce symptoms of anxiety and depression.

Session 3A: Novice Programmer Behaviors

Exploring Novice Programmers' Homework Practices: Initial Observations of Information Seeking Behaviors

Authors: Silvia Muller; Monica Babe?-Vroman; Mary Emenike; Thu D. Nguyen


There are many factors that contribute to the success of students learning to code. For students in introductory programming classes, one source of complexity is the availability of a wide variety of information sources. In this paper, we report observations of students seeking information when working on programming homework assignments. Our data was collected from a think-aloud protocol embedded into semi-structured, individual interviews with students enrolled in a CS1 course. We analyze our data through the lens of information seeking behavior. We observed students using multiple sources of information, including referring back to course materials and searching for information online, and discussing how they sought help from friends, classmates, and family members. Herein, we discuss implications for teaching and future research based on our initial observations. For example, instructors could consider designing early homework assignments that would prompt students to seek information and follow up this assignment with an in-class discussion about homework strategies. Future research could investigate the mechanisms by which students progress from haphazard to more strategic information seeking behaviors.

Sometimes It's Just Sloppiness - Studying Students' Programming Errors and Misconceptions

Authors: Ella Albrecht; Jens Grabowski


Knowledge about students' programming errors is a valuable source to get insights into students deficiencies and misconceptions. In this paper, we use data from an introductory C programming course to identify which errors are often made by students. Previous studies often focused only on syntactic and semantic errors as they can be easily identified by compilers. Studies focusing on logic errors were often restricted to a limited set of concepts or performed for a small set of data. We manually inspect 12371 submission by 280 students and have no restrictions regarding the error types we are looking for. We classify our found errors into six categories: syntactic, conceptual, strategic, sloppiness, misinterpretation, and domain knowledge. Our results show that a big portion of errors made by students is simply caused by sloppiness. But putting sloppiness aside, students seem to have most problems with strategic knowledge, i.e., the problem solving ability. We compare our results to previous studies and provide some implications of our results for future teaching practice.

A Comparison of Two Pair Programming Configurations for Upper Elementary Students

Authors: Jennifer Tsan; Jessica Vandenberg; Zarifa Zakaria; Joseph B. Wiggins; Alexander R. Webber; Amanda Bradbury; Collin Lynch; Eric Wiebe; Kristy Boyer


2nd Best Paper, Experience Reports and Tools

As computer science education opportunities for elementary students (grades K-5) are expanding, there is growing interest in using pair programming with these students. However, previous research findings do not fully support its use with younger learners, and some researchers have begun to examine whether introducing a second computer with a shared coding workspace can provide important benefits. This experience report describes a series of classroom activities in the 4th and 5th grades (ages 9-11 years old) with two different pair programming configurations: one-computer pair programming, in which both students share a keyboard, mouse, and monitor; and two-computer pair programming, in which each student has a separate computer but coding workspaces are synchronized over the web. In both cases the students sat next to each other and engaged in face-to-face conversation. We found that students largely preferred two-computer pair programming over one-computer pair programming. We conducted focus groups and transcribed collaborative dialogues to gain more insight into this preference. We learned that students felt more independence in two-computer pair programming, although they struggled with coordinating their edits with their partner. In one-computer pair programming, students reported not wanting to wait for their turn to drive, but feeling as though they communicated more with their partner. Both configurations can be productive for students, but the tradeoffs described in this experience report are important for CS educators and researchers to consider when determining which collaborative configuration to use in each K-5 classroom context.

Session 3B: CS Instruction

CS Student Laptop and Computer Lab Usage as a Factor of Success in Computing Education

Authors: Béatrice Moissinac; Jennifer Parham-Mocello; Robin Pappas


Computing Education Research (CER) implicitly assumes that CS undergraduate students have no barrier to access a learning platform or software package. The assumption that "everyone has access to and uses a device" endangers the validity of CER studies by overlooking a critical element in what students access and how students use computing resources.

First, in this work, we explicitly investigate undergraduate student usage (how often) of computing resources (laptops and computer labs) on a university campus. Furthermore, we investigate whether CS student usage of laptops and computer labs are factors of success in computing education to close a crucial feedback loop for CER and CS educators.

Second, previous studies studying student's technology equipped used qualitative surveys, and lacked a systematic and continuous view of the student population. In this work, we address this shortcoming by developing a method to use operational data sets from the wireless networks and computer labs on campus. Operational data sets provide a systematic and continuous coverage of all students, that is, a student's absence of usage becomes a data point instead of a missing point.

Results indicate that the use of equipment levels may be lower than national average. Nevertheless, there exists a positive correlation between higher frequency of laptop usage and success in computing education.

Factors Associated with High-Quality Computer Science Instruction: Data from a Nationally Representative Sample of High School Teachers

Authors: Eric Banilower; Laura Craven


This paper shares results from surveys administered in spring 2018 to a nationally representative sample of nearly 300 U.S. high school computer science teachers. It describes the nature of high school computer science instruction and the extent to which teacher background, classroom factors, and school context predict the type of instruction students experience. Data from the study were analyzed using path modeling-a form of regression analysis that estimates both direct and indirect effects (i.e., through intermediary variables)-to examine relationships between teacher, classroom, and school factors, and the extent to which teachers (1) emphasize reform-oriented instructional objectives (e.g., learning about real-life applications of computer science) and (2) engage students in computer science practices (e.g., recognizing and defining computational problems). Sample findings include that students are most commonly engaged in activities related to testing and refining computational artifacts, but are less often engaged in aspects of computer science related to end users (e.g., create a computational artifact to be used by someone else). The path analysis highlights several factors that are related to greater engagement of students in the computer science practices, including teacher participation in professional development and the use of coherent instructional materials.

A Strategy to Enhance Computer Science Teaching Material Using Topic Modelling: Towards Overcoming The Gap Between College And Workplace Skills

Authors: Ingrid Marçal; Rogério Eduardo Garcia; Danilo Eler; Ronaldo Celso Messias Correia


Computer Science teaching materials are biased towards concepts and theoretical aspects. One may consider it difficult to relate concepts to concrete problems. Consequently, it increases the chances of a student not recognizing the relevance of the subject, becoming unmotivated and unprepared to solve practical problems or coping with workplace needs after college. This paper shows the use of social media data as an alternative to minimize the skill gap between what the student learns in college and the skills required in the workplace. The proposed strategy consists of extracting topics from Stack Overflow questions to identify concepts generally unknown or misunderstood and concepts that their practical application represents a challenge. The concepts covered in Stack Overflow questions provide strong cues about how professors and instructors can improve teaching material with useful content for their students, as well as increase their motivation -- since the teaching material becomes clearly related to practical issues in Computer Science. We show, by an example, how to use the proposed strategy to improve teaching material from the generated topics. Also, we demonstrate that the application of topic modeling in Stack Overflow content is promising as a professor support tool to enhance teaching material relevance in Computer Science courses and diminish the college-workplace skill gap.

Session 3C: CS1/CS2

The Practical Details of Building a CS Concept Inventory

Authors: Cynthia Taylor; Michael Clancy; Kevin Webb; Daniel Zingaro; Cynthia Lee; Leo Porter


Concept inventories (CIs) allow researchers and practitioners to measure student conceptual learning within a course or topic area. While they have enabled meaningful pedagogical change in other disciplines, there are relatively few CIs in computer science. In this paper, we report on our experiences as recent developers of a CI for basic data structures. We discuss each step along the route to a CI and offer tips based on what we have learned. We encourage others to create CIs, and we hope that this paper will serve as a practical guide through the process.

Reviewing CS1 Materials through a Collaborative Software Engineering Exercise: An Experience Report

Authors: Jessica Schmidt


3rd Best Paper, Experience Reports and Tools

Many courses incorporate exercises at the end of the semester to help students review course topics. In computer science courses, review activities are often small coding problems, interactive games, or review lectures. In this paper, we describe our experience with replacing student-led review presentations with a collaborative software engineering exercise to serve as a review of CS1 programming concepts. The objective of this review exercise is to increase engagement and self-reflection on course topics by engineering a real-world software application to help students review programming concepts and prepare for the subsequent CS1.5 course. During the final two weeks of the course, students spend three class sessions working to complete the comprehensive exercise. During the final class session, each team presents its software. In this paper, we describe the comprehensive exercise, lessons learned from the past eight semesters (over 1,400 students), and suggestions for adopting the comprehensive exercise in other CS1 courses.

Exploring the Link Between Prerequisites and Performance in Advanced Data Structures

Authors: Sophia Krause-Levy; Sander Valstar; Leo Porter; William G. Griswold


Recent work has identified a mismatch between instructor expectations of students' mastery of prerequisite course content and their actual ability. This invites the question of why this mismatch exists. We first examined grades in prerequisite courses and found they meaningfully correlated with performance on an assessment testing their knowledge of prerequisite material. In addition, we found neither taking alternatives to the primary identified prerequisites nor the delay between taking prerequisite courses and the follow-on course meaningfully impacts performance. Second, we confirmed that prerequisite course grades are significantly correlated with the grade in the follow-on course-confirming that the grades in the previous courses convey some information about student understanding of those topics. Perhaps surprisingly, we found that grades in courses outside computing were similarly correlated as those courses inside computing, suggesting that underlying factors such as general study skills may be as important as the domain-specific knowledge itself.

Session 3D: CS + X

Abstraction Through Multiple Representations in an Integrated Computational Thinking Environment

Authors: Aakash Gautam; Whitney Bortz; Deborah Tatar


We present reflections based on qualitative analysis of data from the CHEM+C Project which promotes computational thinking (CT) in classrooms through integration with science classes. The curriculum utilizes multiple representations, requiring students to work with physical phenomena, chemical equations, digital simulations, and modifiable code-based representations. Much CT focus on abstraction naturally emphasizes (1) extraction of a set of features from an object or process, and (2) finding commonality between objects and processes. But Rosen encourages us to think about abstraction as also including the production of new concepts or actions. Integrating CT into science offers the possibility of enhancing this aspect of abstraction. Changing the representational affordances available to the students allows them to take their CT thinking beyond learning-to-abstract towards learning-through-abstraction. This perspective moves computation from an internally focused exercise into the expression of valued ideas in a computational medium.

Interdisciplinary Computing: Applied Computing for Behavioral and Social Sciences

Authors: Valerie Carr; Morris Jones; Belle Wei


As the digital economy grows, so does the demand for technology-capable workers who have both computing skills and domain expertise. Growing such a workforce is critical to ensuring the nation's competitiveness, according to a recent National Science Board publication. To address this need, faculty from the Colleges of Engineering and Social Sciences at San José State University worked together to create the Applied Computing for Behavioral and Social Sciences minor degree. The minor targets students in majors such as Psychology and Economics, which have a more diverse student population than that of Computer Science or Engineering. The minor, designed with industry input, includes a four-course sequence that focuses on Python and R and includes topics such as data structures, algorithms, data cleaning and management, and data analysis. Our cohort-based program was built specifically for social science students using social science content, helping to foster a sense of community and belongingness among students. The first full cohort of 26 students graduated in Spring 2019, 48% of whom were female and 23% of whom were underrepresented minorities. Our approach of embedding computing education into the social sciences demonstrates a promising model of broadening participation in computing and meeting the nation's increasing demand for interdisciplinary computing workers in the digital age.

Interdisciplinary Minor in Bioinformatics: First Results and Outlook

Authors: Natalia Khuri; Wendy Lee; K. Virginia Lehmkuhl-Dakhwe; Miri VanHoven; Sami Khuri


There is a clear demand and need for more and better training in Computer Science for students of different backgrounds and genders. Traditional academic pathways for Computer Science majors and minors may not always be well suited for attracting underrepresented students. With the aim of recruiting and training students from life sciences, over 60 of whom are female, we established an alternative, programming-intensive pathway that is integrated within an interdisciplinary Bioinformatics minor. In this paper, we summarize the development and implementation of this program in a large public university, which can serve as a framework for new interdisciplinary programs in predominantly undergraduate institutions.

Session 3E: Code Analysis

A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes

Authors: John Edwards; Juho Leinonen; Arto Hellas


We study programming process data from two introductory programming courses. Between the course contexts, the programming languages differ, the teaching approaches differ, and the spoken languages differ. In both courses, students' keystroke data -- timestamps and the pressed keys -- are recorded as students work on programming assignments. We study how the keystroke data differs between the contexts, and whether research on predicting course outcomes using keystroke latencies generalizes to other contexts. Our results show that there are differences between the contexts in terms of frequently used keys, which can be partially explained by the differences between the spoken languages and the programming languages. Further, our results suggest that programming process data that can be collected non-intrusive in-situ can be used for predicting course outcomes in multiple contexts. The predictive power, however, varies between contexts possibly because the frequently used keys differ between programming languages and spoken languages. Thus, context-specific fine-tuning of predictive models may be needed.

Calibration and Analysis of Source Code Similarity Measures for Verilog Hardware Description Language Projects

Authors: James F. Power; John Waldron


In this paper we report on our experiences during a first-year course on digital logic design using the Verilog hardware description language. As part of the course the students were given a series of take-home assignments, which were then marked using an automated assessment system developed by the authors. During the course the instructor was made aware that a set of solutions had been circulated to the students, and was asked to assess the impact that this had on the assessment regime. In order to answer this question, we examined and implemented a number of approaches to calculating similarity between Verilog programs, and we present the results of that study in this paper. An important feature of this work was ensuring that the measurements used were well-understood, properly calibrated and defensible. We report on the results of this study, applied to a class of 115 students who completed up to 11 projects each.

Unlimited Trace Tutor: Learning Code Tracing With Automatically Generated Programs

Authors: Ruixiang Qi; Davide Fossati


Previous research showed that creating specific types of tracing tables helps students learn code tracing, a fundamental skill in computer programming. This paper introduces Unlimited Trace Tutor, the first version of a code tracing tutoring system that can automatically generate tracing problems and create such tracing tables. We conducted a pilot experiment with volunteer students from an introductory level Computer Science course. We found that our software effectively helps student learn tracing "for" loops, "while" loops, and "if" statements. In this paper we describe the system's architecture, our algorithms for generating code and tracing tables, and the promising results of our pilot experiment.

Session 3F: Computing for Social Good

Internationalizing the Student Experience Through Computing for Social Good

Authors: Komal Aheer; Ken Bauer; Cam Macdonell


Information technology has connected our world and its citizens in incredible ways. Despite this connectedness, students are often isolated within the "online bubbles" of their own university, city, or country. Technology provides a great opportunity to connect them to a broader global experience. We have developed and piloted a cross-institution activity as part of an Internationalization at Home (IaH) initiative to expose first year computer science students to the concept of computing for social good in an international context. We explore how differences in culture can influence students' perceptions and approaches to computing for social good. Specifically, we had students from a Mexican and a Canadian university explore how computing for social good could be used to solve issues they faced in their communities. Students participated in surveys to propose and then rank applications for social good. The students also participated in a videoconference discussion with the students from the other school to discuss their choices. Thematic analysis revealed that the students had much more in common with each other than they had differences. Both groups not only focused on similar areas of interest, but they also tended to focus on solving issues with a local scope rather than national or global scope. Despite their cultural differences, the majority students felt they were more similar to their peers of the other culture than they were different.

LibreFoodPantry: Developing a Multi-Institutional, Faculty-Led, Humanitarian Free and Open Source Software Community

Authors: Karl R Wurst; Chris Radkowski; Stoney Jackson; Heidi J. C. Ellis; Darci Burdge; Lori Postner


Engaging students in humanitarian free and open source software (HFOSS) projects allows them to gain real-world software development skills while helping society. Participating in an existing HFOSS project, although ripe with learning opportunities, presents a number of hurdles for faculty and students. An alternative to joining an existing HFOSS project community is to participate in a faculty-led HFOSS project. These projects provide the instructor with more control over the learning environment, but often lack an active community outside of the classroom. This paper describes a multi-institutional effort to engage a community of developers in creating humanitarian open source projects to support their on-campus food pantries. Food insecurity on campus has become a national concern and many institutions have, or are starting, food pantries to support the student, staff, and faculty community. Starting a faculty-led HFOSS project involves making decisions not only about the features of the project but also about community norms, tool choices, project development workflow, and inter-institution cooperation. This paper provides an overview of the creation of LibreFoodPantry, a community who is developing a suite of projects that support on-campus food pantries. It describes instances of using LibreFoodPantry's projects in various classroom settings, the lessons learned from these experiences, and the resulting discussions and decisions made by the LibreFoodPantry Coordinating Committee. This process has led to a community dedicated to easing the on-ramp for faculty who want to help their students contribute to an HFOSS project.

Merging Sustainability and Technology in the Classroom: An Experience Report

Authors: Giovanni Vincenti; Wolf T. Pecher


Formal education often relies on the students' ability to make connections among different subjects, as they navigate through a curriculum packed with different topics. In order to facilitate the preparation to the workforce, we created a course in "Sustainability and Technology" that was cross-listed in the degrees of Applied Information Technology and Environmental Sustainability. This manuscript documents our efforts in creating an indoor farm by letting students from different majors collaborate on this shared project. The results of our survey support the benefits of letting students collaborate as they approach the end of their undergraduate studies.

Session 3G: Collaboration

Strategies for Implementing In-Class, Active, Programming Assessments: A Multi-Level Model

Authors: Shanon Marie Reckinger; Bryce Hughes


As the enrollment in computer science courses increases, faculty face the challenge of maintaining a standard of academic integrity in out of class programming assignments. One strategy to combat academic misconduct, is to incorporate more frequent in-class assessment. Traditional in-class assessment is often perceived as passive and taking away from valuable class time. Also, more in-class assessment could mean increasing the amount of test anxiety. Therefore, this paper presents results from a study that investigated the effectiveness of active, in-class programming assessment. The study was conducted in an introductory programming course with 93 undergraduate students at a mid-size, public university in the western United States. Three different assessment types were tested: collaborative, cooperative, and individual type assessments. A collaborative assessment is equivalent to pair programming. Cooperative assessments are partner quizzes where students must write their own code and are assessed individually, but may collaborative as much or as little as they like with their partner. Student performance was measured under multiple conditions including type of assessment and partner's active learning preference and factors like GPA, time in the semester, and section were controlled for. The results show that students perform better on both collaborative and cooperative type assessment than they do on individual type assessments. Also, the results show that students perform similarly on collaborative and cooperative assessments, which provides a nice alternative to pair programming for educators. Lastly, results show that most students prefer collaborative assessments over cooperative or individual. The majority of students prefer partner assessment to individual assessment.

A Comparison of Peer Instruction and Collaborative Problem Solving in a Computer Architecture Course

Authors: Geoffrey Herman; Sushmita Azad


While substantial research has demonstrated that active learning pedagogies are better for learning than passive lectures, we need to understand the trade-offs between different active learning pedagogies. Computer Architecture at Midwestern University has historically been taught using active lectures, introducing content with a few clicker questions. In Fall 2018 (N = 363), short video lectures were made available to students as a supplemental resource. In Spring 2019, the instructor flipped the course, requiring students to watch the video lectures and complete an assignment before attending class. Two versions of the course were taught concurrently, using the same homework assignments, machine problems, and examinations but with different in-class pedagogies. Version SP19PI (N = 179) was taught using peer instruction. Version SP19CP (N = 73) was taught using collaborative problem solving, organizing students into teams of 3 to work on problems. Students completed surveys that measured their perceptions of time spent on the course, course difficulty, perceptions of stress, and sense of belonging. We compare students' performance on midterm exams and their non-cognitive outcomes to examine the relative effects of these different active learning pedagogies. We find that both flipped offerings (peer instruction and collaborative problem solving) benefited students beyond active lectures. Peer instruction (SP19PI) made learning more efficient. Collaborative instruction (SP19CP) provided greater social support for learning and eliminated gender grade disparities.

Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments

Authors: Marcel Borowski; Johannes Zagermann; Clemens N. Klokmose; Harald Reiterer; Roman Rädle


Programming assignments in computer science courses are often processed in pairs or groups of students. While working together, students face several shortcomings in today's software: The lack of real-time collaboration capabilities, the setup time of the development environment, and the use of different devices or operating systems can hamper students when working together on assignments. Text processing platforms like Google Docs solve these problems for the writing process of prose text, and computational notebooks like Google Colaboratory for data analysis tasks. However, none of these platforms allows users to implement interactive applications. We deployed a web-based literate programming system for three months during an introductory course on application development to explore how collaborative programming practices unfold and how the structure of computational notebooks affect the development. During the course, pairs of students solved weekly programming assignments. We analyzed data from weekly questionnaires, three focus groups with students and teaching assistants, and keystroke-level log data to facilitate the understanding of the subtleties of collaborative programming with computational notebooks. Findings reveal that there are distinct collaboration patterns; the preferred collaboration pattern varied between pairs and even varied within pairs over the course of three months. Recognizing these distinct collaboration patterns can help to design future computational notebooks for collaborative programming assignments.

Session 4A: Learning B

Qualitative Analyses of Movements Between Task-level and Code-level Thinking of Novice Programmers

Authors: Francisco Castro; Kathi Fisler


Cognitive theories of how programmers produce code suggest that novices' approaches are primarily driven by the retrieval of previously-learned plans. These plans can be high-level, focusing on task decomposition and composition, or low-level, focusing on code implementations. These theories, however, do not describe how novices move between high-level tasks and low-level code, especially when faced with novel problems. Understanding these transitions can help concretely tease out why and where novices struggle and how they use their knowledge of plans and design techniques when they get stuck.

We studied this by conducting think-alouds with CS1 students at two universities as they solved multi-task programming problems with unfamiliar components. Our analysis paid particular attention to a series of design techniques that the students had been explicitly taught in their respective courses. We identified patterns of high- and low-level thinking that relate to students' success on the problems, and propose a concrete framework of high- and low-level work that summarizes the transitions that we observed.

Challenges with Learning to Program and Problem Solve: An Analysis of Student Online Discussions

Authors: Paul Piwek; Simon Savage


Students who study problem solving and programming (in a language such as Python) at University level encounter a range of challenges, from low-level issues with code that won't compile to misconceptions about the threshold concepts and skills. The current study complements existing findings on errors, misconceptions, difficulties and challenges obtained from students after-the-fact through instruments such as questionnaires and interviews. In our study, we analysed the posts from students of a large cohort (~1500) of first-year University distance learning students to an online `Python help forum' - recording issues and discussions as the students encountered specific challenges. Posts were coded in terms of topics, and subsequently thematically grouped into Python-related, problem solving/generic programming related, and module specific. We discuss the set of topics and rank these in terms of the number of forum discussions in which they occur (as a proxy for their prevalence). The top challenges we identified concern student understanding and use of a mix of programming environments (in particular, Python IDLE for offline programming and CodeRunner for programming quizzes) and code fragment problems. Apart from these, Python-specific topics include, among others, collections, functions, error messages, iteration, outputting results, indentation, variables and imports. We believe that the results provide a good insight into the challenges that students encounter as they learn to program. In future work we intend to study the discussions in further detail in terms of theories of conceptual change.

A Study on Student Performance Evaluation using Discussion Board Networks

Authors: Urvashi Desai; Vijayalakshmi Ramasamy; James D. Kiper


Node-based social network analysis (SNA) techniques can be used to investigate the significance of actors that play central roles in social networks where the nodes represent people, teams or stakeholders and the links represent the communication, information exchange or collaboration between these nodes (actors). This research investigates how collaborative problem-solving can help in students' learning process. We analyzed the discussion board data collected from online student discussions on Canvas, a Learning Management System (LMS), in a CS1 course of a medium-sized US University. The discussion topics were classified as classroom experiences/learning, question/answers, opinions, and comments and were used to represent the patterns of interactions in the student discussion networks. Node-based network measures were then applied to unravel the students' interaction patterns to gain insights on students' progress. The textual analysis helps find the most challenging/debated topics in a particular course, analyze the leadership and team-based qualities of a group of students, and analyze patterns and trends in female student participation. The experimental analysis reveals that participation in online discussion forums has a positive impact on the students' grades; the study of interaction patterns exhibit similar insights. In conclusion, this research study validates that the analysis of structured discussions can provide useful insights into changes in student collaboration patterns over time and students' sense of belongingness for pedagogical benefits.

Session 4B: Elementary School Experiences

Future-Proofing Kiwi Kids Through the Use of Digital Technology

Authors: Veronica Liesaputra; Guillermo Ramirez-Prado; Bashar Barmada; Lei Song


The common approach for introducing middle school and high school students to computational thinking and engineering is through programming or robotics activities that were designed without much social context. Therefore, one of the challenges such outreach programs pose is further stereotyping of students who are interested in computer science, by presenting them as people who ought to be hyper-intelligent and very much focused on computers with no social skills.

Realizing the above, and wanting to attract a diverse audience to study computer science, we developed a one-day workshop which showcases the interdisciplinary nature of computer science and the various ways computing can help students discover and solve societal problems. We chose multiculturalism, cyber security and air pollution as the context of our computational thinking activities.

This paper describes our experiences in developing and teaching the workshops that runs three to five times a year, hosted either in our institute or at local schools. Regardless of students' previous STEM experiences and socioeconomic statuses, both students and teachers reported that the knowledge gained enabled them to either create a computer program or a robot and also led to a better understanding of how computers can be used in daily life. In other words, the workshop successfully opened the minds and perspectives of the middle school and high school students on computer science. Additionally, over time, we also observed an increased number of students and schools participating in such workshops.

Patterns in Elementary-Age Student Responses to Personalized & Generic Code Comprehension Questions

Authors: Jean Salac; Qi Jin; Zipporah Klain; Saranya Turimella; Max White; Diana Franklin


The CS community has struggled to assess student learning at the K-8 level, with techniques ranging from one-on-one interviews to written assessments. While scalable, automated techniques exist for analyzing student code, a scalable method for assessing student comprehension of their own code has remained elusive. This study is a first step in bridging the gap between the knowledge gained from interviews and the time efficiency and scalability of written assessments and automated analysis. The goal of this study is to understand how student answers on various types of questions differ depending on whether they are being asked about their own code or generic code. We find that while there were no statistically-significant differences in overall scores, questions about generic and personalized code of comparable complexity are far from equivalent. Our qualitative analyses revealed interesting patterns in student responses, inviting further research into this assessment technique. In particular, students answered differently from students with generic code when presented with individual blocks from their code taken out of context and placed into different code snippets, and students answered in a way that demonstrates a functional, instead of structural, understanding on Explain in Plain English (EiPE) questions.

Comparing TUIs and GUIs for Primary School Programming

Authors: Abrar Almjally; Kate Howland; Judith Good


There is considerable interest in using tangible user interfaces (TUIs) to support teaching children programming, but evidence for the benefits is mixed, and their deployment in school environments presents more challenges than graphical user interfaces (GUIs). This study investigates the effect of GUIs and TUIs on learning outcomes, attitudes toward computing, and reported enjoyment in a computer-programming activity with primary-school students aged 6-7 in Saudi Arabia. Forty-two students engaged in a 45-minute learning activity using either a TUI or GUI programming environment. The study used a between-groups design, and quantitative data were collected, including pre-test and post-test results, and ratings on attitudinal and enjoyment surveys. Learning gains were significantly higher for the GUI group than the TUI group. However, post-activity increases in reported attitude toward computing were significantly higher for the TUI group. There was no difference in activity enjoyment scores, which were high for both groups.

Session 4C: BPC

Applying NCWIT Protocol to Broaden Participation in Computing: A Case Study of CS@Mines

Authors: Tracy Camp; Christine Liebe; Michelle Slattery


Best Paper, Experience Reports and Tools

The Department of Computer Science (CS@Mines) at Colorado School of Mines (Mines) was founded in 2016 when the Department of Electrical Engineering and Computer Science (CS) split into separate departments. As a result, CS faculty, who had worked for years to broaden participation in computing without departmental lead- ership support, were able to become more strategic in their efforts. CS@Mines faculty, staff, and students now engage in well-defined recruitment, retention, and evaluation strategies, which includes K-12 outreach programs, flexible CS major and minor tracks, a near-peer mentoring program, scholarship programs, and contin- ual evaluation. Ten years ago, the CS degree program at Mines had 157 majors, 17 women (10.8%), and 12 students from underrepre- sented groups in computing (7.6%). As of Fall 2019, CS@Mines has 679 majors, 146 women (21.5%), and 132 students from underrep- resented groups (20.2%). Although the concentrated effort focused on increasing the number of women majors, the data clearly shows an increase in students from underrepresented groups as well. The changes achieved by CS@Mines are noteworthy considering (1) only 30% of the students at Mines are women, (2) only 17% of the students at Mines are from underrepresented groups in computing, and (3) women and underrepresented groups enrolled in undergraduate CS programs are predominantly not at parity with their respective populations in the United States. CS@Mines achieved positive results by applying the Undergraduate Systemic Change Model developed by the National Center for Women & Information Technology. In this paper, we present CS@Mines as a case study for positive change and discuss the strategies CS@Mines has enacted (74 of the recommended 81).

Experience of Administering Our First S-STEM Program to Broaden Participation in Computer Science

Authors: An-I Andy Wang; David Whalley; Zhenghao Zhang; Gary Tyson


This paper documents the findings of our analysis of the implementation of our six-year NSF S-STEM scholarship program. One major finding was that, for underrepresented students to major in computer science, knowing the major existed and understanding the nature of the program were the most important factors. Also, the academic support system and hands-on nature of the major had a significant impact on scholarship recipients' persistence in the major. Evidence demonstrated that scholarship recipients had a 10%+ higher year-to-year persistence rate from their freshmen to sophomore year than that of all computer science students of the same entering classes. For all computer science students, college computer science major GPAs were not strongly correlated with their high school GPAs, financial need, or ACT math scores. This paper also presents lessons learned and resulting recommendations for future new scholarship administrators, as our lessons can likely be applied to other grants that recruit and deal with underrepresented groups.

Does AP CS Principles Broaden Participation in Computing? An Analysis of APCSA and APCSP Participants

Authors: Linda Sax; Kaitlin N. S. Newhouse; Joanna Goode; Max Skorodinsky; Tomoko M. Nakajima; Michelle Sendowski


A major attempt to broaden participation in computer science has centered on the design and development of a new high school Advanced Placement (AP) course, AP Computer Science Principles (CSP). This course was created to intentionally engage a wider and more diverse group of students in learning about computing than those who had historically enrolled in programming-focused AP Computer Science "A" (CSA). After several years in the pilot phase, the course was officially offered by the College Board in 2016-17. This paper uses nationwide Freshman Survey data to examine the demographics and characteristics of students who took either CSA, the new CSP course, or both courses by the time they entered college in Fall 2017. Using crosstabs, z-tests, and one-way ANOVA, this study uncovered the similarities and differences between students who had participated in one or both of these courses. Our findings suggest that while students who took only CSP were more diverse than those who took only CSA, they exhibited less computing confidence and less interest in computing majors and tech careers. This study suggests that while CSP may be recruiting more females and racially diverse students into the course, this course alone may not serve as a direct pipeline into computing majors and careers.

Session 4D: Rubrics and Evaluation

Adaptive Rubrics

Authors: Marco Carmosino; Mia Minnes


Grading is a notoriously difficult and time-consuming part of teaching. For open-ended programming, mathematical, or design problems, assigning consistent scores and giving useful feedback can be very challenging. Large classes compound this difficulty. Adding TAs to the team can help parallelize the process but may impede grading consistency and quality. We present an adaptive rubric creation and application process to enable high-quality responses to student work, at scale. This process uses exploratory data analysis to discover common patterns in student responses to a problem, then tailors a rubric and feedback to address these patterns. Our method is supported by current grading tools, which allow calculation of the simple population-level statistics we need to extract meaningful features from a corpus of student work. In this case study, we describe using adaptive rubrics for a discrete math class for CS majors: the grading team found that this process produced concrete and transparent justifications of student scores and that it facilitated conversations around grading that were grounded in course learning objectives and values.

A Large-scale Evaluation of a Rubric for the Automatic Assessment of Algorithms and Programming Concepts

Authors: Nathalia Alves; Christiane Gresse von Wangenheim; Jean Hauck; Adriano Ferreti Borgatto


As computing education makes its way into schools, there is still little research on how to assess the learning of algorithms and programming concepts as a central topic. Furthermore, in order to ensure valid instructional feedback, an important concern is the reliability and construct validity of an assessment model. Therefore, this work presents a large-scale evaluation of the CodeMaster rubric for the performance-based assessment of algorithms and programming concepts by analyzing software artifacts created by students as part of complex, open-ended learning activities. The assessment is automated through a web-based tool that performs a static analysis of the source code of App Inventor projects. Based on 88,812 projects from the App Inventor Gallery, we statistically analyzed the reliability and construct validity of the rubric. Results indicate that the rubric can be regarded as reliable (Cronbach's alpha = 0.84). With respect to construct validity, there also exists an indication of convergent validity based on the results of a correlation and factor analysis. This indicates that the rubric can be used for a valid assessment of algorithm and programming concepts of App Inventor programs as part of a comprehensive assessment completed by other assessment methods. The results can guide the improvement of assessment models, as well as support the decision on the application of the rubric in order to support computing education in K-12.

A Validated Scoring Rubric for Explain-in-Plain-English Questions

Authors: Binglin Chen; Sushmita Azad; Rajarshi Haldar; Matthew West; Craig Zilles


Previous research has identified the ability to read code and understand its high-level purpose as an important developmental skill that is harder to do (for a given piece of code) than executing code in one's head for a given input ("code tracing"), but easier to do than writing the code. Prior work involving code reading ("Explain in plain English") problems, have used a scoring rubric inspired by the SOLO taxonomy, but we found it difficult to employ because it didn't adequately handle the three dimensions of answer quality: correctness, level of abstraction, and ambiguity.

In this paper, we describe a 7-point rubric that we developed for scoring student responses to "Explain in plain English" questions, and we validate this rubric through four means. First, we find that the scale can be reliably applied with with a median Krippendorff's alpha (inter-rater reliability) of 0.775. Second, we report on an experiment to assess the validity of our scale. Third, we find that a survey consisting of 12 code reading questions had a high internal consistency (Cronbach's alpha = 0.954). Last, we find that our scores for code reading questions in a large enrollment (N = 452) data structures course are correlated (Pearson's R = 0.555) to code writing performance to a similar degree as found in previous work.

Session 4E: Faculty

Faculty Adoption of CS Education Innovations: Exploring Continued Use

Authors: Christopher Hovey; Lecia Barker


Studies on faculty adoption typically focus on the awareness and trial stages: how instructors find out about a new teaching practice and why they decide to try it. While this knowledge is important, reform is unlikely to occur if innovations are only used experimentally; we need to understand why faculty sustain their use. To that end, this study draws on data from a two-phase project in the U.S. to examine why computer science (CS) faculty continue or discontinue use of a practice after the initial trial. In the first phase, we interviewed and observed 66 CS faculty from various institutional settings to explore theory-driven themes. From these results, in the second phase, we collected survey data from 821 faculty at 595 institutions. In this paper, we briefly discuss qualitative results, and then use quantitative data to model what impact the following factors have on sustaining use: achieved benefits for students' performance and their satisfaction in CS, the perception of usefulness to the instructor, student feedback, and ease of use. Results indicate that benefits to students' performance are paramount in predicting continued use. We also explore why faculty abandon a practice, finding that the decision often relates to not achieving desired outcomes and, in some cases, students not making a good faith effort to do their part. We observed that the latter experience can engender negative beliefs about students-that they have, at best, a passive attitude toward their learning. Implications for encouraging sustained usage of innovative teaching techniques are discussed.

Faculty Views on the Goals of an Undergraduate CS Education and the Academia-Industry Gap

Authors: Sander Valstar; Sophia Krause-Levy; Alexandra Macedo; William G. Griswold; Leo Porter


Previous work has found that recent computer science graduates often experience difficulty transitioning into their new roles in industry due to a significant gap between their academic experiences and industry's expectations. Although multiple studies have identified the views of students and members of industry on the value of a CS degree as preparation for industry, the faculty perspective on this topic remains unclear. Understanding these views could shed light on why the academia-industry gap has persisted despite the attention. This study identified faculty views on the goals of an undergraduate education and a CS major, focusing on preparation for careers in industry. In order to identify a spectrum of faculty views, we interviewed 14 faculty from a variety of backgrounds across three institutions. A phenomenographic analysis of the transcripts reveals that many faculty believe that industry preparation is an important programmatic goal, yet they encounter significant resource obstacles to achieving that goal.

Linking Faculty Attitudes to Pedagogical Choices: Student-Centered Teaching in Introductory Computing Classes

Authors: Christopher Hovey; Kathleen Lehman; Tiffani Riggers-Piehl


Research suggests using student-centered practices in the classroom is a key component of attracting and retaining diverse students. To better understand the link between attitudes toward students and learning and the usage of specific teaching strategies, we analyze survey responses from 54 faculty who teach introductory computer science (CS) courses from 15 U.S. colleges and universities participating in BRAID. Using principal component analysis, we scale responses to 10 attitudinal questions into four dimensions: rugged individualism ("learning and success are the individual student's responsibility"), challenging work ("the pace and workload in CS is hard"); a developmental orientation to learning ("students need individual attention in a non-competitive environment"); and capable students ("all students can do well in my class"). We then test these scales on four types of teaching: three student-centered approaches-collaborative learning approaches, discourse activities, and student-led learning-and one traditional approach, lecturing. Results indicate that a developmental orientation predicts the use of discourse activities and student-led practices, but not collaborative learning approaches. Rugged individualism is associated with frequent lecturing. None of our scales predict the use of collaborative learning approaches, and neither attitudes for "challenging work" nor "capable students" predict any of the pedagogical approaches in our study. We examine differences by certain faculty characteristics and discuss the ramifications of these results for promoting more widespread adoption of student-centered teaching.

Session 4F: Computing Education Research

A Gap Analysis of Statistical Data Reporting in K-12 Computing Education Research: Recommendations for Improvement

Authors: Monica M. McGill; Adrienne Decker


The quality of reporting of experimental results in computing education literature has been previously shown to be less than rigorous. In this study, we first examined research standards set forth by four organizations: American Psychology Association (APA), American Educational Research Association (AERA), What Works Clearinghouse (WWC), and the CONsolidated Standards of Reporting Trials (CONSORT). We selected the most important data standards based on their prominence across all four and the most typical study designs in computing education research. We then examined 76 articles designated as quantitative research studies (K-12) published in ten venues (2012-2018) to determine whether the reporting in these articles met these five standards. Findings indicate that only 48% of these articles report effect size and even fewer (11%) report confidence intervals and levels. We found that reported data did not meet the standard that data should be "reported in a way that the reader could construct effect-size estimates and confidence intervals beyond those supplied in the paper". Additionally, authors used existing instruments less than a quarter of the time (24%) and used instruments with evidence of reliability and validity less than half of the time (39%). We conclude with recommendations for those in the K-12 computing education research community to consider when reporting statistical data in future work so that we can increase the level of rigorous reporting in this growing field.

Three Metrics of Success for High School CSforAll Initiatives: Demographic Patterns from 2003 to 2019 on Advanced Placement Computer Science Exams

Authors: Kip Lim; Colleen Lewis


There has been an expansion of computer science (CS) in high schools in the USA. In most cases, initiatives seeking to expand high school CS offerings are not focused simply on increasing the number of students enrolled, but are focused on broadening participation in computing (BPC). Ideally we can evaluate and replicate effective BPC, or CSforAll, initiatives at the high school level. However, analyses of CS expansion frequently focus on the national landscape, despite the decentralized nature of public education in the USA. State-level analyses, when present, often do not take into account the demographics of high school students in the state, which are also changing over time. We propose three metrics for evaluating the impact of state-level initiatives to broaden participation in computing at the high school level. These metrics and our data set can be helpful benchmarks for evaluating high school CSforAll initiatives.

A Longitudinal Analysis of K-12 Computing Education Research in the United States: Implications and Recommendations for Change

Authors: Bishakha Upadhyaya; Monica M. McGill; Adrienne Decker


The availability of computer science education in primary and secondary schools in the United States has grown rapidly over the last decade. Computing education research in K-12 has been increasing as well. In this study, we conducted a longitudinal analysis of seven years of data (2012-2018), curated from over 500 articles across 10 publication venues to identify trends in K-12 computing education research such as geographic location and curriculum and concepts taught. The data shows a decrease in the number of studies covering K-12 students receiving computing education even while there is an increase in the number of states adopting and implementing standards. The number of different concepts being researched is increasing, potentially reflecting the growth in what is being taught in the classroom. Demographic data is underreported (e.g., socio-economic status (SES) and disabilities of participants) which could directly limit generalizability of the studies to different learners as well as the ability to replicate and compare studies. We conclude with recommendations for how to better position this work for others trying to use the results to guide their efforts in creating standards or adopting techniques into their classrooms.

Session 4G: Database

Worked Examples, Cognitive Load, and Exam Assessments in a Senior Database Course

Authors: Edwin M. Knorr


This paper deals with an upper-year course on internals of relational database systems. It is an experience report about switching from a teaching model using traditional lecturing and homework assignments, to a partially flipped model that replaces regular homework with an abundance of worked examples and more frequent full-length exams. We turn to cognitive load theory to help address the challenges of problem-solving in complex topics such as query evaluation and optimization. Worked examples are used to try to reduce students' cognitive load. Students' confidence levels were captured through surveys. After seeing and practicing many worked examples, students claim confidence in mastering the learning goals. We use Bloom's Dichotomous Key to create a Bloom's Taxonomy weighted index to track the difficulty of our exams over the years, both before and after the switch. Although final exam grades have remained relatively flat, our exams have been getting more difficult, suggesting improved learning, even though we have not seen a significant increase in performance on transfer questions. Principal Component Analysis is used to gain insight into our final exam, showing that a large fraction of the exam questions are needed to account for most of the variance in students' scores.

Mapping the SQL Learning Process in Introductory Database Courses

Authors: Andrew Migler; Alexander Dekhtyar


The Structured Query Language (SQL) is powerful, prevalent across many problem domains, and challenging to master. Sophisticated information requests require programmers to set aside familiar procedural and functional modes of thought. Instead, programmers must learn how to apply unfamiliar set-based relational manipulation rules. This paper analyzes the student learning process in an introductory database course. We do this using detailed analysis of problem-solving attempts by 114 students related to 116 SQL lab exercises assigned over a five-week period. We measure student success rates in mastering these SQL concepts, as well as effort expended by students in solving information retrieval problems.

Group Exams as Learning Tools: Evidence from an Undergraduate Database Course

Authors: Jalal Kawash; Tamer Jarada; Mohammad Moshirpour


Peer-instruction has been shown to be an effective method to support learning. We exploit a form of peer-instruction in an undergraduate course on databases, where students take an exam in teams. Instantly after attempting the exam individually without any immediate feedback, students re-take the exam as a group utilizing immediate feedback instruments. These instruments permit the students to tackle each multiple-choice question several times until a correct answer is uncovered. Our thesis is that this approach provides the students with an opportunity to learn from their mistakes, whether committed individually or as a group, while promoting individual student deep learning. We support this thesis by analyzing data collected from 125 students, 5 group exams, and other assessment instruments, including a final exam.

Session 5A: Software Engineering

What Agile Processes Should We Use in Software Engineering Course Projects?

Authors: An Ju; Adnan Hemani; Yannis Dimitriadis; Fox Armando


While project-based software engineering courses aim to provide learning opportunities grounded in professional processes, it is not always possible to replicate every process in classrooms due to course constraints. Previous studies observed how students react to various processes and gave retroactive recommendations. In this study, we instead combine a field study on professional Agile (eXtreme Programming, XP) teams and an established team process taxonomy to proactively select team processes to incorporate in a project-based software engineering course. With collected knowledge from the field study, we choose three XP processes to augment the design of a mature software engineering project course. We choose processes that are 1) considered important by professionals, and 2) complete with respect to coverage of the taxonomy's main categories. We then compare the augmented course design with the original design in a case study. Our results suggest that 1) even without extra resources, adding these new processes does not interfere with learning opportunities for XP processes previously existing in the course design; 2) student teams experience similar benefits from these new processes as professional teams do, and students appreciate the usefulness and value of the processes. In other words, our approach allows instructors to make conscious choices of XP processes that improve student learning outcomes while exposing students to a more complete set of processes and thus preparing them better for professional careers. Course designers with limited resources are encouraged to use our methodology to evaluate and improve the designs of their own project-based courses.

Assessing Individual Contributions to Software Engineering Projects with Git Logs and User Stories

Authors: Kevin Buffardi


Software Engineering courses often incorporate large-scale projects with collaboration between students working in teams. However, it is difficult to objectively assess individual students when their projects are a product of collaborative efforts. This study explores measurements of individuals' contributions to their respective teams.

I analyzed ten Software Engineering team projects (n=42) and evaluations of individual contributions using automated evaluation of the version control system history (Git logs) and user stories completed on their project management (Kanban) boards. Unique insights from meta-data within the Git history and Kanban board user stories reveal complicated relationships between these measurements and traditional assessments, such as peer review and subjective instructor evaluation. From the results, I suggest supplementing and validating traditional assessments with insights from individuals' commit history and user story contributions.

Teaching Autonomous Systems at 1/10th-scale: Design of the F1/10 Racecar, Simulators and Curriculum

Authors: Abhijeet Agnihotri; Matthew O'Kelly; Houssam Abbas; Rahul Mangharam


2nd Best Paper, Curricula Initiatives

Teaching autonomous systems is challenging because it is a rapidly advancing cross-disciplinary field that requires theory to be continually validated on physical platforms. For an autonomous vehicle (AV) to operate correctly, it needs to satisfy safety and performance properties that depend on the operational context and interaction with environmental agents, which can be difficult to anticipate and capture. This paper describes a senior undergraduate level course on the design, programming and racing of 1/10th-scale autonomous race cars. We explore AV safety and performance concepts at the limits of perception, planning, and control, in a highly interactive and competitive environment. The course includes an ethics-centered design philosophy, which seeks to engage the students in an analysis of ethical and socio-economic implications of autonomous systems. Our hypothesis is that $1/10th-scale autonomous vehicles sufficiently capture the scaled dynamics, sensing modalities, decision making and risks of real autonomous vehicles, but are a safe and accessible platform to teach the foundations of autonomous systems. We describe the design, deployment and feedback from two offerings of this class for college seniors and graduate students, open-source community development across 36 universities, international racing competitions, student skill enhancement and employability, and recommendations for tailoring it to various settings.

Session 5B: Middle School Experiences

Drag and Drop Programming Experiences and Equity: Analysis of a Large Scale Middle School Student Motivation Survey

Authors: Jeffrey Bush; Monica R. Gilmore; Susan B. Miller


Drag and drop programming languages have been promoted for broadening participation and increasing motivation for middle grade students in computer science (CS), yet their impact on motivation and the implications for equity are not well documented. This raises questions about potential issues of equity. Despite considerations to increase participation and diversity, these tools have not been evaluated as to how they provide more equitable outcomes regarding student motivation in CS. This study analyzes data from a survey (n=9,573) to investigate middle school student motivation in CS when considering previous experiences with drag and drop programming. Student responses to motivational items are categorized into four factors based on exploratory and confirmatory factor analysis. Multiple regression interaction analysis shows that previous experience is associated with increases (p<0.01) in three motivational factors: general academic and computer use confidence, programming confidence, and interest in future CS courses. These associations hold regardless of race and gender although they are lower for demographics underrepresented in CS, female and/or non-White and non-Asian students. Previous experience was also associated with an increase in interest in the current course for White and Asian male students and a decrease for female underrepresented minority students. While it appears that drag and drop programming efforts could help increase future interest and confidence for all students, it does so to differing degrees for different demographics. This suggests that these initiatives are not addressing issues of equity since groups overrepresented in CS still report higher gains than underrepresented demographics after early programming experiences.

Exploring Middle School Students' Reflections on the Infusion of CS into Science Classrooms

Authors: Mehmet Celepkolu; David Austin Fussell; Aisha Galdo; Kristy Boyer; Eric Wiebe; Bradford Mott; James C. Lester


In recent years, there has been a dramatic increase in teaching CS in the context of other disciplines such as science. However, learning CS in an interdisciplinary context may be particularly challenging for students. An important goal for CS education researchers is to develop a deep understanding of the student experience when integrating CS into science classrooms in K-12. This paper presents the results of a mixed-methods study in which 75 middle school students engaged in a series of computationally rich science activities by creating simulations and models in a block-based programming language. After two semesters, students reported their experiences on in-class computer science activities through reflection essays. The quantitative results show that both experienced and novice students increased their CS knowledge significantly after several weeks, and a majority of students (72%) had positive sentiment toward the integration of CS into their science class. Deeper qualitative analysis of students' reflections revealed positive themes centered around the visualization and gamification of science concepts, the hands-on nature of the coding activities, and showing science from a different angle. On the other hand, students expressed negative sentiments on weaknesses in the activity design, lack of CS/science background/interest, and failing to make connections between CS and science concepts. These findings inform efforts to infuse CS education into different disciplines and reveal patterns that may foster success of K-12 classroom implementations.

Designing an Assessment for Introductory Programming Concepts in Middle School Computer Science

Authors: Shuchi Grover


Teaching of computer science (CS is rapidly expanding in schools. Learning to program is a key ingredient of school CS curricula, and consequently, there is a need for quality measures of student learning of foundational programming concepts. However, high-quality tools for measuring student learning in introductory CS have been under-developed and under-researched. This experience report shares the process of design and refinement of a summative paper-based assessment (that could also be administered online) for introductory programming in middle grades (6-8). We share our experiences with the use of assessment as a pre-post measure in a middle school introductory programming course in diverse, urban school classrooms in the US and use that data to conduct validity, reliability and item discrimination analyses.

Session 5C: Gender

Competitive Enrollment Policies in Computing Departments Negatively Predict First-Year Students' Sense of Belonging, Self-Efficacy, and Perception of Department

Authors: An Nguyen; Colleen Lewis


Best Paper, CS Education Research

Enrollment in computing at the college level has skyrocketed, and many institutions have responded by enacting competitive enrollment processes. However, little is known about the effects of enrollment policies on students' experiences. To identify relationships between those policies and students' experiences, we linked survey data from 1245 first-year students in 80 CS departments to a dataset of department policies. We found that competitive enrollment negatively predicts first-year students' perception of the computing department as welcoming, their sense of belonging, and their self-efficacy in computing. Both belonging and self-efficacy are known predictors of student retention in CS. In addition, these relationships are stronger for students without pre-college computing experience. Our classification of institutions as competitive is conservative, and false positives are likely. This biases our results and suggests that the negative relationships we found are an underestimation of the effects of competitive enrollment.

Gender Differences in Introductory Programming: Comparing MOOCs and Local Courses

Authors: Rodrigo Duran; Lassi Haaranen; Arto Hellas


We analyzed three introductory programming MOOCs and four introductory programming courses offered locally in a Finnish university. The course has been offered in all instances with roughly the same content, barring adjustments based on course feedback. We sought to understand how gender interacts with participating in the course in both instances. In particular, we looked at the differences in persistence, confidence, interest in CS, prior experience, and performance between men and women.

Overall, we found that men have more prior experience in both instances and have a higher interest in a CS degree. Furthermore, men perform slightly better on the MOOC while there was no significant difference in performance when it came to gender in the local instance. Aligned with prior research, we found a considerable gap in confidence between male and female students in both instances. At the same time, while women are still underrepresented in CS, we observe a considerable increase in women attending the MOOC. Unfortunately, women are also more likely to drop out early on in the MOOC than men.

An Unexplored Relationship: Women in Computing and Athletics

Authors: Khadija Jirari Stewart; Gloria Childress Townsend; Sharmin Tunguz


The literature documents the underrepresentation of women in computer science, reasons for the absence of women in computing, and promising practices for increasing female majors in computer science. Among the many reasons for women's avoidance of computer science are: feelings of intimidation, lack of confidence, and aversion to the competitive climate in computing. This paper reports results from a study whose purpose lies in surveying a sample of 457 students at a Midwest US university to answer the questions: What are the personality traits of female athletes who enroll in computer science courses? How are these traits similar to or different from those of male athletes, non-athletes, and students who do not enroll in Computer Science I (CS1)? How do grades in CS1 differ among these groups? How do personality traits, gender, athletic team membership, and grades interact? The results of this research study indicate that when females' levels of confidence and intimidation are similar to those of their male peers, women outperform males in CS1. This novel study and its results open the door to additional research questions, new strategies for recruiting women into computing, and potential projects involving female athletes and computer science.

Session 5D: Exam Format

Paper Or IDE? The Impact of Exam Format on Student Performance in a CS1 Course

Authors: Jonathan Corley; Ana Stanescu; Lewis Baustark; Michael C. Orsega


Exams are an integral assessment component in CS1 courses. However, little research exists concerning the impact of the exam format on student performance. To address this deficit, we conducted a year-long study to investigate student performance across two test formats, paper and IDE, with the goal of identifying any differences attributable to the format. Data was collected from the second and third exams over the course of two semesters, with five sections per semester. Each exam, whether paper or IDE, was constructed from a common list of concepts to ensure comparability across formats and sections. To assess the impact of exam format, we considered three points of concern. The first factor, grade received, provides a direct indication of student performance. Next, we examined the type of error committed by students (either syntax or logic) to determine influence of exam format (e.g., Does an IDE reduce the occurrence of syntax errors?). Finally, we considered exam completion rates to address concerns such as if the additional feedback might be overwhelming students in an already stressful environment (exam-taking) leading to lower completion rates.

The Reverse Exam: A Gamified Exam Structure to Motivate Studying and Reduce Anxiety

Authors: Pablo Frank-Bolton; Rahul Simha


This experience report describes an attempt to improve student attitudes towards exams by encouraging students to craft exam questions that earn game points and by allowing students to defer some questions to a second attempt at the exam a week later, increasing study time while reducing common timed-test anxiety. The approach, inspired by research in gamification and student-generated questions, focuses on: getting students to study broadly across the material; encouraging students to craft good questions; encouraging an honest first attempt; preventing memorization for the second attempt; incorporating teamwork. Data collected from implementations in two different courses indicate that several finer points of the game design are important and that student-generated questions can be just as effective as instructor-generated questions. Survey data shows that students are very positive about having a second chance at learning.

Inverted Two-Stage Exams for Prospective Learning: Using an Initial Group Stage to Incentivize Anticipation of Transfer

Authors: Patrice Belleville; Steven Wolfman; Susanne Bradley; Cinda Heeren


We propose a novel, inverted two-stage exam format that encourages anticipation of transfer problems. We report on its design, use, and initial assessment for low-stakes quizzes in an algorithms course. A typical two-stage exam, where the group stage comes after the individual stage, emphasizes retrospective learning: reflecting on already-solved problems. Our inverted two-stage format places the group stage first, and incentivizes prospective learning: preparing for transfer to novel problems that will appear on the individual stage and subsequent assessments.

TAs reported the new format leads to reliable engagement. In surveys, most students preferred inverted two-stage quizzes to individual quizzes plus a TA walkthrough. Students who preferred this format cited the value of learning from peers, brainstorming in the problem domain, and working out ambiguities in the domain and problems.

Session 5E: Planning

Applying the Design Process to Life Goals: An Experience Report from a Capstone Course

Authors: Tammy VanDeGrift


A key element in many computing and engineering capstone courses is a team or individual project. The project is often open-ended, requiring students to embrace ambiguity, consider constraints such as time, tools, and finances, and create one or more prototypes. One may also apply design thinking and the design process to their career and lives. This paper presents a "design your life" assignment, adapted from the Stanford Life Design Lab, that could be incorporated into any capstone course. The assignment had five parts: 1) three 5-year prototypes including professional and personal goals, 2) one 10-year plan, 3) a thank you note, 4) a gallery walk to observe and comment on other student's 10-year plans, and 5) reflection from a prototype conversation or experience. The goal of the prototype conversation or experience is to give students a low-stakes method to gather information about themselves. For example, if a student thinks they want to coach a robotics team but has not worked extensively with children, volunteering at an afterschool program would be a great prototype experience. The goal of the 5-year and 10-year plans is to make students create a coherent plan, so they have a roadmap for the first part of their professional journeys. The intention of the gallery walk is to build community. This paper presents the course context, details about the assignment, examples of student work, and suggestions for adoption.

RadGrad: Removing the "Extra" from Extracurricular to Improve Student Engagement, Diversity, and Retention

Authors: Philip Johnson; Carleton Moore; Peter Leong; Seungoh Paek


RadGrad is a curriculum initiative implemented via a web-based application that combines features of social networks, degree planners, and serious games. RadGrad redefines the traditional meaning of "progress" and "success" in the undergraduate computer science degree program, with the ultimate goal of improving student engagement, diversity, and retention. In this paper, we relate RadGrad to other curriculum initiatives, overview its key functionality, present results from an evaluation conducted during its first year of deployment, and discuss our lessons learned and future directions.

Benefits and Pitfalls of Using Capture the Flag Games in University Courses

Authors: Jan Vykopal; Valdemar Švábenský; Ee-Chien Chang


The concept of Capture the Flag (CTF) games for practicing cybersecurity skills is widespread in informal educational settings and leisure-time competitions. However, it is not much used in university courses. This paper summarizes our experience from using jeopardy CTF games as homework assignments in an introductory undergraduate course. Our analysis of data describing students' in-game actions and course performance revealed four aspects that should be addressed in the design of CTF tasks: scoring, scaffolding, plagiarism, and learning analytics capabilities of the used CTF platform. The paper addresses these aspects by sharing our recommendations. We believe that these recommendations are useful for cybersecurity instructors who consider using CTF games for assessment in university courses and developers of CTF game frameworks.

Session 5F: CS1 Interventions

The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior

Authors: Ioannis Karvelas; Annie Li; Brett Becker


It is generally accepted that learning to program could be easier for many students. One of the most important components of this experience is the programming environment. Novices learn in a variety of environments, from basic command-line interfaces to industry-strength IDEs. These environments can differ substantially in compilation behavior and error message presentation - arguably two of the most important mechanisms through which users interact with the programming language. In this study, we utilize Blackbox data to compare the programming behavior of thousands of users programming in Java, who all used BlueJ versions 3 and 4. These two versions differ drastically in terms of compilation behavior and error message presentation. BlueJ 3 is a click-to-compile editor that delivers text-based error messages from javac to the user, but only presents the first error message, even if the compiler produces several. BlueJ 4 automatically compiles in the background but retains click-to-compile ability. In addition, all error messages (not just the first) may be viewed by the user. We find that the programming experience and behavior of these users can be substantially affected by changes in these mechanisms, causing numbers of manual compilations, successful compilations, and error messages presented in each version to differ, in cases, markedly. Our results provide evidence on how changes in programming environment affect user behavior in conditions that reasonably control for variables other than the programming environment. This can inform the decisions of educators, tool designers, and HCI researchers in their work to make learning more effective for novice programmers.

A CS1 Spatial Skills Intervention and the Impact on Introductory Programming Abilities

Authors: Ryan Bockmon; Stephen Cooper; William Koperski; Jonathan Gratch; Sheryl Sorby; Mohsen Dorodchi


This paper discusses the results of replicating and extending a study performed by Cooper et al. examining the relationship between students' spatial skills and their success in learning to program. Whereas Cooper et al. worked with high school students participating in a summer program, we worked with college students taking an introductory computing course. Like Cooper et al.'s study, we saw a correlation between a student's spatial skills and their success in learning computing. More significantly, we saw that after applying an intervention to teach spatial skills, students demonstrated improved performance both on a standard spatial skills assessment as well as on a CS content instrument. We also saw a correlation between students' enjoyment in computing and improved performance both on a standard spatial skills assessment and on a CS content instrument, a result not observed by Cooper et al.

Improving Student Study Choices in CS1 with Gamification and Flipped Classrooms

Authors: Gina Sprint; Erik Fox


Recent research has explored flipped classrooms, lightweight teams, and gamification to improve the CS1 student learning and social experience. In this paper, we expand upon this research by implementing a flipped classroom with team-based gamification of student study choices. We designed and implemented a Moodle plugin for gamification to reward students for practicing "good" study habits, such as turning in assignments early and re-taking quizzes for extra practice. To evaluate our approach, we analyzed data collected from three consecutive semesters of CS1: one semester without gamification, one with gamification, and one with gamification tied to a small grade incentive. Based on log data from our plugin and student survey responses, we found students did make better study choices, including submitting programming assignments and online quizzes significantly earlier; however, these improved study choices did not lead to higher final exam scores. Overall, students enjoyed and were motivated to improve their study habits due to our flipped, team-based, gamified CS1 course.

Session 5G: Scratch

Eliciting Student Scratch Script Understandings via Scratch Charades

Authors: Diana Franklin; Jean Salac; Cathy Thomas; Zene Sekou; Sue Krause


With many school districts nationwide integrating Computer Science (CS) and Computational Thinking (CT) instruction at the K-8 level, it is crucial researchers closely inspect the relationship between program expression and student understandings.

In this study, we propose and report on our use of Scratch Charades, a game in which students act out Scratch scripts while others build them. The purpose of Scratch Charades is to familiarize students with scripts and blocks without the cognitive overhead of the complex user interface. However, in this study, we also used it to elicit student understandings about Scratch blocks and scripts to design mnemonics to help students debug their code. We propose two building and/or debugging strategies based on our observations.

Introducing Coding through Tabletop Board Games and Their Digital Instantiations across Elementary Classrooms and School Libraries

Authors: Victor R. Lee; Frederick Poole; Jody Clarke-Midura; Mimi Recker; Melissa Rasmussen


This experience report describes an approach for helping elementary schools integrate computational thinking and coding by leveraging existing resources and infrastructure that do not rely on 1-1 computing. A particular focus is using the school library and media center as a site to complement and enhance classroom instruction on coding. Further, our approach builds upon "unplugged" knowledge and practices that are already familiar to and motivating for students, in this case tabletop board games. Through these games, students can use their prior knowledge and ease with tabletop gaming mechanics to cue relevant ideas for core computational concepts. We describe a model and an instructional unit spanning across classroom and school library settings that builds upon board game play as a source domain for computing knowledge. Building on expansive framing, the model emphasizes instructional linkages being made between one domain (the tabletop board game) and another (specially designed Scratch project shells with partially complete code blocks) such that the reasoning activities and different contexts are seen as instantiations of the same encompassing context. We present the experiences of three elementary school teachers as they implemented the unit in their classrooms and with their school librarian. We also show initial findings on the impact of the unit on student interest (N=87), as measured by pre- and post- surveys. We conclude with lessons learned about ways to improve the unit and future classroom implementations.

Scratch Encore: The Design and Pilot of a Culturally-Relevant Intermediate Scratch Curriculum

Authors: Diana Franklin; David Weintrop; Jennifer Palmer; Merijke Coenraad; Melissa Cobian; Kristan Beck; Andrew M. Rasmussen; Sue Krause; Max White; Marco Anaya; Zachary Crenshaw


While several introductory computer science curricula exist for children in K-8, there are few options that go beyond sequence, loops, and basic conditionals. The goal of this project is to not only fill this gap with a high-quality curriculum supported by complete instructional materials, but to also do so with an equity-balanced curriculum. That is, a curriculum that values advancing equity equally with student learning outcomes. In this paper, we introduce barriers to equity in public school classrooms, pedagogical approaches to culturally-relevant curricula, and how our Scratch Encore curriculum is designed to support equity-balanced learning. Finally, we present results of our pilot year, including early evidence of students taking advantage of the culturally-relevant design aspects.

Session 6A: Blocks

Pythons and Martians and Finches, Oh My! Lessons Learned from a Mandatory 8th Grade Python Class

Authors: Amal Nanavati; Aileen Owens; Mark Stehlik


As computing technologies continue to have a greater impact on daily life, it becomes increasingly important for the K-12 education system to prepare students for the computerized world. In this paper, we present the curriculum design, implementation, and results from a one-trimester introductory Python course that is mandatory for all 8th graders in our school district. This course is a crucial component of the K-12 computational thinking pathways we are developing at our school district, which take students from block-based programming and computational thinking (elementary school) to text-based programming and applications of computer science (high school). Our mandatory 8th grade course serves as a bridge between these two components. We present qualitative results that highlight the challenges that arose from teaching a course for all students -- not just those with a prior interest in computing -- and how the instructor overcame those challenges. We also present quantitative results that demonstrate the course's positive impact on students' attitudes towards computer science, their intent to re-engage with computer science in the future, and the gender gap with regards to confidence in computer science.

Dual-Modality Instruction and Learning: A Case Study in CS1

Authors: Jeremiah Blanchard; Christina Gardner-McCune; Lisa Anthony


2nd Best Paper, CS Education Research

In college-level introductory computer science courses, students traditionally learn to program using text-based languages which are common in industry and research. This approach means that learners must concurrently master both syntax and semantics. Blocks-based programming environments have become commonplace in introductory computing courses in K-12 schools and some colleges in part to simplify syntax challenges. However, there is evidence that students may face difficulty moving to text-based programming environments when starting with blocks-based environments. Bi-directional dual-modality programming environments provide multiple representations of programming language constructs (in both blocks and text) and allow students to transition between them freely. Prior work has shown that some students who use dual-modality environments to transition from blocks to text have more positive views of text programming compared to students who move directly from blocks to text languages, but it is not yet known if there is any impact on learning. To investigate the impact on learning, we conducted a study at a large public university across two semesters in a CS1 course (N=673). We found that students performed better on typical course exams when they were taught using dual-modality representations in lecture and were provided dual-modality tools. The results of our work support the conclusion that dual-modality instruction can help students learn computational concepts in early college computer science coursework.

Cluster-Based Analysis of Novice Coding Misconceptions in Block-Based Programming

Authors: Andrew Emerson; Andy Smith; Fernando J. Rodríguez; Eric Wiebe; Bradford Mott; Kristy Boyer; James C. Lester


Recent years have seen an increasing interest in identifying common student misconceptions during introductory programming. In a parallel development, block-based programming environments for novice programmers have grown in popularity, especially in introductory courses. While these environments eliminate many syntax-related errors faced by novice programmers, there has been limited work that investigates the types of misconceptions students might exhibit in these environments. Developing a better understanding of these misconceptions will enable these programming environments and instructors to more effectively tailor feedback to students, such as prompts and hints, when they face challenges. In this paper, we present results from a cluster analysis of student programs from interactions with programming activities in a block-based programming environment for introductory computer science education. Using the interaction data from students' programming activities, we identify three families of student misconceptions and discuss their implications for refinement of the activities as well as design of future activities. We then examine the value of block counts, block sequence counts, and system interaction counts as programming features for clustering block-based programs. These clusters can help researchers identify which students would benefit from feedback or interventions and what kind of feedback provides the most benefit to that particular student.

Session 6B: High School Experiences

Investigating Different Assignment Designs to Promote Collaboration in Block-Based Environments

Authors: Nicholas Lytle; Alexandra Milliken; Veronica Cateté; Tiffany Barnes


Pair Programming is often employed in educational settings as a means of promoting collaboration and scaffolding the assignment difficulty for teams. While much research supports its inclusion as a pedagogical practice at the university level, some research has demonstrated in K-12 contexts, it can potentially lead to inequitable learning enviroments and create dynamics between partners that might negatively effect novice learners. New block-based programming environments like Netsblox have attempted to address this by creating ways for both partners to program simultaneously, but this feature has yet to be examined in detail. In this paper, we introduce several modes of Collaboration afforded by Netsblox. This includes Pair-Separate, Pair-Together, and Partner Puzzles - a mode that Splits the necessary blocks to build the assignment between team members. From an initial pilot study involving 25 pairs of middle and high school students, we find that most pairs preferred working on assignments in the Partner Puzzle mode as it presented a fun challenge to teams. We end on recommendations for building assignments using this methodology and future research directions investigating the role of collaboration in programming

A State-Wide Effort to Provide Access to Authentic Computer Science Education to Underrepresented Populations

Authors: Mohammed Qazi; Jeff Gray; David M. Shannon; Melody Russell; Misty Thomas


The Exploring Computer Science (ECS) curriculum provides foundational knowledge of Computer Science (CS) to high school students as a stand-alone course. ECS began in the Los Angeles Unified School District in the late 2000s where it gained eminence for broadening participation in computing (BPC), with Latinx students representing over 70% of enrollment. This experience report describes a partnership that consists of three Universities, dozens of school districts, the ECS team, and other stakeholders to bring the ECS curriculum in mainly rural school districts in Alabama that have a majority African-American student population. Sixty in-service teachers (one teacher per school) have received professional learning opportunities to gain knowledge and skills to teach ECS. Signs of early broader impacts are emerging: 78% of our ECS enrollment are underrepresented minority (URM) students with nearly half of the cohort consisting of female students. Students reported they were engaged in working collaboratively and sharing responsibilities with others. Furthermore, students who reported being more involved in the ECS course had deeper confidence in their ability to succeed in CS, reported greater overall outcomes, had more confidence in development of 21st century skills, found the course more relevant, were more motivated to persist in CS, and exhibited increased interest in CS careers. We provide a comprehensive description of the partnership’s accomplishments and the evaluation findings on student CS experiences and on teacher self-efficacy in ECS preparation and instruction. Our findings contribute to the BPC literature, specifically for schools with predominantly African-American enrollment in rural communities.

Self-Regulation for High School Learners in a MOOC Computer Science Course

Authors: Philip Sands; Aman Yadav


Courses designed for Massive Open Online Courseware (MOOC)platforms provide learners worldwide with extensive learning opportunities. Previous research has explored learner motivation in MOOC courses using self-regulated learning (SRL) theory. How-ever; the majority of this work has focused on adults and we know little about how high school learners use SRL strategies. This paper explores self-regulated learning differences between high school students enrolled in traditional face-to-face and MOOC Advanced Placement Computer Science A courses. We found significant differences in student help seeking abilities between the two courses, but not in other SRL behaviors (such as, metacognition, effort regulation, etc.). In addition, help seeking was the only SRL strategy that was significantly related to MOOC students' grades on programming tasks. We discuss implications for designing MOOC courses for younger learners and directions for future research to disentangle the role of MOOC environment and motivation in influencing student outcome

Session 6C: Program Writing

How Plans Occur in Novices' Programs: A Method to Evaluate Program-Writing Skills

Authors: Umberto Costantini; Violetta Lonati; Anna Morpurgo


Program-writing skills are often assessed with computer-based practical exams where students are required to design, write and test small programs according to specifications written in natural language. The evaluation of such tasks is usually based on the correctness of outputs (possibly with the support of automatic tools) and the quality of code. In this paper, we propose to evaluate program-writing assignments by using Soloway's notion of plans [34]. We illustrate this idea by defining a marking scheme for some typical program-writing tasks and apply this approach to evaluate the programs written during the final examination of an introductory programming course. We then compare the resulting marks with the marks originally assigned by the instructors of the course. Findings show that the use of plans in the marking process is feasible and requires an effort similar to the one required by a traditional evaluation process, and the focus on plans when examining the source code of students gives useful insights about their difficulties.

Crescendo: Engaging Students to Self-Paced Programming Practices

Authors: Wengran (Emma) Wang; Rui Zhi; Alexandra Milliken; Nicholas Lytle; Thomas Price


This paper introduces Crescendo, a self-paced programming practice environment that combines the block-based and visual, interactive programming of Snap!, with the structured practices commonly found in Drill-and-Practice Environments. Crescendo supports students with Parsons problems to reduce problem complexity, Use-Modify-Create task progressions to gradually introduce new programming concepts, and automated feedback and assessment to support learning. In this work, we report on our experience deploying Crescendo in a programming camp for middle school students, as well as in an introductory university course for non-majors. Our initial results from field observations and log data suggest that the support features in Crescendo kept students engaged and allowed them to progress through programming concepts quickly. However, some students still struggled even with these highly-structured problems, requiring additional assistance, suggesting that even strong scaffolding may be insufficient to allow students to progress independently through the tasks.

Using Terminal Histories to Monitor Student Progress on Hands-on Exercises

Authors: Jelena Mirkovic; Aashray Aggarwal; David Weinman; Paul Lepe; Jens Mache; Richard Weiss


Hands-on exercises are often used to improve student engagement and knowledge retention in systems, networking and cybersecurity classes. Even when students comprehend the concepts, they may lack the skills to complete an exercise. Teachers need effective tools to identify these problems \textit{during} an assignment and offer targeted and timely help.

This paper explores the use of terminal histories with milestone detection to enable rapid, automated and on-going assessment of student work while performing hands-on exercises. We describe our system, called ACSLE, which monitors terminal input and output for each student, compares it with desired milestones, and produces both summaries and detailed statistics of student progress.

We analyze data from undergraduates at two colleges, as they performed several well-structured cybersecurity assignments on a network testbed. We show how ACSLE's output can help teachers identify students that struggle, understand why they struggle and offer timely help. ACSLE can also help teachers identify challenging tasks and plan class-wide interventions.

Session 6D: Exams

Measuring the Score Advantage on Asynchronous Exams in an Undergraduate CS Course

Authors: Mariana Silva; Matthew West; Craig Zilles


This paper presents the results of a controlled crossover experiment designed to measure the score advantage that students have when taking exams asynchronously (i.e., the students can select a time to take the exam in a multi-day window) compared to synchronous exams (i.e., all students take the exam at the same time). The study was performed in an upper-division undergraduate computer science course with 321 students. Stratified sampling was used to randomly assign the students to two groups that alternated between the two treatments (synchronous versus asynchronous exams) across a series of four exams during the semester. These non-programming exams consisted of a mix of multiple choice, checkbox, and numeric input questions. For some questions, the parameters were randomized so that students received different versions of the question and some questions were identical for all students. In our results, students taking the exams asynchronously had scores that were on average only 3% higher (0.2 of a standard deviation). Furthermore, we found that the score advantage was decreased by the use of randomized questions, and it did not significantly differ based on the type of question. Thus, our results suggest that asynchronous exams can be a compelling alternative to synchronous exams.

Securing Bring-Your-Own-Device (BYOD) Programming Exams

Authors: Oka Kurniawan; Norman Tiong Seng Lee; Christopher M. Poskitt


Traditional pen and paper exams are inadequate for modern university programming courses as they are misaligned with pedagogies and learning objectives that target practical coding ability. Unfortunately, many institutions lack the resources or space to be able to run assessments in dedicated computer labs. This has motivated the development of bring-your-own-device~(BYOD) exam formats, allowing students to program in a similar environment to how they learnt, but presenting instructors with significant additional challenges in preventing plagiarism and cheating. In this paper, we describe a BYOD exam solution based on \emph{lockdown browsers}, software which temporarily turns students' laptops into secure workstations with limited system or internet access. We combine the use of this technology with a learning management system and cloud-based programming tool to facilitate conceptual and practical programming questions that can be tackled in an interactive but controlled environment. We reflect on our experience of implementing this solution for a major undergraduate programming course, highlighting our principal lesson that policies and support mechanisms are as important to consider as the technology itself.

A Modular, Practical Test for a Programming Course

Authors: Jan Stolarek; Przemyslaw Nowak


In order to evaluate students' programming skills during a university course, a practical programming test can be administered, in which students are required to implement a short yet complete program according to a provided specification. However, such tests often suffer from drawbacks that prevent comprehensive and accurate assessment of students' abilities. In this paper we identify these drawbacks and then present a modular, practical test that avoids common testing pitfalls, as well as show how to design such a test based on course learning outcomes. A key aspect of our approach is adoption of modularity, which ensures independent and comprehensive verification of learning outcomes. We have used our method to evaluate object-oriented programming skills of undergraduate students over several years and have found that our testing approach has proven its validity and superiority over approaches employed previously.

Session 6E: CS Curriculum

A Five-Year Retrospective on an Applied Baccalaureate Program in Software Development

Authors: Kendrick (Ken) Hang; Tina Ostrander; Andy Orr; Josh Archer; Susan Uland


This experience report documents an intervention that started five years ago, in 2014. Green River College, a community college in a suburb of Seattle, Washington, launched a new applied baccalaureate program in Software Development, with a goal of preparing place-bound, underrepresented, and underserved students for employment as entry-level software developers.

At program inception, community college baccalaureate degrees were relatively new across the nation and there was little precedent established in terms of program design, particularly as these new degree programs were not intended to emulate traditional undergraduate Computer Science programs, but rather provide an alternative pathway to the baccalaureate and workforce.

The Bachelor of Applied Science (BAS) in Software Development is designed as a "2+2" program, where students enter the program at third-year/junior-level standing after completing their two-year applied associate degree. The BAS Software Development curriculum includes courses in both "computer science fundamentals" and industry-focused, project-based learning courses. Since program inception, the curriculum has evolved based on evidence and feedback from students, instructors, alumni, and industry partners.

Initial key results at the five-year mark include a review of enrollment, completion, diversity, and placement data. As of June 2019, the program has 132 graduates (degrees conferred), with 52 new students on track to enter the program in the 2019-2020 academic year. A recent snapshot of currently enrolled students indicates the median age is 24, 23% are women, 40% are students of color, 39% are first generation college students, and 40% are receiving need-based financial aid. Graduates of the program have placed into roles that include Software Development Engineers, Solutions Engineers, and Technical Program Managers.

Validating a CS Attitudes Instrument

Authors: Ryan Bockmon; Stephen Cooper; Jonathan Gratch; Mohsen Dorodchi


This paper discusses the validation of a modified computer science attitudes instrument. Dorn and Tew's Computing Attitude Survey was modified by adding questions on gender issues and questions regarding students' perceptions of the utility of computing. Current trends indicate an increasing gap in the genders graduating with degrees in computer science. The new questions explore student attitudes and perceptions of women in computing while items related to computer science utility explore the importance of CS in students' lives and careers. These modifications necessitated the re-validation of the new revised instrument and comparison of the results obtained with in the original instrument by Dorn and Tew.

The Return Statement: Establishing a Continuous Assessment Database System for Consistent Program Feedback

Authors: Stephen Bothwell; Katie Timmerman


Curriculum assessment is a critical but arduous task. It yields insight into the strengths and weaknesses of our programs, outlining paths for curriculum improvement. However, current methods of assessment are often viewed as burdensome, as disorganized, or even as unfruitful. A lack of consistent assessment makes determining the impact of program modifications to be extremely difficult, if not impossible. Even when issues are identified, lengthy and backloaded assessments do not leave any time to address these issues for the assessed students.

To overcome the obstacles of program assessment, we have constructed the Continuous Assessment Database System (CADS). This system conducts direct assessment of student prerequisite knowledge at each semester's outset. By tracking prerequisite knowledge, we determine whether students recall the crucial elements of our curriculum. Furthermore, educators receive data on students' preparedness early in the semester, allowing for them to adapt their courses to best meet student needs. In order to generate informative reports for faculty each term, CADS maps its assessment questions to student information, to CS knowledge topics, to departmental student learning outcomes, and to taxonomic mastery levels. These reports visualize current data against historical data and present deeper examinations of significant data shifts for causal analysis. Different diagrams give insight into specific courses, while others examine student learning across the entire CS program via diverse mappings. Overall, CADS provides a model for using technology to craft an expedient form of program assessment that dynamizes a curriculum. This paper details our CADS implementation.

Session 6F: Computational Thinking Assessment

A Principled Approach to Designing a Computational Thinking Practices Assessment for Early Grades

Authors: Satabdi Basu; Daisy Rutstein; Yuning Xu; Linda Shear


In today's increasingly digital world, it is critical that all students learn to think computationally from an early age. Assessments of Computational Thinking (CT) are essential for capturing information about student learning and challenges. Several existing K-12 CT assessments focus on concepts like variables, iterations and conditionals without emphasizing practices like algorithmic thinking, reusing and remixing, and debugging. In this paper, we discuss the development of and results from a validated CT Practices assessment for 4th-6th grade students. The assessment tasks are multilingual, shifting the focus to CT practices, and making the assessment useful for students using different CS curricula and different programming languages. Results from an implementation of the assessment with about 15000 upper elementary students in Hong Kong indicate challenges with algorithm comparison given constraints, deciding when code can be reused, and choosing debugging test cases. These results point to the utility of our assessment as a curricular tool and the need for emphasizing CT practices in future curricular initiatives and teacher professional development.

Understanding Students' Computational Thinking through Cognitive Interviews: A Learning Trajectory-based Analysis

Authors: Feiya Luo; Maya Israel; Ruohan Liu; Wei Yan; Brian Gane; John Hampton


For K-8 computer science (CS) education to continue to expand, it is essential that we understand how students develop and demonstrate computational thinking (CT). One approach to gaining this insight is by having students articulate their understanding of CT through cognitive interviews. This study presents findings of a cognitive interview study with 13 fourth-grade students (who had previously engaged in integrated CT and mathematics instruction) working on CT assessment items. The items assessed four CT concepts: sequence, repetition, conditionals, and decomposition. This study analyzed students articulated understanding of the four CT concepts and the correspondence between that understanding and hypothesized learning trajectories (LTs). We found that 1) all students articulated an understanding of sequence that matched the intermediate level of the Sequence LT; 2) a majority of students responses demonstrated the level of understanding that the repetition and decomposition items were designed to solicit (8 of 9 responses were correct for repetition and 4 of 6 were correct for decomposition); and 3) less than half of students responses articulated an understanding of conditionals that was intended by the items (4 of 9 responses were correct). The results also suggested questioning the directional relationships of two statements in the existing Conditionals LT. For example, unlike the LT, this study revealed that students could understand "A conditional connects a condition to an outcome" before "A condition is something that can be true or false."

Development and Validation of the Computational Thinking Concepts and Skills Test

Authors: Markeya Peteranetz; Patrick Morrow; Leen-Kiat Soh


Calls for standardized and validated measures of computational thinking have been made repeatedly in recent years. Still, few such tests have been created and even fewer have undergone rigorous psychometric evaluation and been made available to researchers. The purpose of this study is to report our work in developing and validating a test of computational thinking concepts and skills and to compare different scoring methods for the test. This computational thinking exam is intended to be used in computing education research as a common measure of computational thinking so that the research community will be able to make more meaningful comparisons across samples and studies. The Computational Thinking Concepts and Skills Test (CTCAST) was administered to students in several courses, evaluated and revised, and then administered to another group of students. Part of the revision included changing half of the items to a multiple-select format. The test scores using the three scoring methods were compared to each other and to scores on a different test of core computer science knowledge. Results indicate the CTCAST and the test of core computer science knowledge measure similar, but not identical, aspects of students' knowledge and skills, and that item-level statistics vary according to the scoring method that is used. Recommendations for using and scoring the test are presented.

Session 6G: Systems

C + Go = An Alternate Approach Toward the Linux Programming Course

Authors: Saverio Perugini; Brandon M. Williams


The use of the C programming language in a Linux programming course---common in most undergraduate computer science programs---has been the standard practice for nearly thirty years. The use of C is appropriate because Linux is written in C and, thus, programming with the system (i.e., accessing operating system structures and making calls to the kernel) is natural in C. However, this seamless integration of Linux and C can inhibit student assimilation of course concepts---through conflation of concepts with language (e.g., system calls with C, or libraries with gcc, respectively)---and, ultimately, student learning outcomes for a variety of reasons. We challenge the idea of the exclusive use of C in the Linux programming course and alternatively propose the use of the Go programming language, in strategic conjunction with C, to both achieve student learning outcomes and address some of the issues with an exclusive C approach. We explored and studied this approach---the use of Go in the Linux course---over the course of seven consecutive offerings of it. We present our experience with this approach including a collection of desiderata resulting from it as well as student survey data as an evaluation of its use in practice. Overall, the results indicate this approach is feasible, is no worse than an exclusive C approach, and yields advantages. We anticipate this experience report will inspire adoption of the use of Go in similar Linux programming courses.

Exploring Students' Understanding of Concurrency - A Phenomenographic Study

Authors: Filip Strömbäck; Linda Mannila; Mariam Kamkar


This paper continues previous efforts in understanding the problems students face when learning concurrency. In this paper, we explore students' understanding of the subject using phenomenography in order to gain insights that can aid in explaining the underlying causes for common student mistakes in concurrency, which has been studied in depth previously. Students' experience of concurrency and critical sections were analyzed using a phenomenographic study based on interviews with students attending one of two courses on concurrency and operating systems. We present 6 categories describing students' experience of concurrency, and 4 categories describing students' experience of critical sections in this paper. Furthermore, these categories are related to previous results, both to explore how misconceptions in the categores relate to student mistakes and to estimate how common it is for each category to be discerned.

Using an Operating Systems Class to Strengthen Students' Knowledge of C++

Authors: Reva Freedman


Operating systems is the last required course in our undergraduate Computer Science major. Although our students have had two semesters of programming in C++, one in assembler, and a semester of data structures using the C++ Standard Template Library, many of them have weak mastery of some important programming topics. In addition, they have had minimal exposure to using these constructs with real world data structures. In this paper we describe how we have used the assignments in a Linux-based Operating Systems course to remedy these deficiencies while teaching a full semester of OS content. Assignments include retrieving data via standard API calls and from /proc; process management, including fork, wait and system; building a microshell with fork/exec and pipe/dup; semaphore handling; and building an OS simulator. We have rewritten these assignments to provide additional practice with data type conversion, pointers and memory management, file handling, classes and subclasses, and the use of vectors, maps and other data structures in the STL.

Session 7A: Professional Development A

Upper Elementary and Middle Grade Teachers' Perceptions, Concerns, and Goals for Integrating CS into Classrooms

Authors: Mehmet Celepkolu; Erin O'Halloran; Kristy Boyer


As efforts to integrate computer science into K-8 teaching in the US are dramatically rising, professional development workshops for teachers are becoming widespread. An open challenge for the CS education community is to understand teachers' needs and develop empirically grounded best practices for professional development. This experience report describes a five-day professional development workshop in which 22 third through eighth grade teachers learned about fundamental CS concepts, practiced coding, and created lesson plans for integrating what they learned into their classroom. We describe the professional development workshop including its modules and sequencing, and present teachers' perception of CS and how to integrate it into their classrooms. Teachers achieved significant gains in technical knowledge and improvements in attitude toward computer science. In initial focus groups, teachers reported that limited exposure to CS, time constraints, and lack of understanding of CS are barriers to integrating it into their classrooms. After the workshop, focus group feedback indicated that the workshop provided teachers a clearer sense of the potential of CS to enhance their classroom plans. Teachers noted that they felt able to use CS to help students learn critical thinking, prepare them for their futures, and address their individual needs. The results of this experience can inform future workshops that address the needs of teachers and students.

Code, Connect, Create: The 3C Professional Development Model to Support Computational Thinking Infusion

Authors: Robin Jocius; Deepti Joshi; Yihuan Dong; Richard Robinson; Veronica Cateté; Tiffany Barnes; Jennifer Albert; Ashley Andrews; Nicholas Lytle


Despite the increasing attention to infusing CT into middle and high school content area classrooms, there is a lack of information about the most effective practices and models to support teachers in their efforts to integrate disciplinary content and CT principles. To address this need, this paper proposes the Code, Connect and Create (3C) professional development (PD) model, which was designed to support middle and high school content area teachers in infusing computational thinking into their classrooms. To evaluate the model, we analyzed quantitative and qualitative data collected from Infusing Computing PD workshops designed for in-service science, math, English language arts, and social studies teachers located in two Southeastern states. Drawing on findings from our analysis of teacher-created learning segments, surveys, and interviews, we argue that the 3C professional development model supported shifts in teacher understandings of the role of computational thinking in content area classrooms, as well as their self-efficacy and beliefs regarding CT integration into disciplinary content. We conclude by offering implications for the use of this model to increase teacher and student access to computational thinking practices in middle and high school classrooms.

Critical Incidents in K-12 Computer Science Classrooms - Towards Vignettes for Computer Science Teacher Training

Authors: Ursula Pieper; Jan Vahrenhold


The use of vignettes has been shown to be an effective method both in teacher training and in the assessment of teachers' understanding of instructional strategies. In this paper, we report on a study in which we sought to identify real-world classroom situation that can be used as vignettes for K-12 Computer Science teacher education and professional development.

Using semi-structured interviews with five expert teachers involved in in-service teacher training, we elicited a collection of twelve domain-specific critical incidents that can occur in K-12 Computer Science classrooms. These critical incidents were ranked by the experts with respect to their relevance using a Delphi process. As part of the semi-structured interviews, we also collected reactions to each of the critical incidents. We discuss the resulting critical incidents and the experts' comments in the light of the Computing Education Research literature.

Session 7B: Computational Thinking B

The Role of Evidence Centered Design and Participatory Design in a Playful Assessment for Computational Thinking About Data

Authors: Satabdi Basu; Betsy Disalvo; Daisy Rutstein; Yuning Xu; Jeremy Roschelle; Nathan Holbert


The K-12 CS Framework provides guidance on what concepts and practices students are expected to know and demonstrate within different grade bands. For these guidelines to be useful in CS education, a critical next step is to translate the guidelines to explicit learning targets and design aligned instructional tools and assessments. Our research and development goal in this paper is to design a playful, curriculum-neutral assessment aligned with the 'Data and Analysis' concept (grades 6-8) from the CS framework. Using Evidence Centered Design and Participatory Design, we present a set of assessment guidelines for assessing data and analysis, as well as a set of design considerations for integrating data and analysis across middle school curricula in CS and non-CS contexts. We outline these contributions, describe how they were applied to the development of a game-based formative assessment for data and analysis, and present preliminary findings on student understanding and challenges inferred from student gameplay.

Introduction to Computational Thinking: A New High School Curriculum using CodeWorld

Authors: Fernando Alegre; John Underwood; Juana Moreno; Mario Alegre


The Louisiana Department of Education partnered with the Gordon A. Cain Center at LSU to pilot a Computing High School Graduation Pathway. The first course in the pathway, Introduction to Computational Thinking (ICT), is designed to teach programming and reinforce mathematical practice skills of nine-grade students, with an emphasis on promoting higher order thinking. In 2017-18, about 200 students and five teachers participated in the pilot, in 2018-2019 the participation increased to 400 students, and in the current 2019-2020 year about 800 students in 11 schools are involved. After describing the course content and the teacher training, we briefly discuss the data we have collected in the last two years. The overall student reception of the course has been positive, but the course was categorized by most students as hard. The pre-post test content assessments show that students have learned not only the language, but also general principles of programming. Lessons learned during the pilot phase have motivated changes, such as emphasizing during Professional Development the need to provide timely feedback to students, provide detailed rubrics for the projects and reorganize the lessons to increase the initial engagement with the material. After two years of running pilots, the course is becoming student-centered, where most of the code and image samples provided in the lessons are based on code created by previous students.

Work in Progress Report: A STEM EcoSystem Approach to CS/CT for All in a Middle School

Authors: Lijuan Cao; Audrey Rorrer; David Pugalee; Mary Lou Maher; Mohsen Dorodchi; Dave Frye; Tiffany Barnes; Eric Wiebe


This project is a Research to Practice Partnership (RPP) between two middle schools and two universities. It focuses on investigating problems and on identifying solutions around increasing participation and interest in computer science (CS). We aim to do this by identifying, experimenting with, and fine-tuning methods to help students develop computational thinking (CT) skills. The research employs a STEM ecosystem model, which facilitates a support structure that aims to mitigate barriers and impact students as they progress in STEM areas. While this RPP is still a work in progress, we present data from the first year of our collaboration with one of the middle schools. While the research questions guiding this RPP are intended to be iterative and revised annually, year one data provides perspectives on (1) barriers to developing a STEM ecosystem that supports CS/CT for every student through integration into science, math, and language arts courses, (2) the factors or interventions needed for the development of a CS/CT focused ecosystem that supports everyone in the school, (3) the indicators of success for a CS/CT focused STEM ecosystem in a school, and (4) how the ecosystem prepares and engages all students for CS/CT work in high school. Year one data is discussed in terms of the STEM ecosystem framework and in how it will guide the next steps in this partnership. This project contributes to the understanding of how to prepare future generations for participation in a workforce where knowledge of the foundations of CS/CT is integral to success.

Session 7C: Instructional Support

Infrastructor: Flexible, No-Infrastructure Tools for Scaling CS

Authors: Daniel Barowy; William K. Jannen


Demand for computer science education has skyrocketed in the last decade. Although challenging everywhere, scaling up CS course capacities is especially painful at small, liberal arts colleges (SLACs). SLACs tend to have few instructors, few large-capacity classrooms, and little or no dedicated IT support staff. As CS enrollment growth continues to outpace the ability to hire instructional staff, maintaining the quality of the close, nurturing learning environment that SLACs advertise-and students expect-is a major challenge.

We present Infrastructor, a workflow and collection of course scaling tools that address the needs of resource-strapped CS departments. Infrastructor removes unnecessary administrative burdens so that instructors can focus on teaching and mentoring students. Unlike a traditional learning management system (LMS), which is complex, monolithic, and usually administered by a campus-wide IT staff, instructors deploy Infrastructor themselves and can trivially tailor the software to suit their own needs. Notably, Infrastructor does not require local hardware resources or platform-specific tools. Instead, Infrastructor is built on top of version control systems. This design choice lets instructors host courses on commodity, cloud-based repositories like GitHub. Since developing Infrastructor two years ago, we have successfully deployed it in ten sections of CS courses (323 students), and over the next year, we plan to more than double its use in our CS program.

Runestone: A Platform for Free, On-line, and Interactive Ebooks

Authors: Barbara Ericson; Bradley Miller


The Runestone platform is open-source, extensible, and serves free ebooks to over 25,000 learners a day from around the world. The site hosts 18 ebooks for computing courses. Some of these ebook have been translated into several languages. There are ebooks for secondary computer science (AP CSP and AP CSA), CS1, CS2, data science, and web programming courses. The platform currently supports executable and editable examples in Python, Java, C, C++, HTML, JavaScript, Processing, and SQL. Runestone provides features for instructors, learners, authors, and researchers.

Instructors can create a custom course from any of the existing ebooks and their students can register for that course. Instructors can create assignments from the existing material or author new problems, grade assignments, and visualize student progress. Learners can execute and modify examples and answer practice questions with immediate feedback. Runestone includes common practice types, such as multiple-choice questions, as well as some unique types, such as adaptive Parsons problems. Authors can modify the existing ebooks or write new ebooks using restructuredText: a markup language. Researchers can create and test new interactive features, run experiments, and analyze log file data.

This paper describes the architecture of the platform, highlights some of the unique features, provides an overview of how instructors use the platform, summarizes the research studies conducted on the platform, and describes plans for future development.

Teaching Quantum Computing through a Practical Software-driven Approach: Experience Report

Authors: Mariia Mykhailova; Krysta M. Svore


Quantum computing harnesses quantum laws of nature to enable new types of algorithms, not efficiently possible on traditional computers, that may lead to breakthroughs in crucial areas like materials science and chemistry. There is rapidly growing demand for a quantum workforce educated in the basics of quantum computing, in particular in quantum programming. However, there are few offerings for non-specialists and little information on best practices for training computer science and engineering students.

In this report we describe our experience teaching an undergraduate course on quantum computing using a practical, software-driven approach. We centered our course around teaching quantum algorithms through hands-on programming, reducing the significance of traditional written assignments and relying instead on self-paced programming exercises ("Quantum Katas"), a variety of programming assignments, and a final project. We observed that the programming sections of the course helped students internalize theoretical material presented during the lectures. In the survey results, students indicated that the programming exercises and the final project contributed the most to their learning process.

We describe the motivation for centering the course around quantum programming, discuss major artifacts used in this course, and present our lessons learned and best practices for a future improved course offering. We hope that our experience will help guide instructors who want to adopt a practical approach to teaching quantum computing and will enable more undergraduate programs to offer quantum programming as an elective.

Session 7D: Security B

KYPO4INDUSTRY: A Testbed for Teaching Cybersecurity of Industrial Control Systems

Authors: Pavel ?eleda; Jan Vykopal; Valdemar Švábenský; Karel Slaví?ek


There are different requirements on cybersecurity of industrial control systems and information technology systems. This fact exacerbates the global issue of hiring cybersecurity employees with relevant skills. In this paper, we present KYPO4INDUSTRY training facility and a course syllabus for beginner and intermediate computer science students to learn cybersecurity in a simulated industrial environment. The training facility is built using open-source hardware and software and provides reconfigurable modules of industrial control systems. The course uses a flipped classroom format with hands-on projects: the students create educational games that replicate real cyber attacks. Throughout the semester, they learn to understand the risks and gain capabilities to respond to cyber attacks that target industrial control systems. Our described experience from the design of the testbed and its usage can help any educator interested in teaching cybersecurity of cyber-physical systems.

A System for Visualizing the Process Address Space in the Context of Teaching Secure Coding in C

Authors: James Walker; Man Wang; Steven Carr; Jean Mayo; Ching-Kuang Shene


Seemingly small coding errors can create significant vulnerabilities in C programs. This often occurs due to memory being overwritten in unexpected ways. If a student understands where program variables appear in the process address space, then she can understand the effect of writing beyond the memory allocated to a variable. With this understanding, she can tie her code to its effect within an executing process and is more likely to appreciate the significance of these seemingly harmless errors and to avoid them.

We have developed a program analysis and visualization tool to help students understand the impact of common memory errors with the goal to help students avoid introducing these errors into their code. The visualization is through the Program Address Space (PAS) window within a larger system for analysis and visualization of security issues in C programs. The larger system is called SecureCvisual.

In this paper, we describe our experience with teaching students fundamental concepts about process address spaces and the impact of buffer overflows using the PAS window. We also present the results from an evaluation of the tool. Our results indicate that students found the tool useful and that it enhanced the course in which it was used.

A Hands-On Cybersecurity Curriculum Using a Robotics Platform

Authors: Bernard Yett; Nicole Hutchins; Gordon Stein; Hamid Zare; Caitlin Snyder; Gautam Biswas; Mary Metelko; Ãkos Lédeczi


This paper presents a study using a robotics platform for teaching computing and cybersecurity concepts to high school students. 38 students attended a week-long camp, starting with projects such as a simulation-only game and a simple autonomous driving program for the robots in order to learn and apply computational thinking (CT) and networking skills. They were then assigned a series of challenges that required developing progressively more advanced cybersecurity measures to protect their robots. This culminated in a final challenge that required implementing defensive measures such as encryption, secure key exchange and sequence numbers. We used an evidence-centered design framework to construct rubrics for grading student work. The pre- and post-test results show that the interventions helped students learn cybersecurity and CT concepts, but they had difficulties with networking concepts. These results correlate with scores from the game and the final challenge. Overall, surveys show that the competition-based robotics learning framework engaged students and supported their overall learning, but our intervention needs to be modified to help students learn networking concepts

Session 7E: Python Debugging

PABLO: Helping Novices Debug Python Code Through Data-Driven Fault Localization

Authors: Benjamin Cosman; Madeline Endres; Georgios Sakkas; Leon Medvinsky; Yao-Yuan Yang; Ranjit Jhala; Kamalika Chaudhuri; Westley Weimer


As dynamically-typed languages grow in popularity, especially among beginning programmers, there is an increased need to pinpoint their defects. Localization for novice bugs can be ambiguous: not all locations formally implicated are equally useful for beginners. We propose a scalable fault localization approach for dynamic languages that is helpful for debugging and generalizes to handle a wide variety of errors commonly faced by novice programmers. We base our approach on a combination of static, dynamic, and contextual features, guided by machine learning. We evaluate on over 980,000 diverse real user interactions across four years from the popular website, which is used both in classes and by non-traditional learners. We find that our approach is scalable, general, and quite accurate: up to 77% of these historical novice users would have been helped by our top-three responses, compared to 45% for the default interpreter. We also conducted a human study: participants preferred our approach to the baseline (p = 0.018), and found it additionally useful for bugs meriting multiple edits.

Tell Me What's Wrong: A Python IDE with Error Messages

Authors: Tobias Kohn; Bill Manaris


Development environments play a crucial role for novice programmers. Not only do they act as interface to type in and execute programs, but a programming environment is also responsible for reporting errors, managing in- and output when the program is running, or offering the programmer access to the underlying notional machine.

In recent years several new educational programming environments for Python have been presented. However, the important issue of reporting errors has rarely been addressed and evaluations often hint that students main issue is the poor quality of Python's error messages. We have therefore written an educational Python environment with enhanced error messages. This paper presents the design and rationale of its three primary features: modifications to Python, enhanced error messages, and the visual debugger.

Pedal: An Infrastructure for Automated Feedback Systems

Authors: Luke Gusukuma; Austin Cory Bart; Dennis Kafura


This paper describes Pedal, an innovative approach to the automated creation of feedback given to students in programming classes. Pedal is so named because it supports the PEDAgogical goals of instructors and is an expandable Library of components motivated by these goals. Pedal currently comes with components for type inferencing, flow analysis, pattern matching, and unit testing to provide an instructor with a rich set of resources to use in authoring and prioritizing feedback. The larger vision is the loosely-coupled architecture whose components can be readily expanded or replaced. The Pedal library components are motivated by a study of contemporary automated feedback systems and our own experience. Pedal's components are described and examples are given of Pedal-based feedback from three different introductory classes at two different universities. The integration of Pedal into several programming and autograding environments is briefly described.

Session 7F: CS1

Non-Major Peer Mentoring for CS1

Authors: Ryan Rybarczyk


One of the key challenges associated with Computer Science I (CS1) curricula is that of student satisfaction which directly translates to success and retention. This challenge is exacerbated at small institutions in which departmental enrollment is low overall. One area that has shown much promise is that in which peer mentors are provided to augment existing lecture and laboratory components in order to better assist the students. This approach has found success at both large and small institutions where student demographics and populations are widely varied. In most implementations, the peer mentor is a departmental student (CS major) who has previously taken and excelled in the course. This does not however, account for those students enrolled in the course as non-majors. At many institutions, the typical CS1 course may count as an elective course necessary for graduation or to fulfill another majors requirements. In this situation, non-major students are often cited as feeling excluded, which then becomes the reason the student does not seek the necessary help that is required to find success within the course. In this work, we propose a non-major peer mentoring implementation to augment existing departmental-level tutoring in order to better assist all student backgrounds. We provide the feedback from the students that motivated this change, as well as the proposed infrastructure necessary for such an implementation to find success. We conclude this paper with analysis of the impact this implementation has had on non-major students in a small liberal arts CS1 course.

Better Late Than Never: Exploring Students' Pathways to Computing in Later Stages of College

Authors: Kathleen Lehman; Annie Wofford; Michelle Sendowski; Kaitlin N. S. Newhouse; Linda Sax


While introductory computing courses (i.e., CS1) continue to enroll large proportions of first- and second-year computing majors, more non-majors than ever before are enrolling in CS1 at later points in their college careers. These "latecomers" to computing are a unique population that may be more diverse, given that prior research indicates students from historically underrepresented groups often develop an interest in computing later than White and Asian men. Thus, as computing departments grapple with the pressures to broaden participation, latecomers to CS1 may be a population from which to recruit women and underrepresented students of color. Using survey data from a nationwide study of CS1 course takers, this study explores the demographic and academic background characteristics of students who took CS1 in their third year or beyond. We also analyzed the characteristics and CS1 experiences that predict latecomers' decisions to major or minor in computing. Findings indicate that latecomers came from a variety of academic disciplines and were more likely than computing majors who take CS1 courses earlier in college to be women. However, latecomers who were women were less likely than men to be recruited to a computing major or minor. Additional findings and implications for practice are discussed.

Investigating the Impact of Employing Multiple Interventions in a CS1 Course

Authors: Sophia Krause-Levy; Leo Porter; Beth Simon; Christine Alvarado


Given the long-standing concern about students failing introductory programming courses, there is a need for interventions that may aid those students. In this work, we examine the potential benefit of three interventions based on prior computing education research (CER) or STEM education research literature: mindset interventions, the use of "Thinkathons" as an alternative to programming labs, and metacognitive interventions to encourage more productive study habits. We conducted an in-class study that controlled for both time-on-task and selection bias to investigate the potential benefits of integrating these interventions into the existing footprint of an introductory computing course. Despite the previously reported promise of the interventions we implemented, our findings were that in this context these techniques had only a mild positive effect for some students. We discuss possible reasons why these techniques are less successful than instructors might hope and argue for the need for more research on this topic.

Session 7G: Professional Skills

Developing Students' Written Communication Skills with Jupyter Notebooks

Authors: Alistair Willis; Patricia Charlton; Tony Hirst


Written communication skills are considered to be highly desirable in computing graduates. However, many computing students do not have a background in which these skills have been developed, and the skills are often not well addressed within a computing curriculum. For some multidisciplinary areas, such as data science, the range of potential stakeholders makes the need for communications skills all the greater. As interest in data science increases and the technical skills of the area are in ever higher demand, understanding effective teaching and learning of these interdisciplinary aspects is receiving significant attention by academics, industry and government in an effort to address the digital skills gap.

In this paper, we report on the experience of adapting a final year data science module in an undergraduate computing curriculum to help develop the skills needed for writing extended reports. From its inception, the module has used Jupyter notebooks to develop the students' skills in the coding aspects of the module. However, over several presentations, we have investigated how the cell-based structure of the notebooks can be exploited to improve the students' understanding of how to structure a report on a data investigation. We have increasingly designed the assessment for the module to take advantage of the learning affordances of Jupyter notebooks to support both raw data analysis and effective report writing.

We reflect on the lessons learned from these changes to the assessment model, and the students' responses to the changes.

Non-cognitive Abilities of Exceptional Software Engineers: A Delphi Study

Authors: Wouter Groeneveld; Hans Jacobs; Joost Vennekens; Kris Aerts


Important building blocks of software engineering concepts are without a doubt technical. During the last decade, research and practical interest for non-technicalities has grown, revealing the building blocks to be various skills and abilities beside pure technical knowledge. Multiple attempts to categorise these blocks have been made, but so far little international studies have been performed that identify skills by asking experts from both the industrial and academic world: which abilities are needed for a developer to excel in the software engineering industry? To answer this question, we performed a Delphi study, inviting 36 experts from 11 different countries world-wide, affiliated with 21 internationally renowned institutions. This study presents the 55 identified and ranked skills as classified in four major areas: communicative skills (empathy, actively listening, etc.), collaborative skills (sharing responsibility, learning from each other, etc.), problem solving skills (verifying assumptions, solution-oriented thinking, etc.), and personal skills (curiosity, being open to ideas, etc.), of which a comparison has been made between opinions of technical experts, business experts, and academics. We hope this work inspires educators and practitioners to adjust their training programs, mitigating the gap between the industry and the academic world.

Exploring the Participation of CS Undergraduate Students in Industry Internships

Authors: Amanpreet Kapoor; Christina Gardner-McCune


Industry internships offer CS students an opportunity to gain authentic disciplinary experiences, evaluate self-interests, and secure future employment. However, little is empirically known about CS students' participation in industry internships and the preparation process used to successfully securing an internship. This paper presents findings from our multi-institutional study aimed at understanding the participation of CS students in industry internships as well as analyzing the differences between students who intern and those who do not. We surveyed 536 CS undergraduate students across three universities in the United States and analyzed the quantitative data using descriptive and inferential statistical methods. We used thematic analysis on the open-ended survey responses. Overall, we found that 40% of students participate in at least one internship. Demographically, equal proportions of males and females interned. However, we observed that students who have higher socioeconomic status were more likely to intern. Academically, there were no significant differences between students who intern and those who do not. However, through thematic analysis, we found differences regarding students' preparation process. Interns explicitly prepared to secure internship positions by practicing interview questions and dedicating time to career preparation. Students who do not intern were less involved in the application process or relied on coursework for securing internships. Quantitative results from the survey corroborated our qualitative findings that factors outside of coursework are influencing students' ability to secure industry internships.

Session 8A: Professional Development B

Evaluating a Systems Approach to District CS Education Implementation

Authors: Leigh Ann DeLyser; Stephanie Wortel-London; Lauren Wright; Anisa Bora


Computer science education exists within a broader education environment than individual classrooms. Formal, or during the school day, K-12 education requires implementation within schools and local education agencies with leadership structures, budgets, and often competing priorities. In this experience report we share outcomes from a series of district workshops for supporting critical systems planning needed to ensure the integration and sustainability of computer science education within K-12 systems. The paper will briefly discuss the program's theory of change, the actions CSforALL took to deliver the program at scale, and data with short and long term outcomes from program implementation. Discussion will include the intersection of program and curricular and professional development and the critical frame of continuous improvement as a theoretical model for sustainability and rigor in computer science education implementation.

A Pre-service Pathway for Preparing Future AP CS Principles Teachers

Authors: Jeff Gray; Rebecca Odom-Bartel; Jeremy Zelkowski; Karl Hamner; Sierra Rodgers-Farris


The surge of interest in K-12 computer science (CS) over the past decade has led to a deep need for a corresponding expansion of trained teachers. The primary focus of most K-12 CS teacher professional development has been for current in-service teachers who have little background in CS. To raise the importance of CS within Colleges of Education, we believe that new pathways and experiences are needed for pre-service Education majors to learn more about authentic CS topics and pedagogy. This experience report summarizes our efforts over the past two years to prepare Secondary Math Education (SEMA) majors to teach AP CS Principles (AP CSP). Our approach consists of the following curricular activities: 1) a two-course sequence, with the first course mapping to the content topics of the AP CSP Curriculum Framework, and the second course consisting of a reflection of CS methods and pedagogy, including opportunities for SEMA students to develop and present their own AP CSP lesson plans; 2) opportunities for SEMA students to observe AP CSP classrooms in local high schools through our partnership with experienced AP CSP teachers; 3) summer participation in a College Board AP Summer Institute for AP CSP, and 4) a six-week ETS Praxis CS preparation modules-based course, offered to both pre-service SEMA students and in-service teachers. We summarize our lessons learned and present results that suggest our approach is preparing pre-service students with pedagogical and content knowledge that meets or exceeds current in-service training models (including an analysis of recent Praxis results for CS certification in our state).

Project Scoring for Program Evaluation and Teacher Professional Development

Authors: Yvonne Kao; Irene Nolan; Andrew Rothman


Evaluating student learning in K-12 computer science programs is challenging due to the limited number of validated assessments, especially below the high school level. Because computer science is often taught as an engineering-oriented discipline in K-12, it seems natural to use students' programming projects as evidence of student learning. However, this approach is time-consuming and can overestimate what students understand. In this paper, we share our experience with using students' programming projects as a tool for evaluating a middle school computer science program at San Francisco Unified School District. The scoring and analysis of programming projects did generate useful insights about the nature of student learning in the program. More importantly, we found that the process of co-developing a scoring rubric and scoring projects alongside CS classroom teachers generated rich discussions and knowledge-sharing about the current state of the district's CS program, its values, and its long-term goals.

Session 8B: Grading

Auto-Grading Jupyter Notebooks

Authors: Hamza Manzoor; Amit Naik; Clifford A. Shaffer; Chris North; Stephen H. Edwards


Jupyter Notebooks are becoming more widely used, both for data science applications and as a convenient environment for learning Python. Currently, grading of assignments done in Jupyter Notebooks is typically done manually. Manual grading results in students receiving feedback only long after the assignment is complete. We implemented support for auto-grading programs written in Jupyter Notebooks within the Web-CAT auto-grading system. Scores received are directly reported to the Canvas gradebook. A Jupyter notebook extension allows students to upload their notebook files to Web-CAT directly. Survey results from class use show that 80% of students believe that getting immediate feedback from Web-CAT improved their performance. Instructors report that this implementation has significantly reduced their workload.

Autograding Interactive Computer Graphics Applications

Authors: Evan Maicus; Matthew Peveler; Andrew Aikens; Barbara Cutler


We present a system for the automated testing and grading of computer graphics applications. Our system runs, provides input to, and captures image and video output from graphical programming assignments. Instructors use a simple set of commands to script automated keyboard and mouse interactions with student programs at fixed times during execution. The resultant output, including plaintext standard output and mid-execution screenshots and GIFs, are displayed to the student to aid in debugging and ensure compliance with assignment specifications. Student output is automatically evaluated by basic text and image difference operations, or via an instructor-written validation method.

We evaluate the success, implementation, and robustness of our design through deployment of this work in our university's senior undergraduate/graduate computer graphics course. In this course, students implement a variety of graphical assignments using OpenGL in C++. We summarize student feedback about the system gathered from anonymous end-of-term course evaluations. We provide anecdotal and quantitative evidence that the system improves student experience and learning by clarifying instructor expectations, building student confidence, and improving the consistency and efficiency of manual grading.

This research has been implemented as an extension to Submitty, an open source, language-agnostic course management platform which allows automated testing and automated grading of student programming assignments. Submitty supports all levels of courses, from introductory to advanced special topics, and includes features for manual grading by TAs, version control, team submission, discussion forums, and plagiarism detection.

Scored out of 10: Experiences with Binary Grading Across the Curriculum

Authors: Andrew Berns


Perhaps no other task has frustrated us as instructors as much as grading. While we recognize the importance of providing meaningful feedback and accurate assessment, our traditional grading had quickly become a task of administrative minutiae instead of helpful coaching. Like many faculty, we have searched for ways to reduce our grading load without harming student learning. We have tried a variety of approaches, from modifying our existing assignments all the way to rethinking our entire course design. The past few years we have been refining a \emph{binary grading system}: a set of grading principles based upon the practice of scoring student assessments simply as either satisfactory or unsatisfactory ($10_2$ possible scores, thus the paper title). We believe this approach has significantly improved our students' attitudes towards our courses and holds promise as a way to significantly reduce the time we spend grading. In this paper, we describe our experiences with our binary grading system in several undergraduate computer science courses. We discuss the general practices of our binary grading, how we have implemented it in our courses, and what our experiences have taught us regarding the future refinement of our grading practices. We hope our experiences will spur discussion and assist other computer science educators in improving their assessment process.

Session 8C: Learning Tools

User-Centered Design of a Mobile Java Practice App: A Comparison of Question Formats

Authors: Mohona Ahmed; Kimberly Ying; Kristy Boyer


Learning computer science presents many challenges to students, and providing resources for meaningful practice is recognized as a way to support rigorous learning and diverse student participation. At the same time, mobile phones are increasingly ubiquitous, creating an underutilized opportunity for practice outside of traditional methods. This experience report presents a user-centered approach to designing a practice app for introductory Java. We investigated user preferences through a series of small studies, first conducting think-aloud sessions and focus groups, and finally conducting a usability study comparing two prototype versions. The initial studies suggested how to leverage the affordances of small screens, ruling out free-response practice problems in favor of either fill-in-the-blank (FB) or multiple-choice (MC) questions. The comparison study revealed statistically significant differences in students' survey responses: (1) usability scores were significantly higher for the MC version than the FB version; (2) students reported significantly greater satisfaction and desire to learn for the MC version; and (3) students reported enjoying and being more comfortable with the MC version compared to the FB version. We contextualize this observation within related research on question formats. Takeaways from this experience report can provide guidance on designing mobile applications that give students opportunities for meaningful practice.

How Important is Immersion for Learning in Computer Science Replugged Games?

Authors: Andreas Dengel


By following the idea of not using computers at all, Computer Science Unplugged has set the course for many interactive, social, and hands-on activities dealing with concepts and problems of Computer Science Education. Through recent developments in immersive media, technologies like Virtual and Augmented Reality could enhance such activities or even enable new ones. When providing immersive educational media that induce a sense of presence in the virtual environment, the illusion of unmediated learning experience can be delivered. Hence, the concept of "Computer Science Replugged" can benefit from the affordances that Computer Science Unplugged thrives on while facilitating or enabling activities that might be impossible, dangerous, or expensive to carry out in reality. This paper presents three concepts from Computer Science Education that have been modeled as 3-D immersive educational virtual environments: components of a computer, asymmetric encryption/decryption, and finite state machines. To get a first impression of the effectiveness of these approaches and in order to determine the importance of the level of immersion for the learning process, a study with 78 participants was conducted in which the software was tested on different devices. All activities were found to be significantly effective with regards to the pre- and post-tests. When analyzing these results on the basis of comparing the least immersive setting (laptop) with the most immersive setting (head-mounted-display), the findings indicate different effects with effect sizes between beta = -.17 and beta</i> = .41. This raises two questions: "Which topics from Computer Science Education can benefit from immersive technology?" and "What are the opportunities and challenges of the didactical design of Computer Science Replugged activities?" that have to be adressed in further research. </p>

Student Adoption and Perceptions of a Web Integrated Development Environment: An Experience Report

Authors: Martin Valez; Michael Yen; Mathew Le; Zhendong Su; Mohammad Amin Alipour


Students often spend a considerable amount of time and effort installing and configuring programming tools and environments. This can frustrate, and distract them from more important learning objectives, particularly in introductory programming courses. A web-based integrated development environment can serve as a low-threshold, ready-to-use programming environment, and reduce the time and effort needed to start practicing programming.

In this paper, we report our experience of developing and deploying a web-based integrated development environment (web IDE) as an optional tool at a large public university that has been in use for over several years in various programming courses.

We conducted a survey to understand students' perceptions toward the web IDE and usage of its features. Using the data from the survey, we explored potential correlations between student demographic and behavioral traits in adoption of the web IDE. The results of the survey suggest that around half of the students use the IDE often or very often. We also discovered that the likelihood of adoption of the IDE decreases as students to move to upper classes. In this paper, we also describe broader lessons for educators and researchers.

Session 8D: Online

Improving Student Learning Outcomes in Online Courses: An Investigation Into the Effects of Multiple Teaching Modalities

Authors: Julio César Bahamón; Audrey Rorrer


We investigated learning outcomes within an undergraduate C Programming course that is taught in multiple modalities: in-person, online and blended. Because this course has been taught by the same instructor, using the same scaffolding activities, materials and approaches, we were uniquely positioned to conduct a quasi-experimental study of learning outcomes between courses and within students. The overarching goal was to glean knowledge and implications about assessment practices for undergraduate courses that are taught in multiple modalities. The objectives of our research are primarily to discern what differential impacts, if any, are found between the in-person and the online course delivery. We aimed to discover learning outcome patterns among the students who participate in these modalities. Findings from this study provide valuable information for undergraduate Computer Science programs by identifying any differential learning outcomes that students experience between in-person and online course instruction. The research questions addressed by the study were as follows: 1) What impact does modality have on student learning outcomes? 2) What patterns are discernable across student groups? 3) What relationship is there between final course grades and assignment module learning outcomes? In earlier work, we were surprised that no significant differences were obtained between course modality. While this was an encouraging finding, we believed that further data collection and analysis were needed, before making general conclusions about the two modalities. This paper reports on our efforts to collect additional data, while considering additional variables, such as instructor, multiple modalities, and online course design approaches.

Incorporating Active Learning Strategies and Instructor Presence into an Online Discrete Mathematics Class

Authors: Sandy Irani; Kameryn Denaro


Online education offers an attractive alternative to face-to-face classes by providing flexibility to students and efficiencies for educational institutions. Leveraging online technology has the potential to help computer science departments offer a quality educational experience in the face of burgeoning enrollments. However, effective online course design is critical to student satisfaction and learning outcomes. In this paper, we describe the experience of converting a large face-to-face course in Discrete Mathematics to an online format. Particular care was taken to incorporate active learning strategies, such as clicker questions and interactive discussions, in order to enhance student engagement. We describe ways in which we cultivated an active instructor presence in the course through carefully designed pre-recorded videos, online video conferencing, and participation in Piazza, an online social learning platform.

In-class tests were specifically designed to provide a meaningful comparison of learning outcomes between a face-to-face and online offering of the course taught by the same instructor. The results indicate that there is no loss in student performance in the online course, even after accounting for demographic and academic differences between the students enrolled in the two courses. There is also no significant difference in performance for "at-risk" students. End-of-quarter student evaluations show a high level of student satisfaction with the online format, especially in regards to the opportunities to have questions answered and the positive presence of the instructor in the course.

Who Benefits? Positive Learner Outcomes from Behavioral Analytics of Online Lecture Video Viewing Using ClassTranscribe

Authors: Lawrence Angrave; Zhilin Zhang; Genevieve Henricks; Chirantan Mahipal


Lecture material of a sophomore large-enrollment (N=271) system programming 15-week class was delivered solely online using a new video-based web platform. The platform provided accurate accessible transcriptions and captioning plus a custom text-searchable interface to rapidly find relevant video moments from the entire course. The system logged student searching and viewing behaviors as fine-grained web browser interaction events including full-screen-switching, loss-of-focus, incremental searching events, and continued-video-watching events with the latter at 15-second granularity. Student learning behaviors and findings from three research questions are presented using individual-level performance and interaction data. Firstly, we report on learning outcomes from alternative learning paths that arise from the course's application of Universal Design for Learning principles. Secondly, final exam performance was equal or better to prior semesters that utilized traditional in-person live lectures. Thirdly, learning outcomes of low and high performing students were analyzed independently by grouping students into four quartiles based on their non-final-exam course performance of programming assignments and quizzes. We introduce and justify an empirically-defined qualification threshold for sufficient video minutes viewed for each group. In all quartiles, students who watched an above-threshold of video minutes improved their in-group final exam performance (ranging from +6 % to +14%) with the largest gain for the lowest-performing quartile. The improvement was similar in magnitude for all groups when expressed as a fraction of unrewarded final exam points. Overall, the study presents and evaluates how learner use of online video using ClassTranscribe predicts course performance and positive learning outcomes.

Session 8E: Data Science B

Collective Development of Large Scale Data Science Products via Modularized Assignments: An Experience Report

Authors: Bhavya -; Assma Boughoula; Aaron Green; ChengXiang Zhai


Many universities are offering data science (DS) courses to fulfill the growing demands for skilled DS practitioners. Assignments and projects are essential parts of the DS curriculum as they enable students to gain hands-on experience in real-world DS tasks. However, most current assignments and projects are lacking in at least one of two ways: 1) they do not comprehensively teach all the steps involved in the complete workflow of DS projects; 2) students work on separate problems individually or in small teams, limiting the scale and impact of their solutions. To overcome these limitations, we envision novel synergistic modular assignments where a large number of students work collectively on all the tasks required to develop a large-scale DS product. The resulting product can be continuously improved with students' contributions every semester.

We report our experience with developing and deploying such an assignment in an Information Retrieval course. Through the assignment, students collectively developed a search engine for finding expert faculty specializing in a given field. This shows the utility of such assignments both for teaching useful DS skills and driving innovation and research. We share useful lessons for other instructors to adopt similar assignments for their DS courses.

Cross-Disciplinary Faculty Development in Data Science Principles for Classroom Integration

Authors: Brandeis Marshall; Susan Geier


Data science in practice leverages the expertise in computer science, mathematics and statistics with applications in any field using data. The formalization of data science educational and pedagogical strategic remain in their infancy. College faculty from various disciplines are tasked with designing and delivering data science instruction without the formal knowledge of how data science principles are executed in practice. We call this the data science instruction gap. Also, these faculties are implementing their discipline's standard pedagogical strategies to their understanding of data science. In this paper, we present our cross-disciplinary instructional program model designed to narrow the data science instruction gap for faculty. It is designed to scaffold college faculties' data science learning to support their discipline-specific data science instruction. We provide individualized and group-based support structures to instill data science principles and transition them from learners to educators in data science. Lastly, we share our model's impact on and value to faculty as well as make recommendations for model adoption.

Expanding Undergraduate Exposure to Computer Science Subfields: Resources and Lessons from a Hands-on Computational Biology Workshop

Authors: Layla Oesper; Anya Vostinar


Computational biology is an exciting and ever-widening interdisciplinary field. Expanding the participation of undergraduate students in this field will help to inspire and train the next generation of scientists necessary to support this growing area. However, students at smaller institutions, such as those focused on undergraduate education, may not have access to courses related to or even faculty interested in computational biology. Providing more opportunities for such undergraduate students to be exposed to computational biology, or other subfields within computer science, will be important for ensuring these students are included in the pipeline of scientists contributing to these diverse fields. To this end, we hosted a computational biology workshop that brought together undergraduate students from three different liberal arts colleges. The goal of the workshop was to provide an introduction to how computer science can be used to help answer important problems in biology. A diverse set of six faculty members from different institutions each created and taught a hands-on module as an introduction to a different area of computational biology at the workshop. We describe how we went about organizing this undergraduate workshop, summarize the workshop materials that are freely available, and discuss the outcomes and lessons learned from the workshop. We further propose that the workshop structure used is adaptable to other subfields of computer science. Workshop materials available at the workshop website:

Session 8F: K12 Teaching

Teaching Elementary Computer Science through Universal Design for Learning

Authors: Maya Israel; Gakyung Jeong; Meg Ray; Todd Alan Lash


Given the academic diversity of today's classrooms, elementary teachers engaged in computer science (CS) and computational thinking (CT) instruction must create CS/CT experiences that are accessible and engaging to a broad range of learners, including those with disabilities. One method of developing inclusive instructional experiences is through the Universal Design for Learning (UDL) framework, wherein teachers proactively design instruction for the broadest range of learners. Doing so may be challenging as elementary teachers may not be familiar with the UDL framework or may not have experience with applying UDL within CS/CT instruction. The purpose of this qualitative study was to investigate how four elementary teachers provided UDL-based instruction to academically diverse learners during CS/CT instruction. Teachers received professional development and instructional coaching related to UDL within CS/CT education. Data included teachers' lesson plans, coaching logs, and teacher interviews which were qualitatively analyzed and triangulated. Data revealed that teachers generally addressed all three UDL principles, with an emphasis on two of the principles (multiple means of engagement and multiple means of representing content) above the third principle (multiple means of action and expression). They focused on breaking tasks into steps, emphasizing student choice, and presenting information in multiple ways. Findings revealed nuanced implementation differences among the teachers as well.

Design Studios for K-12 Computing Education

Authors: Megan Venn-Wycherley; Christine Bennett; Ahmed Kharrufa


Following the decline of pupil engagement in compulsory computing education (K-12) in the UK, advocates have called for further research into computing-specific pedagogies. Aiming for an improvement in pupil engagement, the subsequent report explores the viability of the applied Design Studio approach in K-12 computing education. We report on two case studies involving two 12-week curricula co-developed and delivered between researchers and teachers for four Year 8 classes in a secondary school in England. Common to both case studies were the design brief, desk crit, design review, presentation and portfolio elements proposed by Koutsabasis et al, with the key difference being a change in scope and structure of the challenge presented to pupils. Our findings demonstrate the need for a well-structured, personalizable and challenge-driven Design Studio approach, which showed evidence of improved pupil resilience and confidence, while engaging reluctant pupils. We conclude with a proposed model for applying Design Studios in K-12 computing education, to support educators aiming to adopt project-driven computing pedagogy in the classroom.

Impacts of Creating Smart Everyday Objects on Young Female Students' Programming Skills and Attitudes

Authors: Mazyar Seraj; Eva Katterfeldt; Serge Autexier; Rolf Drechsler


In computer programming education, learning to program tangible objects has become a common way to introduce programming to young students. In an effort to address this intervention, scientific research has been done on the effectiveness of using tangible hardware platforms such as robots and wearable products to teach basic programming concepts to children. However, there is a lack of research on how young students' attitudes and programming skills are influenced over time, when they learn to program tangible objects and make them smart. In this paper, we investigate the impacts of using a tangible everyday object and making it smart on young female students' attitudes towards programming and the acquisition of basic programming skills. During a 4-day non-formal programming workshop with 12 6th grade students, they were introduced to basic programming concepts, and learned how to apply them to turn a houseplant into a smart object. In a pilot study, we employed a block-based programming environment and analyzed the students' trajectories of attitudes towards programming and performance based on repeated open-ended qualitative questionnaires and programming questions throughout the workshop. The results show that all students had high confidence regarding programming skills, regardless of creating smart objects. Furthermore, it indicates that experienced students highly valued the programming of tangible everyday objects compared with inexperienced students. The findings of this work contribute to our understanding of how making tangible everyday objects smart can support the development of a positive attitude and keep up of interest throughout a programming workshop among girls.

Session 8G: Graduate Programs

Enrollment Motivations in an Online Graduate CS Program: Trends & Gender- and Age-Based Differences

Authors: Alex Duncan; Bobbie Eicher; David Joyner


Demand for CS education has risen, leading to numerous new programs, such as the rise of affordable online degrees. Research shows these programs meet an otherwise untapped audience of working professionals seeking graduate level CS education. In this study, we examine the motivations for enrollment among students in one such online MSCS program. Based responses to an open ended question, we develop a typology of motivations, including goals (e.g. career transition), opportunities (e.g. enrolling without taking time off work), and assurances that their goals will be met (e..g the program's accreditation). We then issue a closed survey question to a new group of students to further explore these motivations. In this paper, we discuss both aggregate and demographic trends in motivations, including the different motivations of men and women and what they imply about the program's impact on the gender divide in computing. We also examine older students' tendency towards intrinsic motivation to pursue an MSCS degree.

An MS in CS for non-CS Majors: Moving to Increase Diversity of Thought and Demographics in CS

Authors: Carla Brodley; Megan Barry; Aidan Connell; Catherine Gill; Ian Gorton; Benjamin Hescott; Bryan Lackaye; Cynthia LuBien; Leena Razzaq; Amit Shesh; Tiffani Williams; Andrea Danyluk


We have created, piloted and are growing the Align program, a Master of Science in Computer Science (MS in CS) for post-secondary graduates who did not major in CS. Our goal is to create a pathway to CS for all students, with particular attention to women and underrepresented minorities. Indeed, women represent 57% and underrepresented minorities represent 25% of all bachelor's recipients in the U.S., but only 19.5% and 12.6% of CS graduates, respectively. If we can fill this opportunity gap, we will satisfy a major economic need and address an issue of social equity and inclusion. In this paper, we present our "Bridge" curriculum, which is a two-semester preparation for students to then join the traditional MS in CS students in master's-level classes. We describe co-curricular activities designed to help students succeed in the program. We present our empirical findings around enrollment, demographics, retention and job outcomes. Among our findings is that Align students outperform our traditional MS in CS students in grade point average. To date we have graduated 137 students and 827 are enrolled.

A Constructivist Redesign of a Graduate-level CS Course to Address Content Obsolescence and Student Motivation

Authors: Lorenzo Angeli; Juan José Jara Laconich; Maurizio Marchese


The last decade has seen a rising popularity of active learning methodologies in Computer Science, empowering students and developing their soft skills as well as their technical knowledge. In parallel, the speed of technological obsolescence also increased, creating challenges for teachers to keep their course content fresh and up to date. In this paper, we present a constructivist redesign of a Graduate-level laboratory course in Web Service Design and Engineering that leverages latent pockets of student knowledge to tackle these challenges through Learning by Teaching. We illustrate how such redesign was planned, deployed and evaluated, highlighting the guiding role of teachers in the process and discussing how this approach was able to solve the problem of keeping content updated while broadening both content and tools students were exposed to. Furthermore, we will discuss how the additional motivation stemming from their empowerment allowed students not only to perform more work compared to a lecture-based implementation, but also to perceive it in the end as a lesser load.