Research Methods

Introduction
Mission Statement
Course Enrollment
Areas of Study
Survey & Interviews
Card Sorting
Technology Review

Introduction

We employed a number of qualitative and quantitative methods to understand the misalignment between reality and expectations at SIMS.

1. Mission Statement: To avoid over-emphasizing the short-term perspective of students in a two year program, we reviewed the proposal to establish the school, presented to the U.C. Regents in 1993.
2. Course Enrollment: Our research delved into overall course enrollment data to identify larger trends in student interests.
3. Areas of Study: To determine the skill sets that SIMS students seek, we evaluated the courses each student took to see how courses clustered into specializations.
4. Interviews & Survey: We compiled a survey to gauge students’ opinions about SIMS’ expertise and their experience at the school.
5. Card Sorting: A number of terms related to Information Management and Systems were captured during each of the methods listed above. We used card sorting as a technique to determine how individuals organize the SIMS Corpus.
6. Technology Review: Because use of technology was referenced during interviews, we evaluated technical options that could possibly address some issues and facilitate the completion of a few recommendations.

The information we gathered while employing these techniques serves as the foundation for our findings and subsequent recommendations.

We began our research by evaluating the school’s mission and charter. Doing so was critical because we wanted to avoid making rash judgments about the school’s performance or future directions. Of special interest was the intent of the school’s founders when they formulated the proposal for the school in December of 1993. We used it to identify if (or how) SIMS, and the niche the school sought to fill, has changed since the school’s inception in 1995.

The School’s Challenge
The original proposal made to the U.C. Regents for the establishment of a school like SIMS described an issue that needed to be addressed both academically and professionally. It states “Information is one of the world’s most important and rapidly changing resources. The challenge is to filter what is most useful out of the vast quantity of information available: to select, evaluate, describe, store, retrieve, manipulate, and present information in all of its forms, including text, still and moving images, sounds, and numeric data.” Despite the degree to which the tools and strategies we use have changed since the early 1990’s, this issue is still at the core of SIMS’ mission and purpose.

Educational Mission & Structure
The proposal also described the means for educating students to address the numerous issues surrounding information. The Master’s program was envisioned as a professional program that would teach people about the design and operation of information systems and services, the nature and properties of information, and information-related behavior at the individual, group, and societal levels. Graduates were expected to apply their skills in corporate, government, and academic environments to develop innovative approaches for handling information, designing and managing information functions, and merging them with other aspects of the organization. It was recognized that students would work in a broad range of companies - from those primarily concerned with information access, technology, organization, and preservation to “old economy” firms. In each case, the skills taught at SIMS are essential if these organizations want to effectively manage and exploit their stores of information.

Job Functions
An appendix to the proposal described the types of functions SIMS graduates would be prepared to perform. This element of the document is particularly relevant to the feedback received on the survey. Thus, we have chosen to look at the types of functions that were described at the school’s inception and evaluate how job market that SIMS’ graduates will fill has changed in the past 10 years.

The areas described fell into four basic categories: Design, Management, Users, and Policy. Design and management functions include designing and managing information services, systems, and products as well as content for those systems and the mechanisms to deliver the information. User interaction functions were primarily in the areas of training and customer feedback to inform requirements. And finally, policy analysts were anticipated as being needed to establish information access policies and crafting broader public policies for privacy and intellectual property.

Course Enrollment

After obtaining a strong grasp of the school’s original mission and curriculum, we sought to determine if students’ areas of interest aligned with the courses being offered by the school. We did so by aggregating course enrollment figures. Three types of calculations were run on the course enrollment data for SIMS courses from Fall 2000 to Fall 2003.

• Plot of Students per Course (SIMS): The average size of a SIMS class overall and per semester.
• Elective Enrollment by Department: The percentage of elective courses taken at SIMS vs. other departments.
• Percentage of Course Enrollment (SIMS): Actual enrollment figures were compared to the number of available seats per course outlined in the UCB Course Catalog. To compensate for fluctuations in the SIMS student body, the data was normalized against yearly IS 204 enrollment figures.
• Percentage of Course Enrollment (MOT): Actual enrollment figures were compared to the number of seats allocated to SIMS students in each MOT course offered at Haas.

Even though our objective was to focus on the Master’s classes of 2003, 2004, and 2005, it was important to incorporate course enrollment data from prior years as a benchmark. Such information would provide additional insight into the duration of some trends and allow us to analyze courses which were not offered during the targeted years.

Plot of Students per Course (SIMS)
Course enrollment data was obtained from SIMS and used to determine the average number of students per class across all semesters being analyzed. SPSS was used to produce a summary graph,

Elective Enrollment by Department
This technique was used to find trends in the percentage of time Master’s students as a whole spend in SIMS courses vs. non-SIMS courses; MOT courses were also isolated in this analysis. Using course enrollment data, we created a department by semester pivot table which calculated the number of classes any SIMS student took in a given department. A detailed analysis as well as graphs can be found in Findings and Recommendations – Continuity and Community at SIMS.

Percentage of Course Enrollment (SIMS)
Using course enrollment data, we identified the Top 5 SIMS courses for each semester. The “Top 5” are the courses with the greatest enrollment. It was defined by dividing the Total Number of Students Enrolled by Available Seats; the number of available seats was given in the schedule of classes.

Percentage of Course Enrollment (MOT)
In order to identify areas of interest outside SIMS, another set of graphs were developed. Data from the UCB Course Catalog was used to identify if SIMS students were filling up seats allocated to them in MOT courses offered at Haas. See Appendix A.3.

The SIMS curriculum is promoted to students via the website as being organized in several degree tracks--Interface Design & Evaluation, Design & Management of Information Services, Information Resources (Collection) Management, Information Systems Design & Implementation, Management of Information Organizations, and Information Policy. We wanted to determine if students were actually following these tracks, and if not, what de-facto degree tracks they were pursuing.

We broke down course enrollment data and interpreted the information from a student perspective, starting with the incoming class of 2000 and going up to the current first year students (i.e. the graduating class of 2005). This was an ideal time frame to examine the tracks students were pursuing because it allowed some time for the school to become established and define itself; SIMS was established in 1995 and took on its first incoming class in 1997. The period between 1997 and 2000 was undoubtedly an adjustment period where the school and students were feeling each other out, and just beginning to realize the potential of the curriculum. Thus, these years were excluded from our analysis.

We received the data as a list of every course taken by every SIMS student, anonymized with pseudo student IDs. From this raw data, we proceeded to identify every unique class taken by creating a course key in the format semester.year.dept.course#.section. This step was necessary to isolate unique classes, especially at SIMS; many courses are listed as IS 290 special topic courses with section numbers that are not unique for a given class (i.e. course numbers rotate and vary from semester-to-semester and year-to-year). Then, to more quickly identify and isolate de-facto degree tracks, we created a matrix, shown below, with unique courses as columns and student IDs as rows. Courses students had taken received a 1 in the corresponding cell and all other cells were assigned a value of 0. The 1s and 0s were concatenated to create a large binary vector which then could be analyzed to reveal original degree tracks.

The final step was to eliminate the class of 2005 and students who dropped out. This was necessary because, in both cases, the individuals had not completed enough coursework for us to gain any insights and make assumptions about their academic focus. Also, having only one semester of data for the class of 2005 prevents any analysis of their choices for elective courses, making it impossible to decipher any patterns. Without this information, there was no way to determine whether students took a course for pure interest rather than professional development.

Somewhat surprisingly, upon filtering the concatenated numbers that represented the degree tracks, we discovered that no two students had taken exactly the same courses during their time at SIMS. In the absence of identical tracks, we sought to find smaller clusters of courses. To identify these, we ordered the courses by highest enrollment and began to see multiple smaller groups of two and three courses that were taken collectively. These smaller groupings essentially came to represent the core skill sets or de-facto degree tracks. For example, there was a strong cluster of students who took UI Design, Needs Assessment and Usability, Human-Centered Computing, Computer Networks, Database Management, and Document Engineering which can be construed as Systems Design and/or HCI track.

While the course enrollment and areas of study analyses relied more on objective methods, it was important to incorporate subjective student input as well. Without their comments, we would have been unable to truly define the perceived identity of the school. A survey was used to gather their feedback and help identify students’ perceptions of SIMS’ core competencies. The survey included questions about how students found out about the school, as well as what influenced their decision to attend SIMS. Additionally, there were questions pertaining to the skill sets students’ seek when they attend SIMS and the expertise they use in jobs acquired after graduating.

The survey was comprised of 26 questions which were pre-tested with Nancy Van House’s IS 214 – Needs Assessment class. After taking the survey, students provided helpful feedback about the wording and flow of questions, helping us to be more effective at eliciting student perceptions. Subsequently we performed two or three interactions before posting the survey to an online survey site, SurveyMonkey. Emails were then sent out to the SIMS community to solicit responses. Altogether we had 106 respondents, with 60% of respondents attending SIMS after 2000.

Card sorting is a technique that many information architects use as an input to the structure of a website or data model. As part of the SIMS Corpus project, we performed a card sorting exercise to determine how students in the school organize and relate the information management concepts taught at SIMS. Participants’ resulting schemes differed considerably, supporting our other research conclusions, suggesting SIMS does not have a widely-shared identity.

Participants were given cards, each labeled with a single word with no pre-determined groupings. They were asked to sort cards into groups they felt were appropriate and then describe each group. A number of information management and systems related terms were used, such as Information Architecture, Information Policy, Human-Computer Interaction, and Systems Design. Analysis of the resulting classification schemes, described in detail in the SIMS Identity Findings and Recommendations of this document, provided more insight into the inconsistency of opinions about SIMS and prioritizes some of the areas that could use the most definition.

As the School of Information Management and “Systems”, it was also important to evaluate any technology that was used to represent SIMS – specifically SIMS internal and external website. We gathered feedback from students and faculty, and based on their insights, were prompted to research web-based portal technologies.

Portals have fundamentally changed the way information is presented on the web. In the earliest versions of the web, organizations presented a one-size-fits-all webpage. Advances in web services in 2000 and 2001 enabled a greater degree of interactivity, but the user still had to know how to navigate to or browse for relevant information. The portal limits the type of information available to a user based on their role and then allows the user to select and arrange the information components that are most important to them. The net effect is that each person has a webpage customized to meet their needs.

More insight into portals and comparative offers, please review our IS213 project site - specifically the Comparative Analysis completed for Assignment 3.