Assessment of Student Learning

Introduction
Systems engineering (SYSE) at Portland State emphasizes general and fundamental concepts. Two of the core courses present general systems engineering topics. The other two core courses are modeling courses taught by other programs for a wide range of student interests. The sixteen credits of electives may provide a specialization in systems engineering or a related area. However, the online electives are a general treatment of subject matter important to many systems engineers: additional modeling, risk and decision making, reliability engineering. The rationale for this emphasis is related to needs of the engineer in the Portland region. Even though we are an online program, connection to PSU and its community are important. Our students typically do not work for large government agencies who often seek systems engineers with a specialty, such as system-test, logistics or acquisition engineering. As demonstrated by our course offerings, this program caters to the following student profile:

works for a small to mid-sized commercial company
has both systems engineering and non-systems engineering duties
may be the only systems engineer within the firm
may have a bachelors degree in engineering, science, or business

Learning Objectives

Core and Required Courses----------------------------------

SYSE 506 : Masters Project
(9) Credits, Professor: Herm Migliore

Student can identify a system, potentially for their current employer, that provides a capstone experience.
Student can develop a systematic approach.
Student includes and executes all major systems engineering activities: a) identification of customer and stakeholder needs, b) requirements management, c) validation and verification, d) formal interface management e) assessment of results.
Student incorporates and integrates the views related to technical engineering, student learning, and systems engineering.
Student produces stand-alone final report that captures changes documented in progress reports and relates to the initial proposal.
Student demonstrates system thinking.

SYSE 590 : Integrative Workshop
(4) Credits, Professor: Herm Migliore

Student shows understanding of concepts and big picture through organization.
Student has integrated various tools of program as portrayed through portfolio and conversation.
Student is proficient in examination through system life-cycle model and time management.
Portfolio accurately tracks student’s development with organization detail and insight.
Student proficient in team building and maintenance through various forms of communication and creative approach.
Student identifies differences and similarities in approaches of engineering specialty and systems engineering.

SYSE 591 : Systems Engineering Approach
(4) Credits, Professor: Ike Eisenhauer

This course provides the beginning knowledge and skills necessary to engineer complex, multi-disciplinary systems. It serves as a cornerstone course for the Systems Engineering program.
The student will gain interdisciplinary knowledge and skills necessary to:
i. Define the system life cycle and the particulars of stakeholder involvement
ii.Cover critical tools and methods for implementing Systems Engineering
iii.Explain the various structure and tasks of the Systems Engineering process
Upon completion of this course, each student should be able to:
a.Understand systems engineering as an interdisciplinary process.
b.Demonstrate the value of systems concepts in the development of products, processes, and services.
c.Access case studies, templates, and checklists that support the systems engineering approach.

EMGT 540 : Operations Research in Engineering and Technology Management
(4 Credits), Professor: Tim Anderson

Student appropriately applies linear programming to engineering management decisions.
Student appropriately applies goal programming to engineering management decisions.
Student is able to identify and formulate the problem for real situations.
Student is proficient in mathematical model building.
Student has integrted principles of the Simplex algorithm.
Student competently synthesizes multiple objective linear optimizations.

SYSC 514 : System Dynamics
(4) Credits, Professor: Wayne Wakeland

Student can adequately portray understanding of generic dynamic system.
Student can apply concepts to real models.
Student can formulate comprehensive structure of a new continuous system.
Student can apply information and impacts fro feedback into systemic changes.
Student project covers key components and dynamic with depth and insight.
Student show adequate comprehension of simulation language (Stella).

SYSC 527 : Discrete System Simulation
(4) Credits, Professor: Wayne Wakeland

Student displays evidence of comprehension of basic DSS premises.
Student appropriately applies concepts to working models.
Student can comprehensively brek down the application.
Student is able to measure, analyze, apply, and integrate results in relation to the system and needed functionalities.
Student can adequately identify factors and entities.
Student has integrated concepts into student’s specialty or area of interest.

SYSC 529 : Business Process Modeling and Simulation
(4) Credits, Professor: Wayne Wakeland

Student has advanced understanding of Discrete Systems.
Student can apply Discrete Systems concepts to business processes.
Student has adequate comprehension of Continuous Systems.
Student can delineate points of change fro improvement based on needed functionalities relating to specific processes or queues.
Student can adequately break down, interpret, and apply statistics.
Student has working comprehension of Monte Carlo.

SYSE 573 : Requirements Engineering
(4) Credits, Professor: Dorothy McKinney/Jacob Goldstein

Student can write high quality requirements and requirement documents
Student can define professional scale requirement management processes
Student can derive an appropriate set of system requirements from the needs of an organization, project, or other goal focused starting point
Student can adequately develop derived requirements from a set of high level requirements
Student can integrate individual requirements and the big picture with adequate comprehension
Student can develop system concepts from mission needs and market opportunities as well as from available technology standpoints
Student can apply various requirement engineering approaches to real situations
Student can test, measure, compare, improve and prioritize system requirements
Student can analyze a set of requirements, a requirements document, and a requirements management process and identify area for expansion and/or refinement, where appropriate

Electives----------------------------------------------------------------------

SYSE 595 : Hardware – Software Integration
( 4) Credits, Professor: John Blyler

Student adequately identifies factors relating to hardware and software needs.
Analysis leads to appropriate hardware and software choices.
Student can show beneficial application of various information sources (I.E.: studies) as efficiency in issue integration.
System level functions and relations to haredware/software components are methodically portrayed (concept map).
Student effectively applies skills and knowledge to real situations (case studies).
Student has working knowledge of design checklist and templates.

EMGT/SYSC 553 : Manufacturing System Simulations
(4) Credits, Professor: Wayne Wakeland and Tim Anderson

Student has high level working knowledge of ProModel.
Student attained adequate level of comprehension of concepts.
Student can integrate concepts with ProModel.
Student can appropriately apply software to real situations.
Student can apply concepts to real situations.
Student can use concepts and software congruently in creating and analyzing applications, designs, and systems.

SYSE 575 : Reducing Risk in Decision Making
(4) Credits, Professor: Ike Eisenhauer

Assess risk, quantitatively and qualitatively
Model Risk through various methods
Understand the concept of valuation of risk [i.e. utility] and its inherent subjectivity
Evaluate the value of information to enable the assessment of consultants and advice
Probabilistic and stochastic representations of risk and how to utilize these concepts
Develop multiple, potential conflicting, objective models for decision making

SYSE 561 : Logistics Engineering
( 4) Credits, Professor: Has Not Been Offered for Two Years

Student can apply principles to existing systems.
Student can create appropriate hierarchy from all components.
Student can apply principles to creating new system from hierarchy.
Student can articulate early phase emphasis and priorities with vision of outcome.
Student can map process (concept map) with all components, outcomes, and logistical perspectives.
Student can flow backwards and forwards from logistics to systems approach.

EAS 561 : Reliability Engineering
(4) Credits, Professor: Ike Eisenhauer

Student can delineate reliability needs.
Student can test, measure, and assess.
Student adequately integrates maintainability with reliability requirements.
Student comprehensively creates new models.
Student appropriately applies knowledge to real systems.
Student in able to evaluate and integrate such systems in conjunctioin with other parameters and within total life-cycle.

Tools

Focused Email Discussion
Given that faculty and students are remote, course issues are generally communicated by email with the faculty acting as facilitator. The faculty is central to distributing topics for group discussion, whether initiated by the Director, faculty, or students.

Practicing Engineers
Systems Engineering (SYSE) courses are taught by experts in the field who are full time practioners. Not only are they familiar with the technical aspects of their courses, but they are also aware of the impact of the program on the systems engineering community. Virtually all students are practicing engineers who already have several years experience in systems engineering. By virtue of their particiaption, they are well aware of the important of the course objectives and well aware if they are being met. Their comments are solicited by the instructors and often conveyed to the program director.

Student Portfolios
All masters students are required to maintain an e-portfolio of their learning experience. Students are asked to compare their learning objectives with the course objectives and whether course objectives have been met. The portfolio also provides an opportunity to reflect on the integration of courses relative to their learning objectives and to program objectives.

Rubrics
As part of university accreditation, quantitative rules for assessment are required. Although the systems engineering program is just beginning the development of rubrics, they are well suited for a discipline that prides itself on validation and verification. Professor Ike Eisenhauer has formulated a set of rubrics for the courses that he teaches: SYSE 506, SYSE 591, SYSE 575, EAS 561. The scores and planned corrective action will be documented in the next section, Data and Reflection, under specific terms and courses.

Existing Student Assignments

A number of tools are used to gage the success of various aspects of Requirements Engineering Course.

Weekly discussion and journal assignments are used to ensure that students understand each module’s learning objectives and stay on track with the course material
Homework assignments measure the student’s ability to employ the skills covered in an individual module to real-world situations
The final project ensures that students have mastered all learning objectives for the course and can apply them appropriately to real-life scenarios
Assessments focused on the course content are delivered through the quarter as well as in the form of a midterm and final exam. These assessments provide an additional opportunity for the instructor to measure the student’s understanding of the course material.
Students provide real-time feedback through the quarter in a discussion forum focused specifically on course content and mechanics. Students contribute questions and suggestions and instructors provide feedback and engage in ongoing discussions in order to better understand the most effective way to deliver course material

Feedback provided in discussion forums are reviewed on an ongoing basis by instructors as it is provided. Suggestions that can be incorporated into the course immediately are reviewed further an implemented as soon as possible. Suggestions that would require a large scale change to the course are recorded and reviewed once the quarter is complete.

Formal assessments on the course are completed twice during the quarter – once at the midway point and once at the end of the course. Students using a likert-style rating scale to assess various aspects of the course material and delivery style.

Feedback from the midterm assessment is reviewed as soon as it is received and, like suggestions from discussions, is implemented immediately when possible. End of term feedback is reviewed once the quarter is complete and implemented as appropriate for the next quarter.

Data and Reflection

Fall 2006, SYSE 591, Systems Engineering Approach-----------------------------------------

Clearer, more pointed and varied assignments.Currently assessments are made through 3 written assignments, weekly quizzes, a mid term and a final.These written assignments are long and involved, which based on end of term and midterm student feedback has been challenge at times to complete while trying to understand the concepts involved. This will be changed to 8 written assignments, the quizzes will be removed, the mid term and final will remain. The written assignments will be more varied and have both conceptual understanding exercises to ensure proper understanding of the concepts involved, along with more complex application demonstrations. The increased frequency will allow quicker feedback to students. The introduction of conceptual understanding exercises will allow the student to identify conceptual weakness earlier and provide a clear frame of reference for both student and instructor to discuss.
Increasing explanatory notes to cover areas of common misunderstanding by students. Unfortunately this had to wait until a batch of students came through since the instructor is familiar with the material, and the best test cases of confusing areas are the students themselves.
Use a common case study through out the course to better show integration of the Systems Engineering method and to provide a common-built up framework for the students to follow along with. The disjointed method of minicase study for each part may have been clear in usage of method portions, but did not allow the integral aspects of the method to be demonstrated effectively.

Winter 2007, SYSE 575, Reducing Risk in Decision Making---------------------------

Student Assessment of First Version. The course used the systems engineering approach as a means to reduce risk when making decisions and consisted of :a. review of the FRAT view of systems engineering, b. review of decision making, qualitative and quantitative, c. introduction of risk concepts and tools. Most students considered the material too much of an overview. The relationship between systems engineering and decision making was too loosely presented, as was the relationship between decision making and risk. Students wanted a textbook.
Student Assessment of Recent Offering. Decision making was presented in Rex Brown’s book, “Rational Choice and Judgment”. Some students liked the book because it took a personal approach; others found it too qualitative. Introduction to risk concepts is covered in the course notes and in the second textbook, M. Modarres’ “Risk Analysis in Engineering”. The Modarres concentrates on PRA, probabilistic risk assessment. The students found that this text required prior knowledge of modeling and did not set the stage for many of the difficult concepts.
A new textbook is being considered for the next offering. "Risk Modeling for Determining Value and Decision Making", by Glenn Koller.

Spring 2007, SYSE 573, Requirements Engineering------------------------

Assessment Vehicle: For SysE 573 we received feedback from three distinct sources (1) Course Evaluations delivered through the WebCT tool in the same fashion as a quiz, i.e. asked students to rate different aspects of the course and/or provide written comments (2) Discussions regarding how the course was functioning and what could be done to improve it – captured in the discussion forums in WebCT (3) E-Mail exchanges with students regarding things that are working well and things that could be improved.
What I Learned: I took the following main points from the feedback we received (1) Students find value in having access to a large list of external references and resources that they can use to better understand the course content (2) Students would like more direct correlations or distinctions between how the course content applies to different fields within engineering (i.e. software vs. hardware) (3) Certain areas of the course could benefit from an expanded description of the associated content to help students understand the key points (4) URLs that changed mid-quarter prevented students from access references in a timely manner and negatively impacted the student’s impression of the course
What I am going to do: Based on the feedback provided by students I plan to make the following revisions to the course (after first planning and discussing with Dorothy): (1) Expand the list of references for every section of the course so that students have the opportunity to explore every area of the course further (2) Include links to references that are specific to a discipline within engineering (3) Expand the course material in areas that generated confusion for students during the quarter (4) Capture articles from the internet and provide them to students through WebCT as downloadable attachments. This will prevent the possibility that a URL will change mid-course and students will have trouble accessing material.

SYSE 595, Hardware-Software Integration------------------------------
Summer 2007

In response from past students to expand coverage and applicability of the course to include all electronic systems, e.g., chips and boards developers. Also, several students wanted less emphasis on analog and RF systems.
Align the course more closely with a systems engineering, both in order of content as well as illustration of basic concepts in each weekly learning module. Basically, the first three-four learning modules will introduce the problem and clearly define terminology. This is essential since both the chip and board electronic communities (including related topics like embedded, hardware, software and others) all use "systems engineering" terms to mean similar but different things.
After the problem is clearly defined and understood, various solutions and implementations will be covered (learning modules 5 through 9). This approach follows the systems approach to managing complex systems.

SYSE 595, Hardware-Software Integration------------------------------
Spring 2008

John Blyler gave a presentation at the Embedded Systems Conference, which incorporated new material that he plans to also incorporate in his graduate course. Some of the attendee comments can be found on John's Blog site. Other recommendations can be found in John's course review

EAS 561, Reliability Engineering---------------------------------------
Spring 2008, Course Assessment

SYSE 573, Requirments Engineering-----------------------------------
Spring 2008, Course Assessment