Software Development for Large Systems ETSN05: Lecture 5 Alma Orucevic-Alagic, Fall 2016
Lecture 5: Overview Evaluation and Assessment: PDSA Quality Improvement Paradigm GQM Final Report Individual Assignment Issues within Large Software Development Context Course Project Discussion TODO: Brainstorm -How the topics discussed relate to your experience on the course project? -What would you like to discuss (what went well, what didn t, what should be changed)?
Lecture 5: Evaluation and Assessment PDSA Cycle Deming Wheel, Deming Cycle Act Plan Study Do
Lecture 5: Evaluation and Assessment Act Study Plan Do Plan: -Identify goal or purpose -Formulate a theory -Define success matrix -Put plan into action Do -Execute plan, e.g. create a product
Lecture 5: Evaluation and Assessment Act Study Plan Do Study: -Monitor outcomes -Test plan validity -Progress, Success, Problems Act -Integrate lessons learned -Apply necessary adjustments: -Goal, methods, or theory
Lecture 5: Evaluation and Assessment Quality Improvement Paradigm 1. Characterize the current project and its environment with respect to the appropriate models and metrics 2. Set the quantifiable goals for successful project performance and improvement 3. Choose the appropriate process model and supporting methods and tools for this project 4. Execute the process, construct the products, collect, validate and analyze the data to provide real-time feedback for corrective action 5. Analyze the data to evaluate the current practices, determine problems, record findings, and make recommendations for future project improvements. 6. Package the experience in the form of updated and redefined models and other forms of structured knowlefge from prior projects and save it in experience base. The Maturing of the Quality Improvement Paradigm in the SEL, Victor A. Basili, SEW Proceedings
Lecture 5: Evaluation and Assessment Lessons Learned: Quality Improvement Paradigm Understand paradigm, process, project, environment Build own models to understand and characterize the environment Models are environment specific hard to generalize Understand which factors create differences and similarities among the projects to identify appropriate model Evaluation and feedback are integral part of project control: data collection needs to be goal driven, not collected to later figure out what to do with it Major paradigm: Goal, Question, Metric The Maturing of the Quality Improvement Paradigm in the SEL, Victor A. Basili, SEW Proceedings
Lecture 5: Evaluation and Assessment GQM: Goal, Question Metric Conceptual level (GOAL): defined for an object with respect to various models of quality, from some points of view, relative to an environment. It is measured through: Product : Artifact, deliverables, documents produced during the system life cycle Process: Software related activities: specifying designing, testing, interviewing.. Resources: Personnel, hardware, software, office space The Goal Question Metric Approach: Victor R. Basili, Gianluigi Caldiera, H. Dieter Rombach
Lecture 5: Evaluation and Assessment GQM: Goal, Question Metric Operational level (Question) Characterize the object of measurement (product, process, resource) w.r.t. a selected quality issue to determine its quality from the selected viewpoint. The Goal Question Metric Approach: Victor R. Basili, Gianluigi Caldiera, H. Dieter Rombach
Lecture 5: Evaluation and Assessment GQM: Goal, Question Metric Qualitative level (Metric): Objective: Depend only on object being measured, e.g. number of versions of document, program size, hours spent Subjective: Depend on both the object being measured and viewpoint from which they are taken: e.g. readability of a text, level of user satisfaction. The Goal Question Metric Approach: Victor R. Basili, Gianluigi Caldiera, H. Dieter Rombach
Lecture 5: Evaluation and Assessment The Goal Question Metric Approach: Victor R. Basili, Gianluigi Caldiera, H. Dieter Rombach
Lecture 5: Evaluation and Assessment A learning organization is an organization skilled at creating, acquiring, and transferring knowledge, and at modifying its behavior to reflect new knowledge and insights D. A. Garvin, Building a Learning Organization, in Harward Business Review on Knowledge Management, pp. 47 80, Harward Business School Press, Boston, USA, 1998. Requires: systematic problem solving, experimentation, learning from past experiences, learning from others, and transferring knowledge
Lecture 5: Evaluation and Assessment Postmortem Analysis Ensures that the team members recognize and remember what they learned during the project Identifies improvement opportunities and provides a means to initiate sustained change IEEE Software, May/June 2002
Lecture 5: Final Report Historical overview of the project: Figures, tables, diagrams, etc Comparison between actual values and estimations Evaluation of what went well and what went not so well Analyze reasons for problems/issues/etc Software process improvement proposals
Lecture 5: Final Report To be included in the final report: Effort per phase Start and end dates for each phase Effort per document Start and end dates for each document Effort for different activities in each phase Effort per group & week Analysis of problem reports in phases
Lecture 5: Individual Report Course home page (Individuell Uppgift) http://fileadmin.cs.lth.se/cs/education/etsn05/ individual_assignment.pdf Objectives: To stimulate reflection on large-scale software development continuously through the course. To encourage a viewpoint on industrial practice in large-scale software development. To build on and integrate with what you have learned from previous courses. Identify relevant areas and motivate your conclusions.
Lecture 5: Individual Report What was challenging and what was easy in your own work? What was challenging and what was easy in your fellow project members work? What are the similarities and differences between the controlled course situation and industrial practice? What is realistic about the course setting, and what is not so realistic? What role does the scale (in terms of size and complexity) play in the challenges you have had during the project? What is easy in a small-scale project while significantly more challenging in a large-scale project? Which problems are not more or less difficult to address when carried out in a large-scale setting compared to a smaller scale setting?
Lecture 5: Individual Report Deadline Monday 10.10.2016, 23:59 Email project to Alma Orucevic-Alagic in pdf format Max length 3 pages (without the cover page, table of contents, etc...), 11pt Times Include your name, project group, and project role on the report.
Lecture 5: Issues within Large Software Development Context What is actually the best set of requirements? How much uncertainty in effort estimation can we cope with? At what level of detail should we document requirements? How to minimize waiting time for other parts to be ready before we can start our part? How to make more parts in parallel without generating confusion and unnecessary rework? How to know when the product is reliable enough to be released? How to incorporate changes without generating spaghetti and excessive cost of rework?
Lecture 5: Issues within Large Software Development Context Multi-project environment, geographically distributed development sites, varied communication norms: Sub-optimization, uncoordinated. Elementary or advanced process Chasm between marketing and development Working with changing requirements Innovation capability Managing diverse human resources
Lecture 5: Issues within Large Software Development Context Stakeholders often don t know what they want from the computer system Stakeholders naturally express requirements in their own terms and with implicit knowledge of their own work Different stakeholders have different requirements, which they express in different ways Political factors may influence the requirements The economic and business environment in which the analysis takes place is dynamic à requirements may change during the project
Lecture 5: Issues within Large Software Development Context Conclusion: Large-scale development requires processes, organizations and tools that can cope with increasing complexity Your engineering skills are defined by: Ability to combine technology and economics Work in teams and with big organizations
Lecture 5: Discussion Your Questions
Lecture 5: Discussion (Reflections) Att få testgruppen att arbeta parallellt med de andra Koordinera ut information till grupperna Lätt göra själv, men svårt dela ut i rätt tid Tighta deadlines i början. Inget att göra i början men sedan mycket Hantering av PR. De rör många dokument. Obalanserad arbetsbelastning Skillnad i ambitionsnivåer i gruppen à dålig arbetsmoral Teknisk kompetens skiljer sig i grupperna
Lecture 5: Discussion (Reflections) Ibland har det uppstått situationer där det inte har gått att jobba med någonting, tex att det inte känns bra att börja skriva på SVVI förrän SVVS är satt i baseline, helt enkelt för att man hade fått göra om för mycket arbete. Jag skulle gärna ha någon diskussion kring tillgången till automatiserade verktyg för att hantera projekten. Tidsrapportering och planering känns som att det borde finnas rätt bra verktyg till.
Lecture 5: Discussion (Reflections) Det skulle även vara intressant att diskutera hur mycket som är lagom när det gäller hård koll på rapportering, rutiner o s v. Någonstans måste det ju finnas en gräns där systemet snarare är i vägen än att det hjälper. T ex kan jag tänka mig att utvecklare börjar lösa problem utan att rapportera om proceduren att rapportera är för jobbig.
Lecture 5: Discussion (Reflections) Jag har funderat kring hur vårt arbete skiljer sig från verkliga fall vad gäller hur mycket av de inblandades tid som tas i anspråk. Med detta menar jag att många moment förutsätter att medarbetare finns på plats alla veckans dagar och ofta kan utföra arbetsuppgifter med kort varsel eller jobba extra på helgen. I en verklig situation arbetar man ju ofta med flera parallella projekt, medarbetare reser i tjänsten och vill inte komma in en söndag. Hur tar man hänsyn till detta och planerar på bästa sätt i ett verkligt industriprojekt? I hur stor utsträckning brukar olika projektdeltagare ha totalt olika scheman i ett verkligt projekt? I den här kursen har vi ca 14 personer som läser 2-3 olika kurser var, och därmed har i princip alla helt olika scheman, så det är mycket svårt att hitta t.ex. mötestider där alla kan.