Do Patterns Affect Design Undertandability?


Javier Garzás , Félix García y Mario Piattini

Abstract

At the present moment in time, patterns have reached a zenith in popularity and diffusion thanks to the concern of the community to discover, classify and spread knowledge about all types of patterns. Patterns are useful elements, but there are still many features to be studied if we want to apply them in a rational manner. The improvements in quality that patterns can inject into design is a key issue to analyze, so there needs to be a complete body of empirical knowledge dealing with that. This paper tackles the question of whether design patterns can help to improve the ease with which designs are understood. An empirical study, composed of one experiment and a replica, has been conducted with the aim of validating our conjecture. The results suggest that patterns contribute to improving the understandability of the design.

Keywords: Design Patterns, Understandability

1. Introduction

For many years now, design patterns have been associated with the improvement of software maintainability. Popularly, patterns and maintainability are synonymous with well-designed software. A quality design that has a high level of maintainability could thus have a clear effect on the productivity of developers. The effectiveness of maintenance is, furthermore, greatly hindered by poor design [1].

These seemingly obvious statements are questioned by many professionals. Glass [2] states that well-designed systems will be improved more rapidly, have a longer life, and hence, paradoxically, consume more maintenance. Reibing [3]comments that if we have two similar designs, A and B, for the same problem, B using design patterns and A not using design patterns, B should have a higher quality than A, but if we apply “classic” object-oriented design metrics to both designs, the metrics tell us that design A is better. In the present state of affairs, there is a serious lack of hard scientific evidence for the usefulness of patterns in many situations [4].

The relationship between maintainability and patterns, then, is not clear, nor are the positive or negative effects of patterns.

Regarding design maintainability, understandability is one of the key characteristics to consider, as it can influence the cost and reliability of software design evolution or maintenance. As a matter of fact, understandability is one of the five characteristics a “good” model must have, along with abstraction, accuracy, predictiveness and inexpensiveness [5].

There is no firm and formal theory that links patterns to these software quality concepts, however. Many authors claim that Design patterns claim to bring about understandability [6], but we need to build an empirical body of knowledge to confirm these assertions. The question, therefore, is this- what effect do design patterns have on understandability?

This paper proposes to seek empirical evidence about whether using design patterns is beneficial for software design understandability. We conducted two experiments to compare performances of designs with patterns to alternatives where patterns were not used.

The remainder of this article is organized as follows: Section 2 illustrates the experiment settings; Section 3 summarizes data analysis and its interpretation; Section 4 outlines the limits of the experiment. Finally, conclusions are drawn and future work is set out in section 6.

2. Description of the Experimental Settings

As the main goal of the presented work is to ascertain if the use of design patterns can make the design easier to understand, our research question can be stated as:

Does the use of patterns improve the understandability of design?

From the above question, the general goal of our empirical study is derived as follows:

  • Analyze Design diagrams with and without patterns
  • With the purpose of comparing
  • With respect to their understandability
  • From the point of view of the designers
  • In the context of undergraduate students of Software Engineering

2.1 Subjects

The subjects of the first experiment were 10 students from the 3rd course of Computer Science at the ReyJuanCarlosUniversity (URJC, Madrid, Spain). To confirm the results, a replica was conducted at the Department of Technologies and Information Systems at the University of Castilla-La Mancha (UCLM, Ciudad Real, Spain) where 37 students from the final year (5th) of the MSc participated. The students in both experiments were familiar with UML and Design Patterns, as these were topics which they had been taught in their course of Software Engineering. Additionally, a short training session was provided for them, in which the design patterns and UML Class Diagram constructs were commented on and one example of the tasks to be performed by them in the experiment was explained. So we consider that the level of experience they brought to the experiment was acceptable.

2.2 Material

The material was prepared by the experiment designers and was composed of:

  • Four Universes of Discourse (UoD) (Aeronautics, Bank, University and Newspaper Agency). For each UoD, two semantically equivalent UML diagrams were prepared: one which included some design patterns and another one without patterns. Hence, a total of eight UML Class Diagrams were prepared.
  • Each diagram included a Questionnaire composed of four questions (answer yes/no) related to the model. For each UoD the same set of four questions was provided.

For the running of the experiment two sets of experimental material were obtained by grouping the diagrams in this way:

  • Group A: Diagram 1 (Aeronautics without Patterns), Diagram 2 (Bank with Patterns), Diagram 3 (University without Patterns), Diagram 4 (Newspaper Agency with Patterns).
  • Group B: Diagram 1 (Aeronautics with Patterns), Diagram 2 (Bank without Patterns), Diagram 3 (University with Patterns), Diagram 4 (Newspaper Agency without Patterns).

2.3 Variables

The dependent variable of our empirical study was the architectural design understandability and we measured this through the efficiency of the subjects in understanding the diagrams, i.e. the relationship between the number of right answers and the time they spent on answering them. Our main independent variable was the use or not of patterns in the diagrams. To provide a complete analysis of the experimental data, the UoD was also considered as an independent variable.

2.4 Hypotheses

The experiments were planned with the purpose of testing mainly the following hypotheses:

  • Null Hypothesis, H0P: the use of patterns does not improve the understandability efficiency of the architectural design.
  • Alternative Hypothesis, H1P: the use of patterns improves the understandability efficiency of the architectural design.

In addition to the above objectives, the testing of the following set of hypotheses was a secondary goal of the empirical study:

  • Null Hypothesis, H0UoD: there are no differences in understandability efficiency due to the Universe of Discourse of the diagram.
  • Alternative Hypothesis, H1UoD: there are differences in understandability efficiency due to the Universe of Discourse of the diagram.
  • Null Hypothesis, H0PxUoD: there are no differences in understandability efficiency due to the combined effect of Pattern usage and Universe of Discourse of the diagram.
  • Alternative Hypothesis, H1PxUoD: there are differences in understandability efficiency due to the combined effect of Pattern usage and Universe of Discourse of the diagram.

2.5 Experimental Tasks and Treatment

We applied an experimental design of confounded factorial with interaction (see Table 1). In each experiment, the subjects were randomly divided into two groups: Group A, whose members received the material set A; Group B, who were given the material set B. Each subject was required to answer the questionnaires of each diagram. In total, each subject received four diagrams of four different UoD (two of them designed by applying patterns and the other two without patterns).

 

Universe of Discourse

Aeronautics

Bank

University

Newspaper Agency

With

Patterns

Group B

Group A

Group B

Group A

Without Patterns

Group A

Group B

Group A

Group B

Table 1. Experimental Design

3. Data Analysis and Interpretation

When execution was over, we collected the forms filled out by the subjects, checking they were complete. As all the forms were indeed complete, all subject results were considered. Once the results were obtained, we first of all performed a descriptive analysis of the data. Table 2 shows the main descriptive statistics for the two experiments carried out:

Dependent Variable Measures

EXPERIMENTS

1st Experiment (URJC)

2nd Experiment (UCLM)

WITH

PATTERNS

WITHOUT PATTERNS

WITH

PATTERNS

WITHOUT PATTERNS

Mean

Std Dev

Mean

Std Dev

Mean

Std Dev

Mean

Std Dev

Time [seconds]

238,8

179,17019

243,6

116,51131

92,0811

58,07736

129,8378

80,54166

Number of Right Answers

3,35

1,137

2,95

1,146

3,68

0,599

3,20

0,682

Effectiveness

0,024245

0,0176492

0,015575

0,0110731

0,056780

0,0369626

0,034261

0,0204559

Table 2. Descriptive Statistics

The results of the first experiment show that the time subjects employed in understanding and answering the questions related to the diagrams where patterns were used was slightly lower than in non-pattern diagrams. Added to this, the correctness of answers was 13,5% better (0,4 positive difference) and the effectiveness was improved by 55% (0,00867 better score) with respect to non- use of pattern diagrams.

The replica at UCLM also confirmed a better performance when understanding diagrams with patterns and in this case the differences were more significant. As a matter of fact, the time was improved by 37,7 seconds on average (58% less time), the correctness of answers was 0,48 ( a 15 % improvement) and the effectiveness score went up to 0,022519 (65%).

The next step was to obtain insight as to whether these differences were statistically significant. We therefore performed an ANOVA statistical test (significance level α=0,05) in order to analyze the interaction between the independent variables under study when the measurement of the dependent variable is repeated. The statistical results for the hypotheses testing are summarized in Table 3:

Sum of Squares

df

Mean Squared

F

Significance Level

Experiment URJC

Subject (Group)

0,002

8

0,000

2,367

0,046

Error

0,003

26

,000(b)

UoD

0,003

3

0,001

8,313

0,000

Error

0,003

26

,000(b)

Patterns

0,001

1

0,001

8,806

0,006

Error

0,003

26

,000(b)

Group

0,000

1

0,000

0,008

0,931

Error

0,002

8

,000(a)

Interaction

0,015

1

0,015

55,421

0,000

Error

0,002

8

,000(a)

Experiment UCLM

Subject (Group)

0,036

35

0,001

1,844

0,009

Error

0,059

107

,001(b)

UoD

0,035

3

0,012

21,347

0,000

Error

0,059

107

,001(b)

Patterns

0,019

1

0,019

33,608

0,000

Error

0,059

107

,001(b)

Group

0,000

1

0,000

0,211

0,649

Error

0,036

35

,001(a)

Interaction

0,306

1

0,306

300,614

0,000

Error

0,036

35

,001(a)

a=MS(subject(group)), b=MS(Error)

Table 3: ANOVA results for understandability efficiency

As we can observe in Table 3, we obtained evidence that:

  • Design diagrams with patterns were easier to understand than non-pattern versions. These differences were confirmed as statistically significant (p=0,006 and 0,000 < α, H0P can be rejected in both experiments).
  • The UoD also had an influence on the understandability efficiency, i.e., subjects obtained different understandability performances when they worked with different UoDs (p=0,000 < α, H0UoD can be rejected in both experiments).
  • There were no significant differences between subjects of the two groups A and B. This is an indicator that the group to which subjects belonged did not affect the ease with which they understood.

Finally, in order to analyse the influence of the interaction of UoD and Usage of Patterns on understandability efficiency, a second and specific ANOVA analysis was performed with a significance level α fixed to 0,05. There were no significant differences of understandability efficiency found to come from the combined influence of the two independent variables (first experiment, p=0,183> α; second experiment α=0,062> α: H0PxUoD can be rejected in both experiments). As we can observe in box plots of Figures 3 and 4, the use of patterns allowed the obtaining of a better efficiency, irrespective of the UoD (except in UoD 1 in the first experiment where there is a very slight decrease in efficiency). This is a good insight into the usefulness of patterns for the improvement of understandability for all UoDs. It can also be deduced that different scores were obtained for each UoD, which indicates, furthermore, that the application domain can also affect the ease of understanding, as confirmed by statistical analysis.

5. Limits of the Experiments

In the planning process of the experiments we analysed the various issues that could threaten the validity of the results. With regard to construct validity, since the measures of dependent variable were obtained from the questionnaires filled in by subjects, the measurement was objective. We provided subjects with an example of a completed form and we explained the process carefully during the prior training session.

On the other hand, we tackled the aspects that could threaten the internal validity: the experiment was run with subjects who had never done a similar experiment so as to avoid persistence effects. The average duration of the experiments was an hour, which avoided the appearance of effects of fatigue; subjects were motivated to participate, since the experiments could bring benefits both as regards their training for the final examinations of the course (remembering that patterns and UML are topics included in their regular syllabus) and for their everyday work as future professionals. Plagiarism and influence among and between subjects were controlled by supervising the experimental runs.

In relation to external validity, the materials and tasks used were designed bearing in mind the restrictions of time and were fitted to an average level of students. However, the subjects were not professional designers and obviously we would expect much better results if the subjects were more experienced. The limited difficulty of the tasks and the different UoD’s do make the students become suitable experimental subjects, however, Despite the above remarks on the students’ lack of expertise, these individuals are much easier to work with than some others. Nevertheless, further replications of these experiments using people already working in the field will be considered, the purpose of which will be to find the confirmation of the obtained results. The diagrams of the material were relatively simple ones, due to the experimental settings and time available. In future, more realistic diagrams (representative of real projects) must be used.

6. Concluding Remarks

In the quest to improve micro architectures and understand the effect of using patterns, we have presented an empirical study. Our goal was to verify if the use of design patterns really does improve the understandability of design, and hence help to improve its maintainability. We found evidence that design patterns do indeed affect design understandability, measured through efficiency. These results constitute a good starting point towards building up a complete body of knowledge about this issue, as there has not been much effort made in the direction of empirical studies about micro-architectures.

Furthermore, as the MDA approach has reinforced an emphasis on the importance of models, we believe patterns can greatly contribute to building better applications from good quality models.

We are currently working on the replication of the empirical study, this time using professionals as subjects and employing more realistic material. The extent to which patterns can affect the modifiability of the design will be a further focus of future research.

7. Aknowledgements

This research has been partially supported by the projects: MECENAS (Junta de Comunidades de Castilla-La-Mancha, Consejeria de Educacion y ciencia, PBI06-0024) and ESFINGE (Dirección General de Investigación of the Ministerio de Educación y Ciencia, TIN2006-15175-C05-05).

8. References

  • [1] G. Wiederhold, “What is your software worth?” Communications of the ACM 49(9), 2006, pp. 65-75.
  • [2] R.L. Glass, “Facts and Fallacies of Software Engineering”, Addison Wesley, 2003.
  • [3] R. Reibing, “The impact of Pattern Use on Design Quality”, Proc. Of OOPSLA Workshop Beyond Design: Patterns (mis)used, Tampa Bay, Florida, USA.
  • [4] P. Wendorff, “Assessment of Design Patterns during Software Reengineering: Lessons Learned from a Large Commercial Project”, Proc. of the Fifth European Conference on Software Maintenance and Reengineering (CSMR), Lisbon, Portugal, IEEE Computer Society, pp. 77-84.
  • [5] Selic, B., “The Pragmatics of Model-Driven Development”, IEEE Software, 20 Vol. 5, 2003, pp. 19-25.
  • [6] E. Gamma, R. Helm, R. Johnson, J. Vlissides, “Design Patterns: Elements of Reusable Object-Oriented Software”, Addison-Wesley Professional, 1995.