Evaluation of Design Feedback Modality in Design for Manufacturability

Representation of information influences the internal mental representation of designers during design [1]. Communicating ideas between designers can be influenced by the modality of the information exchanged [2]. The quality and modality of information affect interpretability [3]. This study evaluates how different manufacturing feedback modalities affect performance, confidence, workload, and usability.

Critical decisions, such as materials and product complexity, are fixed during the conceptual design of components that determine a significant percentage of total manufacturability cost [4]. Designers motivated to meet functional goals may lack critical knowledge of suitable tooling, fixtures, machining processes, and manufacturing equipment. Designing a manufacturable product without collaboration with manufacturing engineers is challenging [5].

As project teams grow larger and more distributed, designers and manufacturing engineering are often not co-located and have limited interaction [6]. To alleviate these problems, design for manufacturability (DFM) techniques provide early manufacturability guidance to designers, promote good practice by defining empirical rules to follow, and provide manufacturability measures [7]. Current state-of-the-art DFM tools perform complex analyses and provide feedback ranging from simple textual information to complex simulations. To enable an improvement in redesign, feedback should be in a language understandable to designers.

Although some DFM tools are moving toward advanced three-dimensional (3D) visualization (e.g., see Refs. [8] and [9]), textual (e.g., see Refs. [10] and [11]), and two-dimensional (2D) (e.g., see Refs. [8] and [12]), feedback modalities are still commonly used. Providing effective feedback is an important component of DFM tools, however, most researchers focus on developing manufacturability analysis algorithms (e.g., see Refs. [13] and [14]). Empirical evidence is needed to understand how a designer's performance is affected by the feedback modality. Benefits in performance and workload have been shown to be very dependent on the design of representations [15] and degree to which feedback format is matched to information processing requirements [16,17]. Studies have shown that information modality affects interpretability, usability, user performance, and workload. Many architecture and manufacturing industries make intensive use of text and technical 2D drawings [18]. Verbal, textual, and visual depictions, ranging from text to 2D sketches to 3D models, act as external representations that can decrease memory load and support the design process [19]. Research on 3D displays has shown mixed effects on performance, speed, accuracy, and memory. Furthermore, people differ in their 3D visualization abilities [18], and it is particularly hard for novices to interpret 2D sketches [20]. Interpretation of 3D from 2D figures can result in low accuracy and high reaction time, because of high working memory requirements [21]. The use of 3D stereoscopic visualization resulted in only slightly better reaction time and accuracy compared to using a complex engineering drawing [22]. The goal of this study is to quantify the differential impacts of text, 2D, and 3D manufacturing feedback in the early design phase of conceptual designers, who often get limited feedback in informal text modalities.

It is an open question whether the precision of numbers is better than graphical depictions of size and location. In dynamic decision-making tasks, graphical feedback has been shown to increase performance but a tabular textual format demonstrated stronger support for learning [15,16,23]. Mixed results of graphical versus tabular text data have been explained by researchers through the theory of cognitive fit, which posits that the problem-solving efficiency and effectiveness are dependent on the fit between the task and the problem representation [24]. Humans have two information processing systems: (1) a slower, more logical analytical system and (2) a faster, more associative perceptual system [25]. Tabular text formats may better support the analytical approach given its precision of numbers. Tabular data are symbolic and support precise estimates with specific data [26] that can be easier to process [27]. For complex systems with uncertainty, tabular presentation of feedback may be a more precise and accessible [28]. Graphical formats may better support the perceptual system given the integration of the feedback into patterns and context [16]. Graphical representation is spatial and supports reasoning over relationships between data. Furthermore, research in decision-making and information representation finds different advantages between 2D and 3D graphical data, where 2D is often preferred [29] but 3D can improve decision accuracy [30]. Task complexity interacts with both formats to moderate the effects of time and precision [31]. It is not clear that one feedback format is always better than another for every task. Matching the cognitive processing induced by the task and by the feedback may favor one format over another [17].

Novice designers have been shown to mostly depend on trial and error method and have very limited design strategies [32] and therefore are expected to benefit most from DFM feedback. Providing designers with usable feedback is expected to streamline the overall design process, expedite redesign, and decrease redesign iterations to achieve high-quality manufacturable designs. A prototype manufacturability feedback tool developed to provide design engineers with early, fast, and usable feedback is described next. The remainder of this paper presents a user study, which was conducted with novice design engineers to evaluate how different manufacturability feedback modalities (textual, 2D, and 3D) affect performance, workload, confidence, and usability.

Three-dimensional integrated feedback (3DIF) is a generalizable visualization tool coupled with different DFM analyses (casting, machining, welding, or a combination). Three-dimensional integrated feedback provides manufacturability feedback as colored 3D data and supplementary textual information in a portable document format.

Figure 1 shows 3DIF visualizations of four different cast-ana (casting analysis) results [33]: constant cross section (top left), isolated heavy section (top right), visibility analysis (bottom left), and cored areas (bottom right). Constant cross sections are areas of consistent thickness that can result in shrinkage porosity. Avoidance of porosity increases cost. Isolated heavy sections require a riser to feed liquid metal during solidification. Creating and removing risers increase cost. Visibility analysis reveals hidden features (e.g., undercuts and cavities) when viewed from certain orientations. Casting these requires cores. Cored areas need extra pieces of mold material (sand cores) to be cast (e.g., severe undercuts and hollow inner cavities). Cores increase time and cost.

H1: Early manufacturing feedback in any modality will help novice design engineers to eliminate more existing problems with lower mental workload compared to no-feedback.

H2: The number of new manufacturability problems introduced during the redesign process is independent of feedback modality.

H3: The 3D modality will help novice design engineers to redesign faster, eliminate more existing problems, and experience less mental workload and higher confidence compared to other feedback modalities.

H4: Usability with 3D feedback will be higher compared to other modalities.

Participants were recruited through email and posted advertisements. Participants were not compensated. A power analysis of the first ten subjects indicated a minimum sample size of 13 to detect a significant effect of graphical versus textual feedback with power of 0.80 and alpha of 0.05. The study was conducted with 24 participants (23 males and 1 female). Age averaged 22.8 yr (range 19–28). Participants were advanced engineering students trained in computer-aided design (CAD) design and casting and averaged 11.4 months (range 0.25–42) of professional casting experience. Experience was non-normally distributed (skewness = 2.03 and SE = 2.01). Only 4 of the 24 participants were familiar with manufacturing analysis tool feedback in any modality (all 3D). The other 20 participants analyzed designs for manufacturability through self-analysis (11), analysis by team/mentor/supervisor (5), none (4), oral feedback (4), and 2D sketch feedback (2). The design task was performed on a PC equipped with the participant's choice of CAD system. Three participants took part remotely using teamviewer for remote screen and file sharing and voice-over-internet-protocol calls.

Pilot studies established the level of training, ensured high usability for all the modalities, and established complexity equivalence between part models. The resulting methods were initially described in Ref. [34]. Participants were given four CAD models of bracket parts (see Fig. 2), one per trial, with casting design problems. In each trial, participants had access to the 3D model in their own CAD system. Using the feedback provided, participants redesigned the model to reduce the area of consistent cross sections, remove or decrease the area of isolated heavy sections, and eliminate cores. Redesign constraints required that the redesigned model (1) be within 10% of the initial weight, (2) lie within a shape envelope, and (3) meet stress constraints. Thus, participants could not alter the weight or topology of the part significantly or use external tools to verify their design. Participants were instructed to improve manufacturability as best they could in 20 min. They stopped once satisfied or when they finished their last design change after reaching the time limit. Of the 96 total trials, 40 (41.6%) reached the 20 min limit.

There were four levels of feedback modality: none, textual, 2D, and 3D. The feedback focused on isolated heavy sections, consistent cross sections, and cored areas. The textual, 2D, and 3D feedback had equal amounts of casting analysis information and were designed to be highly usable. The goal was to test the best example of each modality, based on two principles: (1) information equivalency and (2) best format to support task performance.

Studies that compare modalities must ensure that each modality conveys the same information. Tversky et al. [15] stated that “research on static graphics has shown that only carefully designed and appropriate graphics prove to be beneficial for complex system. Effective graphics… [are those in] which the content and format of the graphic should correspond to the content and format of the concepts to be conveyed.” Often when differences were found, it is because there was a difference in information content [15]. Thus, for each flaw, we provided the same information (location, size, and type) in the text, 2D, and 3D modalities.

The information format of each modality was carefully designed to be highly usable. Three-dimensional feedback was given in 3DIF. The 2D feedback used 3DIF with rotation disabled, resulting in six orthogonal projection views: top, bottom, left, right, front, and back. Textual feedback was tested in pilot studies to ensure that participants could understand the feedback and use it effectively with the CAD view of the part. Textual feedback contained separate tables for each analysis. For complex technical information, research has found that critical parameter information is easier to select and access in table form than in prose [35]. Prose tends to be slower to use and more error prone than tables [36]. The region of interest was provided as (x,y,z) coordinates. Separate columns provided volume and surface area information.

Dependent variables are listed in Table 1. Each model design had controlled casting design problems. Every problem had a cost based on the area of consistent cross sections, number and area of attachment for isolated heavy sections, number of cores, and overall weight. After redesign, the part was reanalyzed with the same cast-ana software that generated the initial manufacturing feedback. Comparing the second set of cast-ana results to the first identified flaws eliminated and identified new flaws introduced. For each usability measure, participants were asked to distribute 100 points between all the three modalities of feedback.

The study is a 1 × 4 (feedback modality) within-subjects design. Participants performed four trials; each with a different feedback modality and a different part model. Pilot tests established that each of the four parts was of a similar design complexity, and the model complexity level was neither too simple (resulting in overconfidence and a ceiling effect) nor overly complex (resulting in disengagement). Trial order and part assignment were counterbalanced to reduce learning effects.

After consent and demographics questionnaire, participants reviewed casting imperfections. They trained on the feedback types, cost model, and workload survey. Participants conducted a training questionnaire related to feedback modality. They proceeded to the practice task only when they scored 80% or higher. They were corrected on mistakes before proceeding. Participants practiced with a sample design until comfortable. Participants trained for 20 min on average redesigning a sample part; overall training averaged 40 min. Each of the four trials of the design task was followed by a workload and confidence survey. A postexperiment usability questionnaire was followed by a debriefing to reiterate study purpose and answer questions.

A within-subjects, repeated measures analysis of variance (ANOVA) was performed. A factor was considered highly significant for p < 0.001, significant for 0.001 < p < 0.05, and marginally significant for 0.05 < p < 0.10. A post hoc Tukey's honest significant difference distinguished pairwise means significantly different from each other. Cohen's d calculated effect size, which was reported as small for 0.20 < d < 0.50, medium for 0.50 < d < 0.80, and large for d > 0.80.

Results are presented in Table 2. Tukey's honest significant difference post hoc comparisons between groups were significantly different when they do not share the same letter. Trial order and part number had no significant effect on any dependent variable.

Hypothesis H1 was partially supported. Participants eliminated significantly more existing problems provided feedback in 2D and 3D compared to no-feedback. Providing textual feedback was not significantly better than no-feedback. Mental workload in textual feedback was very high compared to other modalities. Textual feedback may be better than no-feedback when redesign time is not limited.

Hypothesis H2 was fully supported. This was expected because the feedback did not provide redesign suggestions, and so redesign decisions were made by participants based on their expertise. It is anticipated that iterative redesigning could have a learning effect on the ability to identify features of interest. Designers could potentially improve over time and introduce fewer design problems in subsequent iterations.

Hypothesis H3 was partially supported. Participants eliminated more existing problems with less mental workload in 3D compared to textual and no-feedback. However, while on average 3D resulted in greater performance, lower mental workload, and higher confidence when compared to 2D, the differences did not rise to the level of significance, given the limited number of participants. The only difference between 3D and 2D was the ability to rotate the model in 3D versus selecting between orthogonal 2D views. The 3D feedback may have better performance than 2D when designs are highly complex. The small difference in mental workload may be because most of the workload came from redesigning the model and not much from interpreting feedback. Mental workload may be affected by model complexity, where interpretation in 2D may cause more mental workload than 3D.

Hypothesis H4 was fully supported. Subjective usability ratings of comprehensiveness, helpfulness, ease of navigation, learning, and preference was higher in 3D compared to other modalities. Textual feedback was rated least usable. While use of representations from words to pictures increases the user's ability to visualize, impact is moderated by the degree of abstraction and degree of difficulty in representation manipulation [20]. The text modality was an abstract, symbol-based representation that mapped information to numeric digits, which is not an intuitive way to comprehend spatial information. Two-dimensional and 3D feedback mapped spatial information to colored geometric regions that were easier to comprehend. Three-dimensional provided a higher level of user interactivity than 2D, in terms of model rotation, and so was rated more usable. Interactivity may be a key component for designing preferable and usable visualization tools.

Participants were given part models of medium complexity and located/eliminated design problems easily using both 2D and 3D feedback; however, textual feedback caused the highest workload and a performance level not significantly different from no-feedback. This underscores the importance of providing feedback that is readily interpretable and locating feedback within the context of the part in an easy-to-visualize manner. Participants strongly preferred 3D feedback.

Initial conceptual design can be influenced by the representation used to exchange information. For instance, in a study of undergraduate mechanical engineering students, group idea generation techniques during the initial design phase were influenced by how concepts were viewed and the mode used to communicate ideas, where modes varied from written words only, sketches only, or a combination [2]. As designers and manufacturing engineers have become more distributed, each works more in isolation. Methods to infuse manufacturing considerations into the conceptual design process use DFM tools to analyze and communicate manufacturing feedback to design engineers. This work demonstrated that representation of manufacturing feedback in different modalities impacted novice design engineers' performance, workload, and confidence. Graphical feedback was significantly better than nongraphical modalities in eliminating design flaws, lowering workload, and increasing the confidence in the designs. In conceptual design, 3D spatially based feedback was more effective than number-based text feedback. Furthermore, higher interactivity in 3D was a key factor for being perceived as more usable than 2D. Mentally constructing internal 3D representations from 2D or text feedback can be cognitively challenging, especially for novices [20]. This has implications in the design of DFM tools as they continue to move toward integrated 3D representations of manufacturing analysis feedback. Future work will explore the impact of varying part complexity, different levels of participants' expertise, and mixed modalities.

Casting feedback was generated by CAST-ana software developed at Iowa State University and University of Alabama-Birmingham.

• Defense Advanced Research Projects Agency (Contract No. HR0011-12-C-0075)