A new study by engineers at MIT has revealed that if ChatGPT is to excel at engineering design it must be taught to transcend statistical similarity and learn to innovate.
ChatGPT and other deep generative models are proving to be uncanny mimics, excelling at generating new content that resembles something that they’ve seen before. But as the new study shows, similarity isn’t enough if you want to truly innovate in engineering tasks.
Advertisement
‘Deep generative models (DGMs) are very promising, but also inherently flawed,’ said mechanical engineering graduate student Lyle Regenwetter. ‘The objective of these models is to mimic a dataset. But as engineers and designers, we often don’t want to create a design that’s already out there.’
Regenwetter and his colleagues make the case that if mechanical engineers want help from AI to generate novel ideas and designs, they will have to first refocus those models beyond ‘statistical similarity’.
Advertisement
‘The performance of a lot of these models is explicitly tied to how statistically similar a generated sample is to what the model has already seen,’ said Faez Ahmed, an assistant professor of mechanical engineering at MIT. ‘But in design, being different could be important if you want to innovate.’
In their study, Ahmed and Regenwetter revealed the pitfalls of DGMs when they’re tasked with solving engineering design problems. In a case study of bicycle frame design, the team showed that these models end up generating new frames that mimic previous designs but falter on engineering performance and requirements.
Advertisement
When the researchers presented the same bicycle frame problem to DGMs that they specifically designed with engineering-focused objectives, rather than only statistical similarity, these models produced more innovative, higher-performing frames.
The team’s results show that similarity-focused AI models don’t quite translate when applied to engineering problems. But, as the researchers also highlighted in their study, with some careful planning of task-appropriate metrics, AI models could be an effective design ‘co-pilot’.
‘This is about how AI can help engineers be better and faster at creating innovative products,’ Ahmed said. ‘To do that, we have to first understand the requirements. This is one step in that direction.’
Advertisement
Because of their ability to learn from data and generate realistic samples, DGMs have been increasingly applied in multiple engineering domains. But for the most part, the models have mimicked existing designs, without improving their performance.
‘Designers who are working with DGMs are sort of missing this cherry on top, which is adjusting the model’s training objective to focus on the design requirements,’ Regenwetter said. ‘So, people end up generating designs that are very similar to the dataset.’
Advertisement
In the new study, the researchers outlined the main pitfalls in applying DGMs to engineering tasks, and showed that the fundamental objective of standard DGMs doesn’t take into account specific design requirements. To illustrate this, the team invoked a simple case of bicycle frame design and demonstrated that problems can crop up as early as the initial learning phase. As a model learns from thousands of existing bike frames of various sizes and shapes, it might consider two frames of similar dimensions to have similar performance, when in fact a small disconnect in one frame – too small to register as a significant difference in statistical similarity metrics – makes the frame much weaker than the other, visually similar frame.
The researchers carried the bicycle example forward to see what designs a DGM would actually generate after having learned from existing designs. They first tested a conventional ‘vanilla’ generative adversarial network – a model that has been widely used in image and text synthesis, and is tuned simply to generate statistically similar content. They trained the model on a dataset of thousands of bicycle frames, including commercially manufactured designs and less conventional, one-off frames designed by hobbyists.
Once the model learned from the data, the researchers asked it to generate hundreds of new bike frames. The model produced realistic designs that resembled existing frames. But none of the designs showed significant improvement in performance, and some were even a bit inferior, with heavier, less structurally sound frames.
The team then carried out the same test with two other DGMs that were specifically designed for engineering tasks. The first model is one that Ahmed previously developed to generate high-performing airfoil designs. He built this model to prioritise statistical similarity as well as functional performance. When applied to the bike frame task, this model generated realistic designs that also were lighter and stronger than existing designs. But it also produced physically ‘invalid’ frames, with components that didn’t quite fit or overlapped in physically impossible ways.
‘We saw designs that were significantly better than the dataset, but also designs that were geometrically incompatible because the model wasn’t focused on meeting design constraints,’ Regenwetter says.
The last model the team tested was one that Regenwetter built to generate new geometric structures. This model was designed with the same priorities as the previous models, with the added ingredient of design constraints, and prioritising physically viable frames, for instance, with no disconnections or overlapping bars. This last model produced the highest-performing designs, which were also physically feasible.
‘We found that when a model goes beyond statistical similarity, it can come up with designs that are better than the ones that are already out there,’ Ahmed says. ‘It’s a proof of what AI can do, if it is explicitly trained on a design task.’
For instance, if DGMs can be built with other priorities, such as performance, design constraints and novelty, Ahmed foresees ‘numerous engineering fields, such as molecular design and civil infrastructure, that would greatly benefit. By shedding light on the potential pitfalls of relying solely on statistical similarity, we hope to inspire new pathways and strategies in generative AI applications outside multimedia.’
The research has been published in Computer Aided Design.