Engineers often use visual-language models to generate new designs, such as for airplane or automobile components. To simulate how those components will perform in realistic conditions, they’ll use tried-and-true computer-aided design (CAD) software to create 3D models of those designs, which they can then put through virtual crash or durability tests.
Researchers at MIT and elsewhere have now developed a system that can teach a vision-language model to automatically convert 2D designs into CAD programs that is more accurate and functional than other approaches, while using only a fraction of the computation.
By improving the performance and efficiency of AI-powered CAD generation, this technology can streamline the rapid prototyping process and reduce costs. It can also help engineers identify beneficial design options that they might otherwise overlook.
The system generates new data based on the capabilities of the model as it attempts to convert the 2D image into a CAD program. The framework fixes model failures and includes them in the dataset along with their successful solutions.
It uses these data to teach the model how to fix specific mistakes and deal with trickier problems that it would grapple with on its own.
“We want engineers to be able to point our framework at poorly performing CAD models, set a computation budget, and let the system transform the model’s own mistakes into better training data,” says lead author Giorgio Giannone, a research associate in the Design Computation and Digital Engineering (DECODE) Lab at MIT and a principal research scientist on the AI Innovation team at Red Hat.
He is joined on the paper by Anna Claire Dorris, a mechanical engineering graduate student at MIT; Amin Herrani Nobari, an MIT postdoc; Kai Xu of RedHat; and co-senior author Akash Srivastava, director of Core AI at IBM and a principal investigator at the MIT-IBM Computing Research Lab; and Faiz Ahmed, associate professor of mechanical engineering at MIT, leader of the DeCoDE Lab, and a principal investigator in the MIT-IBM Computing Research Lab. This research was recently presented at the International Conference on Machine Learning.
“Almost every physical product around us, from airplanes to appliances, begins its life as a CAD model. Industry teams are eager for AI that can help accelerate the creation of these designs, but today’s models often produce shapes insufficiently simple for practice. What excites me about this work is that it gives many image-to-CAD-code models a way to improve themselves, instead of waiting for more human-generated data. Instead it learns from its errors – and this brings reliable AI design tools much closer to everyday engineering,” says Ahmed.
model-aware data
Researchers are working towards creating a vision-language model (VLM) for CAD generation. These VLMs take a 2D image and some descriptive text, and output Python code that can be executed in a CAD software program to produce a 3D model of a physical object.
They studied the challenges of deploying existing VLMs for this task and determined that the main obstacle limiting their capabilities is the lack of diverse, high-quality CAD datasets to train them.
To address this, they tried to create new data to teach a model how to perform CAD generation using a process called data augmentation.
In data augmentation, scientists typically create new data by randomly altering existing data to generate more samples, often by adjusting the color, size, and shape of objects in images.
Instead, the MIT researchers created a data augmentation system called GIFT (which stands for Geometric Estimate Feedback Tuning), which generates data designed to improve the performance of a VLM for a specific task.
Develops an understanding of the strengths and weaknesses of the GIFT model by testing it. It then uses this knowledge to generate data that can improve the model’s performance on the CAD generation problems it struggles to solve.
“We want to get data enrichment that is informed by the model itself,” says Giannone.
learning from mistakes
To do this, GIFT asks the model to generate code that solves the CAD generation problem multiple times in parallel. It tests the validity of these assumptions to understand how well the model can solve the problem.
“For a model, generating CAD query code is approximately It’s not that hard to get right, but producing code that is completely correct and can be executed is much more challenging for a standard VLM,” says Giannone.
For guesses that are approximately correct, GIFT adjusts them to create successful solutions. It saves these “near misses” and successful solutions into a new dataset that can teach the model how to overcome the problems that commonly trouble it.
“If we sample the model 10 times and it generates 10 correct answers to the same problem, there’s not much to learn. We care about the cases in between, where the model can only solve the problem 50 percent of the time,” he says.
Using these in-between cases allows GIFT to generate data enrichments that are both model-aware and task-aware. Furthermore, by including multiple correct solutions to the same problem, the new data expands the general knowledge of CAD code generation models.
This automated system does not require human intervention to correct model errors.
GIFT creates data augmentations from pre-trained VLMs using a process called inference-time scaling. This process allows a static model that has already been trained to produce better output without the high computational cost of re-training the entire model.
Using estimate-time scaling, the user can determine how much computation they want to use for GIFT, tailoring it to their time and budget constraints.
GIFT outperformed many competing techniques, producing CAD programs that were more accurate while using only 20 percent more calculations. The CAD models generated by VLM using GIFT were better aligned with the shape of the ground truth models.
“With GIFT, we started with geometry because with engineering problems, if the geometry of a 3D shape isn’t right, nothing will go right, but there are so many other aspects to consider,” says Giannone.
In the future, researchers want to expand GIFT so that the framework can teach modelers to generate CAD programs that improve 3D model performance and manufacturability. They also want to apply this system to larger models and more diverse CAD generation tasks.
This research was partially funded by the MIT-IBM Computing Research Lab.
