Hugging Face shows how 9.3M LoRA weights lifted a 4B 3D model’s shoe scores

Hugging Face has published a step-by-step guide to fine-tuning 3D generation models. Its worked example adapts Microsoft’s TRELLIS.2 to footwear with LoRA adapters of about 9.3 million parameters, lifting the mean F-score on nine held-out shoes from 0.276 to 0.803, according to the authors.

Two of three stages get adapters, one stays frozen

The post, written by Suvaditya Mukherjee and merve, uses TRELLIS.2, a 3D generator that builds an asset in three stages: a sparse structure, then a shape, then a texture. The guide attaches LoRA adapters through Hugging Face PEFT to the first two stages only. The sparse-structure flow transformer gets rank-8 adapters in its attention and MLP layers. The shape flow transformer gets rank-16 adapters in its attention layers. Together they add about 9.3 million trainable parameters.

The texture stage is left untouched. The authors explain that their dataset has diffuse color maps but lacks the separate material maps that stage needs for training. The guide adds that the diffuse maps contain baked lighting. The post sums up the lesson in a meme graphic: “Choosing what to freeze matters as much as choosing what to fine-tune.”

Training uses the same rectified flow matching objective as pretraining, with AdamW, an effective batch size of 16 (4 samples times 4 gradient accumulation steps), adapter weights in fp32 and bf16 autocast for the forward pass. One practical detail: PEFT’s usual PeftModel() wrapper does not work with TRELLIS.2’s custom forward pass, so the authors inject adapters with inject_adapter_in_model() instead.

Eleven GPU hours on 188 scanned shoes

The data comes from Google Scanned Objects, a set of 1,030 items, of which 188 footwear objects were used: 179 for training and 9 held out. Each object was rendered from 16 Blender views, and one random view is picked per training sample. Preprocessing took about an hour on a single A100 80 GB through Hugging Face Jobs. Sparse-structure training took 7.2 hours and shape training 3.6 hours, so the adapters took roughly 11 hours of training in total.

On the nine held-out objects, the authors report these results for the base model and the model with both adapters:

• Mean F-score at 0.02: 0.276 base, 0.803 with adapters
• Chamfer distance (lower is better): 0.174 base, 0.033 with adapters
• Normal consistency: 0.530 base, 0.804 with adapters

The post says the F-score improves on every one of the nine objects. In the qualitative comparison, it says the soles from the fine-tuned model sit closer to the scans.

Who can run this today, and what is unproven

Everything needed is public. The guide ships two container images, one for preprocessing and one for training, plus a companion repository with standalone preprocess.py and finetune.py scripts. Jobs run with hf jobs run on the a100-large flavor, with timeouts of 6 hours for preprocessing and 12 hours for training, or locally through Docker with docker run --gpus all. Users supply their own meshes uploaded to the Hub, and the sparse-structure and shape stages are trained with separate --config ss and --config shape runs. The trained adapters and the preprocessed footwear latents are published on the Hub. The image encoder, DINOv3-L, is gated and requires access approval. Hugging Face Jobs is a paid service, and the post’s page does not give a cost estimate.

The evidence has limits. The test set is nine objects from one category, scored by the authors themselves, and nothing here has been independently reproduced. The adapters also leave texture quality unchanged by design, so the gains concern geometry only. Whether the recipe carries to other categories, or to datasets that include full material maps, is not shown.

Why a small adapter can move a large generator

The angle here is technical. According to the TRELLIS.2 repository, the model has about 4 billion parameters, is released under the MIT license, and outputs from 512³ to 1536³ voxels. Microsoft lists generation times on an Nvidia H100 of about 3 seconds at 512³, 17 seconds at 1024³ and 60 seconds at 1536³, and says the model needs a GPU with at least 24 GB of memory. Against that, the 9.3 million trainable weights are a fraction of a percent of the total, which is why one A100 80 GB is enough for the job.

The difficulty with 3D is that a generic LoRA recipe does not transfer cleanly. The model’s structured voxel representation, its custom forward pass and its separate geometry and appearance stages all affect where adapters can go, which is why the guide spends its effort on what to train and what to freeze. The authors also vary the noise schedule by stage, sampling timesteps logit-normally for the sparse structure and uniformly for the shape.

The release lands in a week of 3D activity. Nvidia’s 117M-parameter Déjà View reconstructs scenes from photos but is limited to research use. TRELLIS.2’s MIT license, by contrast, leaves room for commercial fine-tunes, which is what makes a reproducible domain recipe useful for product teams that need accurate shapes of a narrow class of objects, such as footwear.