Hot AI News

A FLOP estimate missed fine-tuning times by 12 hours; a CatBoost model missed by 1.7

Dharma-AI's 272 production fine-tuning runs on H200s show the standard compute estimate is far off, while a model using only pre-dispatch information cuts the mean error to 1.71 hours.

A FLOP estimate missed fine-tuning times by 12 hours; a CatBoost model missed by 1.7
Image: Hugging Face

Dharma-AI, a team publishing on Hugging Face, says the usual back-of-the-envelope FLOP estimate explains little of how long fine-tuning really takes. In a preview of its upcoming paper, a CatBoost model cut the mean error from 12.41 hours to 1.71 hours.

272 production runs on shared H200 GPUs

The data comes from 272 supervised fine-tuning jobs in a corporate production environment, about 7,382 GPU-hours in total. Each job used one, two or four accelerators from a shared fleet of between 8 and 32 NVIDIA H200s spread over up to four machines, alongside other training and inference work. Durations ranged from 1.6 to 172.4 hours, with a mean of 23.43 hours and a median of 15.29 hours. The runs covered several model families up to 15 billion parameters, both LoRA and full fine-tuning, using Hugging Face Transformers and Unsloth.

Mean absolute error in hours (lower is better)
Image: Hugging Face, chart by Aiholics

The baseline is the common compute formula: roughly 6 times parameters times training tokens, divided by GPU throughput and an assumed utilization. The authors added evaluation tokens to it to be fair, then swept utilization from 25% to 65%. Its best setting, 30%, reached an R² of 0.31 and a mean absolute error of 12.41 hours. At 45% the error was 13.01 hours, and at 60% it was 14.62 hours.

The learned models see only what is known before a job is dispatched: hyperparameters, dataset sizes, model identifiers and GPU allocation. They never see logs or partial progress. Derived features capture overheads such as evaluation volume, number of checkpoint saves, parameters saved and per-GPU token load. The FLOP estimate itself is fed in as one more feature. Queue time is excluded. In stratified 5-fold cross-validation, the final CatBoost model with 24 features scored an R² of 0.98, a root mean squared error of 3.52 hours and a median error of 0.51 hours. XGBoost came close at a 1.95-hour mean error, while Ridge regression stalled at 7.23 hours.

Distribution of end-to-end duration across the 272 fine-tuning runs
Image: Hugging Face

A mean error of 1.71 hours, against 12.41 hours for the FLOP estimate, on jobs that average 23 hours.

Advertisements

Evaluation and checkpoints matter more than expected

According to the post, the strongest predictor is num_eval_data, the total number of samples processed in evaluation over a job’s life. It is followed by trainable parameter count, evaluation batch size, total parameter count and the product of GPU count and training batch share. Checkpoint volume sits close behind. SHAP analysis shows high evaluation volume consistently pushing predicted time up. The authors stress that feature importance is not a causal claim, and that GPU count partly acts as a proxy for job size, so it does not show that more GPUs slow a job down.

A FLOP estimate missed fine-tuning times by 12 hours; a CatBoost model missed by 1.7
Image: Hugging Face

Accuracy depends on job length. For runs of 5 to 10 hours the median error is 0.18 hours, and for 30 to 60 hours it is 1.06 hours. Only 13 runs exceed 60 hours, and there the median error is 5.19 hours, with a 90th percentile of 18.51 hours. Overall, 80% of predictions land within 2.20 hours and 90% within 4.31 hours. The team reads the long tail as a data shortage and reports that error on a fixed holdout was still falling as training data grew.

Why a detailed simulator did not close the gap

The comparison with Calculon is the most instructive part. Calculon, from a 2023 Supercomputing paper by Mikhail Isaev, Nic McDonald, Larry Dennison and Richard Vuduc, takes three inputs: the model architecture, the execution strategy and the hardware description. It can estimate training time for a given setup in about a millisecond, according to the abstract of the paper. That speed is what makes it useful for searching large design spaces, which is a different job from forecasting a single production run.

Dharma-AI extended the public code to cover LoRA, grouped-query attention, fused attention, bfloat16 and SwiGLU. Even so, its mean absolute error was 20.36 hours and its R² was negative, at -0.51. The estimates covered a median of only 11% of the observed wall-clock time. The authors say this reflects what happens outside the optimization step: environment setup, model download, tokenization, periodic evaluation, checkpoint uploads and contention for network and storage on a shared cluster. A model that learns those costs from logs can beat a more physically detailed one that does not represent them.

That is the technical lesson, and it is a modest but practical one. Teams already hold the execution logs needed to repeat it. The authors say the full paper will give the feature formulas, selection procedure, search spaces and final hyperparameters so the pipeline can be rebuilt on any comparable dataset.

Advertisements

What the preview does not yet establish

No model, code or dataset is released with this post. The results rest on one organization’s logs, so they are the team’s own figures and not independently verified. The paper is described as coming to arXiv soon, with no date.

The authors list three limits. They tested only supervised fine-tuning, not RLHF, DPO or RLVR. Every run used H200s, so the fitted model should not be expected to transfer numerically to other GPU generations without retraining. A mixed-hardware predictor would need descriptors such as throughput, memory bandwidth and interconnect topology. Planned next steps include hardware-specific features, validation on other GPUs, transfer learning for rarely seen configurations and more data on long runs.

For anyone running a shared cluster, the practical takeaway is already visible in the numbers. Settings often chosen by habit, such as how often to evaluate and how many checkpoints to keep, can move a job’s wall-clock time as much as model size does. The post gives schedulers a reason to plan around learned durations when they decide what runs when.

Sign Up for the Daily AI Pulse

One email a day. All the stories that matter.

Daniel ReedAI Research, Safety & Ethics Analyst

Daniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…

Leave a Reply

Your email address will not be published. Required fields are marked *

You may also like