Hovimo
person

LoRA

LoRA — launched — FFT

Key facts

LoRA — launched — FFT [single source] [D1239]

Reported claims

It does not mean adding GPUs makes a job slower. [reported assertion] [D385]

That says nothing against Calculon as a step-level tool, which is what it was designed for. [reported assertion] [D385]

LoRA adapters on modules vLLM does not wrap are refused ( #63489 ); misconfigured NCCL weight transfer raises ( #57022 ). [reported assertion] [D547]

Figure 11: MMLU general-knowledge retention across hyperparameter configurations

Hence, it is important to note that FFT isn't universally superior to LoRA, and its effectiveness is extremely sensitive to hyperparameter choices. [reported assertion] [D1239]

Each targets a different dimension of model quality: semantic correctness, code generation fidelity, and general knowledge retention. [reported assertion] [D1239]

Figure 2: Radar chart comparing the base model, LoRA, and FFT across quality dimensions

The details of how we arrived at these results, the coding datasets used for model customization, and the experimentation process are explained in detail in the following sections. [reported assertion] [D1239]

As tasks grow more complex and datasets scale beyond tens of thousands of examples, LoRA hits an accuracy plateau where increasing rank (and thus trainable parameters) fails to bridge the accuracy gap with standard fine-tuning. [reported assertion] [D1239]

LoRA aims to strike a balance between efficiency and expressiveness, however, the low-rank nature of the weight update inherently limits the representation power of LoRA fine-tuned models, potentially compromising performance on complex tasks. [planned] [D1239]

Abstract:Federated low-rank adaptation (LoRA) enables clients with different resource budgets to collaboratively fine-tune foundation models. [reported assertion] [D841]

Timeline

  1. Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference [D1445][D1436][D1435]
  2. Post-Training NVIDIA Nemotron to Increase Accuracy in Your Scientific Domain [D662]
  3. NVIDIA Sweeps MLPerf Training and Endpoints as Vera Rubin Debuts With Leading Performance [D664][D395]
  4. Amazon SageMaker AI real-time endpoints now support KV cache aware routing [D1294]
  5. Why your fine-tuning job takes longer than the FLOPs say [D385]
  6. v0.34.0 [D547]
  7. Full Fine-Tuning (FFT) on SageMaker AI Serverless Model Customization [D1239]
  8. RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents [D1300]
  9. Improving LLM Final Representations with Inter-Layer Geometry [D1276]
  10. A Hybrid RAG Framework for Deceptive Review Detection: Integrating RoBERTa-LoRA, FAISS-Based Retrieval, and LLM Reasoning [D1175]
  11. Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models [D1001]
  12. Modeling LLM Reasoning Dynamics with Structural Representations [D951]
  13. Same Rank, Different Needs: Rethinking LoRA Redistribution in Federated Learning [D841]
  14. Learning through Discernment: when a small language model learns from user feedback on the job, and when it decays [D838]
  15. When Does Subspace Direction Matter for LoRA? Regime Analysis of the Magnitude Principle in Few-Shot Adaptation [D818]

Sources