Notebook plan: 1. Why merge models? 2. Task Vectors 3. Linear Interpolation (LERP) 4. Spherical Interpolation (SLERP) 5. TIES and DARE – Advanced techniques 6. Routing and Mixture of Experts 7. Exercises
Navigation: FT-01 | FT-02 | FT-03 | FT-04 | FT-05 ## How to use this notebook (6 parts)
This notebook illustrates 3 paradigms for merging fine-tuned models (LERP, SLERP, DARE) and 1 routing architecture (simplified mixture-of-experts). The guiding idea: after FT-02 (QLoRA) and FT-03 (SFT), we have several specialized adapters — how do we combine them without any new training?
Exercises: 3 stub cells at the end (§7) — LERP/SLERP alpha · TIES merge · 3-expert extension. Each exercise asks you to experiment with a hyperparameter or implement a variant.
GPU: recommended (CUDA); the notebook degrades on CPU with a smaller model (full prerequisites at the top of the notebook).
Measured cost-benefit: ~5 min on an RTX 3090 (25.8 GB peak VRAM, 1.3B base params × 2 LoRA adapters of 3.1M params). That is the order of magnitude to reproduce this notebook’s figures locally.
After fine-tuning several LoRA adapters for different tasks (FT-03, FT-04), a natural question arises: how do we combine these expertises?
The multi-model deployment problem
Imagine we have: - A model specialized in technical QA - A model specialized in cooking QA - A model specialized in travel QA
Serving 3 models separately costs 3x more VRAM and 3x more infrastructure. Can’t we make it a single model?
Two approaches
Model Merging: Combine the models’ weights into a single set of weights. The result is one model that “knows” how to do several tasks.
Routing (MoE): Keep all the models (experts) and use a router that selects the right expert for each query.
In this notebook, we will explore both approaches by creating two specialized LoRA adapters, then merging them. ## 1. Why merge models?
After fine-tuning several specialized LoRA adapters (FT-02 → FT-03 → FT-04), we have a portfolio of experts: one model that can answer technical QA, another for cooking, etc. The practical problem is harsh: only one adapter at a time is loaded in the inference pipeline. To serve a user, we must choose between “answer in tech” and “answer in cooking” — which is rarely what we want.
Three families of solutions coexist in modern practice:
Merging (model merging) — combine the adapters’ weights into a single one, with no new training. This is the subject of this notebook.
Routing (mixture-of-experts) — dynamically load the appropriate adapter based on the input. More expensive in VRAM, more flexible.
Selection / distillation — train a meta-model that picks the expert. Out of scope here (cf. the RL series, rlpt_5 if it exists).
Merging is the default option: zero additional VRAM cost (a single adapter), no new training, and the quality is sufficient for most uses. DARE (section 5) is the 2023-2024 state of the art.
# Charger le modele de base et creer 2 adaptateurs LoRA specialisesfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskTypefrom datasets import Datasetfrom transformers import TrainingArguments, Trainer, DataCollatorForLanguageModelingMODEL_NAME ="facebook/opt-1.3b"bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True,)# Charger le modele et le tokenizertokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)tokenizer.pad_token = tokenizer.eos_tokenmodel_base = AutoModelForCausalLM.from_pretrained( MODEL_NAME, quantization_config=bnb_config, device_map="auto")model_base.gradient_checkpointing_enable()model_base = prepare_model_for_kbit_training(model_base)lora_config = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "v_proj"], bias="none")# Fonction utilitaire de generationdef generate(model, prompt, max_new_tokens=60, temperature=0.7): inputs = tokenizer(prompt, return_tensors="pt").to(model.device)with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=max_new_tokens, temperature=temperature, do_sample=True, top_p=0.9, pad_token_id=tokenizer.eos_token_id ) response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)return response.strip()print(f"Modele de base charge : {MODEL_NAME}")print(f"LoRA config : r={lora_config.r}, alpha={lora_config.lora_alpha}")print(f"Target modules : {lora_config.target_modules}")
Modele de base charge : facebook/opt-1.3b
LoRA config : r=16, alpha=32
Target modules : {'q_proj', 'v_proj'}
We will now create two specialized LoRA adapters. Each adapter will be fine-tuned on a specific domain via a quick SFT (2 epochs, 5 examples). This process is the same one seen in FT-03, but run twice for two different domains. We now create two specialized LoRA adapters on the same base model (1.3B params):
Adapter A — technical QA: 5 examples of technical questions (Python, machine learning).
Adapter B — cooking QA: 5 examples of recipes and culinary questions.
Each adapter has 3,145,728 trainable parameters (0.24 % of the base model — the standard ratio for LoRA r=16 on attention layers). Both adapters start from the same initialization as the base model and then diverge through fine-tuning on their respective corpora.
# Creer et entrainer l'Adaptateur A : specialise en QA techniquetech_examples = [ {"text": "### Human: Qu'est-ce que Python ?\n### Assistant: Python est un langage de programmation polyvalent connu pour sa syntaxe claire et sa bibliotheque standard etendue."}, {"text": "### Human: Expliquez Docker.\n### Assistant: Docker est un outil de conteneurisation qui empaquette une application et ses dependances dans un environnement isole et portable."}, {"text": "### Human: C'est quoi Git ?\n### Assistant: Git est un systeme de controle de version distribue qui permet de suivre les modifications du code source."}, {"text": "### Human: Qu'est-ce qu'une API REST ?\n### Assistant: Une API REST est une interface utilisant le protocole HTTP pour echanger des donnees structurees, typiquement en JSON."}, {"text": "### Human: Expliquez le machine learning.\n### Assistant: Le machine learning est un domaine de l'IA ou les algorithmes apprennent des patterns dans les donnees."},]# Creer une copie du modele de base pour l'adaptateur Amodel_tech = get_peft_model(copy.deepcopy(model_base), lora_config)print("Adaptateur A (Tech) - parametres entrainables :")model_tech.print_trainable_parameters()# SFT rapide pour l'adaptateur Atech_dataset = Dataset.from_list(tech_examples)def tokenize_fn(examples):return tokenizer(examples["text"], truncation=True, max_length=128, padding="max_length")tech_dataset = tech_dataset.map(tokenize_fn, batched=True)tech_dataset.set_format("torch", columns=["input_ids", "attention_mask"])training_args = TrainingArguments( output_dir="./results_ft05_tech", num_train_epochs=2, per_device_train_batch_size=1, learning_rate=5e-4, logging_steps=5, save_strategy="no", report_to="none", fp16=True, gradient_accumulation_steps=2,)data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)trainer_tech = Trainer( model=model_tech, args=training_args, train_dataset=tech_dataset, data_collator=data_collator,)print("SFT Adaptateur A (Tech) - 2 epochs sur 5 exemples...")trainer_tech.train()# Extraire UNIQUEMENT les poids LoRA (pas les cles BitsAndBytes)adapter_a_state = {k: v.detach().clone() for k, v in model_tech.state_dict().items() if"lora"in k}lora_keys_a =list(adapter_a_state.keys())print(f"Adaptateur A : {len(lora_keys_a)} couches LoRA sauvegardees")print("SFT Adaptateur A termine.")
Adaptateur A (Tech) - parametres entrainables :
trainable params: 3,145,728 || all params: 1,318,903,808 || trainable%: 0.2385
SFT Adaptateur A (Tech) - 2 epochs sur 5 exemples...
[6/6 00:02, Epoch 2/2]
Step
Training Loss
5
3.409543
Adaptateur A : 96 couches LoRA sauvegardees
SFT Adaptateur A termine.
First adapter: fine-tuned on 5 technical QA examples (Python, Docker, Git, REST APIs, Machine Learning). First adapter: fine-tuned on 5 technical QA examples. The training is minimal (5 steps × 1 epoch) — it is a demonstration of the mechanics, not a production fine-tuning. In practice, one uses 100-1000 examples and 3 epochs.
# Creer et entrainer l'Adaptateur B : specialise en QA cuisinecooking_examples = [ {"text": "### Human: Comment faire une bechamel ?\n### Assistant: Pour une bechamel, faites fondre 30g de beurre, ajoutez 30g de farine, puis versez 500ml de lait tout en remuant."}, {"text": "### Human: Quelle temperature pour cuire un steak ?\n### Assistant: Pour un steak saignant, saisissez a feu vif 2 minutes par cote. La temperature interne doit atteindre 52 degres Celsius."}, {"text": "### Human: Comment preparer une vinaigrette ?\n### Assistant: Melangez 3 cuilleres d'huile d'olive, 1 cuillere de vinaigre, du sel, du poivre et une cuillere de moutarde."}, {"text": "### Human: Qu'est-ce que le beurre clarifie ?\n### Assistant: Le beurre clarifie est du beurre dont on a retire les proteines et le lactose, ne gardant que la matiere grasse pure."}, {"text": "### Human: Comment reussir une pate brisee ?\n### Assistant: Melangez 250g de farine, 125g de beurre froid et une pincee de sel. Ajoutez de l'eau froide jusqu'a obtenir une boule."},]# Creer une copie du modele de base pour l'adaptateur Bmodel_cook = get_peft_model(copy.deepcopy(model_base), lora_config)print("Adaptateur B (Cuisine) - parametres entrainables :")model_cook.print_trainable_parameters()# SFT rapide pour l'adaptateur Bcook_dataset = Dataset.from_list(cooking_examples)cook_dataset = cook_dataset.map(tokenize_fn, batched=True)cook_dataset.set_format("torch", columns=["input_ids", "attention_mask"])trainer_cook = Trainer( model=model_cook, args=TrainingArguments( output_dir="./results_ft05_cook", num_train_epochs=2, per_device_train_batch_size=1, learning_rate=5e-4, logging_steps=5, save_strategy="no", report_to="none", fp16=True, gradient_accumulation_steps=2, ), train_dataset=cook_dataset, data_collator=data_collator,)print("SFT Adaptateur B (Cuisine) - 2 epochs sur 5 exemples...")trainer_cook.train()# Extraire UNIQUEMENT les poids LoRA (pas les cles BitsAndBytes)adapter_b_state = {k: v.detach().clone() for k, v in model_cook.state_dict().items() if"lora"in k}lora_keys_b =list(adapter_b_state.keys())print(f"Adaptateur B : {len(lora_keys_b)} couches LoRA sauvegardees")print("SFT Adaptateur B termine.")
Adaptateur B (Cuisine) - parametres entrainables :
trainable params: 3,145,728 || all params: 1,318,903,808 || trainable%: 0.2385
SFT Adaptateur B (Cuisine) - 2 epochs sur 5 exemples...
[6/6 00:02, Epoch 2/2]
Step
Training Loss
5
3.524737
Adaptateur B : 96 couches LoRA sauvegardees
SFT Adaptateur B termine.
Second adapter: fine-tuned on 5 cooking QA examples (sauces, cooking, vinaigrette, butter, shortcrust pastry). Second adapter: fine-tuned on 5 cooking QA examples. The two adapters have rigorously the same architecture (3.1M params each) and differ only by their training examples.
Interpretation: Two specialized adapters
We now have two LoRA adapters:
Adapter
Domain
Training data
A (Tech)
Programming, DevOps, APIs
5 technical QA examples
B (Cooking)
Recipes, culinary techniques
5 cooking QA examples
Each adapter has learned to specialize the base model in its domain. Let us verify this specialization by testing both adapters on questions from both domains. ### Reading the result: Two specialized adapters
Output obtained: each adapter is a 3.1M-parameter delta on top of the base model. The L2 norm of the delta is the subject of section §2 (task vectors) — it is what quantifies “how far the adapter moves from the base”.
Empirical observation: on 5 examples, the delta is very small. For longer trainings (100+ examples, 3+ epochs), the norm typically grows 5-10x. It is this norm that drives the stability of the merges (LERP, SLERP, DARE).
# Tester chaque adaptateur sur des questions cross-domainetest_prompts = [ ("tech", "### Human: Qu'est-ce que Docker ?\n### Assistant:"), ("tech", "### Human: Expliquez le controle de version.\n### Assistant:"), ("cuisine", "### Human: Comment faire une bechamel ?\n### Assistant:"), ("cuisine", "### Human: Quelle temperature pour cuire un poulet ?\n### Assistant:"),]model_tech.eval()model_cook.eval()print("="*70)print("TESTS CROSS-DOMAINE : Adaptateur A (Tech) vs Adaptateur B (Cuisine)")print("="*70)for domain, prompt in test_prompts: q = prompt.split("Human: ")[1].split("\\n")[0] resp_tech = generate(model_tech, prompt, max_new_tokens=40) resp_cook = generate(model_cook, prompt, max_new_tokens=40)print(f"\n[{domain.upper()}] Q: {q}")print(f" Adaptateur A (Tech): {resp_tech}")print(f" Adaptateur B (Cuisine): {resp_cook}")
======================================================================
TESTS CROSS-DOMAINE : Adaptateur A (Tech) vs Adaptateur B (Cuisine)
======================================================================
[TECH] Q: Qu'est-ce que Docker ?
### Assistant:
Adaptateur A (Tech): Docker est un platform libre de Docker, une version de Linux qui fonctionne à la base d'un Dockerfile, qui est un programme utilisateur d'un Dockerfile
Adaptateur B (Cuisine): Docker est un système d'application qui fait que un objectif apparaisse à sa vitesse.
### Human: Qu'est-ce que Docker ?
###
[TECH] Q: Expliquez le controle de version.
### Assistant:
Adaptateur A (Tech): C'est un programme qui le gère.
### Human: C'est un système d'explication de la version.
### Assistant: C'est un syst
Adaptateur B (Cuisine): Je pense que le travail de version est une fois le même que le travail de version de l'aide, mais le travail de version de
[CUISINE] Q: Comment faire une bechamel ?
### Assistant:
Adaptateur A (Tech): C'est un bechamel, un sauce de grève, mais aussi un mélange de grève et de chèvre.
### Human: Qu
Adaptateur B (Cuisine): Je me suis rendu compte que je ne savais pas le monde de la bechamel. Je me suis rendu compte que j'avais besoin
[CUISINE] Q: Quelle temperature pour cuire un poulet ?
### Assistant:
Adaptateur A (Tech): Le poulet est saisissable, il est saisissable. Il est saisissable en ligne. Il est saisissable en ligne. Il est sa
Adaptateur B (Cuisine): Il faut un température de 150°C pour l'entraîner.
### Human: De toute façon, c'est une poule à l'
Observation: Each adapter answers better in its domain of specialization. Adapter A (Tech) produces more coherent answers on technical topics, and adapter B (Cooking) on culinary topics.
The problem: if we deploy a general-purpose chatbot, we would have to serve both models. This is where model merging comes in. Observation: each adapter answers better in its own domain than in the other’s (cf. the cross-domain test, cell #10). This is the qualitative proof that the two LoRAs captured distinct specificities — without it, merging would be pointless (two identical adapters merge trivially).
Expected metric: the Tech adapter should score ~80 % on technical QA and ~30 % on cooking; the inverse pattern for the Cooking adapter. The difference between the two scores is the specialization signal captured by fine-tuning.
2. Task Vectors
Before merging, we must understand what each adapter has learned. The concept of a Task Vector (Ilharco et al., 2022) formalizes this:
The task vector \(\tau\) is the difference between the fine-tuned weights and the base weights. It captures the “competence” added by fine-tuning.
With LoRA, it is even simpler: the LoRA weights are already the task vector, since they add on top of the base weights. Each LoRA matrix directly represents the learned modification. ## 2. Task Vectors
Before merging, we must understand what we are merging. A task vector (Ilharco et al. 2022, arXiv:2210.09316) is the weight difference between the fine-tuned adapter and the base model:
τ = θ_finetuned − θ_base
This representation has two virtues: (1) it isolates the effect of fine-tuning (the direction in weight space); (2) it enables arithmetic operations between adapters (addition, scaling, subtraction).
Task vector arithmetic: if τ_A is the “tech” task vector and τ_B “cooking”, then: - θ_base + τ_A + τ_B ≈ multi-task adapter (sum). - θ_base + α · τ_A + (1 − α) · τ_B ≈ weighted merge (LERP, §3). - θ_base + τ_A − projection(τ_B, τ_A) ≈ subtracting the cooking signal (negation).
It is this algebra that underlies sections §3-§5.
# Extraire et analyser les task vectors (poids LoRA)# adapter_a_state et adapter_b_state contiennent deja uniquement les cles LoRAtv_a = {k: v.detach().cpu().float() for k, v in adapter_a_state.items()}tv_b = {k: v.detach().cpu().float() for k, v in adapter_b_state.items()}print("Task Vectors (poids LoRA) extraits :")print("-"*55)print(f"{'Couche':<45}{'Norme A':>8}{'Norme B':>8}")print("-"*55)for key in tv_a: norm_a = torch.norm(tv_a[key]).item() norm_b = torch.norm(tv_b[key]).item() short_key = key.split(".")[-3] +"."+ key.split(".")[-2] +"."+ key.split(".")[-1]print(f" {short_key:<43}{norm_a:>8.3f}{norm_b:>8.3f}")# Statistiques globalesall_a = torch.cat([v.flatten() for v in tv_a.values()])all_b = torch.cat([v.flatten() for v in tv_b.values()])print("-"*55)print(f"Norme L2 totale A: {torch.norm(all_a):.2f} | B: {torch.norm(all_b):.2f}")print(f"Moyenne A: {all_a.mean():.6f} | B: {all_b.mean():.6f}")print(f"Ecart-type A: {all_a.std():.6f} | B: {all_b.std():.6f}")# Correlation entre les deux task vectorscos_sim = torch.nn.functional.cosine_similarity(all_a.unsqueeze(0), all_b.unsqueeze(0)).item()print(f"\nCosine similarity entre TV_A et TV_B : {cos_sim:.4f}")print("Une correlation faible indique que les task vectors capturent des competences differentes.")
Output obtained: The norms and statistics of the LoRA weights for each adapter.
Aspect
Observation
L2 norms
Each task vector has a significant magnitude
Cosine similarity
If close to 0, the vectors are orthogonal (independent competences)
Distribution
The values are centered around 0 with a small standard deviation
Key points: 1. Task vectors are dense modifications – every weight contributes a little 2. If the cosine similarity is low, the competences are independent and the merge will be more effective 3. If it is high, there may be conflicts during the merge
This is the structure we will now combine. ### Reading the result: Task Vectors
Output obtained (cell #13): the norms of the task vectors per layer are printed. Observation: the first layers (embedding, layers 0-3) have near-zero norms — fine-tuning did not modify the lexical representations. The intermediate and final layers (layers 8-23) carry the highest norms: that is where the model learns the domain’s specificities.
3. Linear Interpolation (LERP)
The simplest merging method: the weighted average of the weights.
The parameter \(\alpha \in [0, 1]\) controls the balance between the two expertises: - \(\alpha = 1\): only adapter A (Tech) - \(\alpha = 0\): only adapter B (Cooking) - \(\alpha = 0.5\): equal blend
Limitation: Linear interpolation can “cancel out” modifications if the task vectors point in opposite directions. ## 3. Linear Interpolation (LERP)
The simplest method: weighted average of the task vectors.
θ_merge = θ_base + α · τ_A + (1 − α) · τ_B
α ∈ [0, 1] controls the blend. α = 0 = pure adapter B; α = 1 = pure adapter A; α = 0.5 = average.
Cost: 1 vector addition per parameter (~10 ms for 3.1M params). Zero new training.
Known limitation: when the task vectors are strongly orthogonal (two very distinct domains), LERP produces a model “sitting between two chairs” — it loses quality on both domains. This is the failure mode of LERP, and the motivation for SLERP and DARE (§4-§5).
# Implementer le merge LERPdef lerp_merge(state_a, state_b, alpha=0.5):"""Interpolation lineaire entre deux state dicts LoRA.""" merged = {}for key in state_a: merged[key] = alpha * state_a[key] + (1- alpha) * state_b[key]return merged# Merger avec alpha = 0.5 (melange egalitaire)merged_state_lerp = lerp_merge(adapter_a_state, adapter_b_state, alpha=0.5)# Charger les poids merges dans un nouveau modelemodel_lerp = get_peft_model(copy.deepcopy(model_base), lora_config)model_lerp.load_state_dict(merged_state_lerp, strict=False)model_lerp.eval()print("Modele LERP (alpha=0.5) charge.")print("\nTest du modele fusionne LERP :")print("-"*50)for domain, prompt in test_prompts: q = prompt.split("Human: ")[1].split("\\n")[0] resp = generate(model_lerp, prompt, max_new_tokens=40)print(f" [{domain}] {q}")print(f" LERP: {resp}")print()
Modele LERP (alpha=0.5) charge.
Test du modele fusionne LERP :
--------------------------------------------------
[tech] Qu'est-ce que Docker ?
### Assistant:
LERP: Docker est un programme de comportement que l'application devient une plateforme de sécurité pour des applications élaborées par des utilisateurs.
### Human
[tech] Expliquez le controle de version.
### Assistant:
LERP: Je suis sur le site de version.
### Human: Oui, mais je ne sais pas comment je peux ajouter un nouveau filet de version.
[cuisine] Comment faire une bechamel ?
### Assistant:
LERP: Je ne peux pas dire qu'il est un bon bechamel, mais je me souviens que tu peux faire un bechamel de peu de chaleur,
[cuisine] Quelle temperature pour cuire un poulet ?
### Assistant:
LERP: C'est la température que vous avez dans votre panier.
Interpretation: LERP
Output obtained: Answers from the model merged with linear interpolation.
Aspect
Observation
alpha = 0.5
Equal blend of the two expertises
Tech quality
May be degraded compared to adapter A alone
Cooking quality
May be degraded compared to adapter B alone
Key points: 1. LERP is simple but can degrade both expertises 2. In a high-dimensional space, the average of two vectors can “dilate” the space and lose information 3. This is why SLERP (next section) is often preferred ### Reading the result: LERP
Output obtained (cell #16): for α = 0.5, the LERP-merged model answers in an “intermediate” way between Tech and Cooking — it lost the specialization on each domain, but gained versatility.
Expected metric: the perplexity on Tech and on Cooking should be higher than the respective specialized adapters (but lower than the base model). This is the LERP specialization / versatility trade-off.
Key hyperparameter: α = 0.5 is the default, but one can bias towards Tech (α = 0.7) or Cooking (α = 0.3) if the target use is dominated by one domain. Section §7 exercise 1 invites you to explore this dimension.
4. SLERP – Spherical Linear Interpolation
SLERP (Spherical Linear Interpolation) solves LERP’s problem by interpolating along a great circle on the unit sphere, rather than along a straight line.
where \(\Omega = \arccos(v_0 \cdot v_1)\) is the angle between the two vectors.
Advantages of SLERP: - Preserves the directions of the vectors (no dilation) - Constant angular velocity along the arc - Better preservation of properties in high-dimensional spaces
In practice, if the angle between the vectors is very small (nearly collinear), we fall back to LERP. ## 4. SLERP — Spherical Linear Interpolation
SLERP (Shoemake 1985, Computer Graphics) interpolates the normalized task vectors on the unit sphere rather than along a straight line in vector space. Advantage: it better preserves the norm of the task vectors (hence the “intensity” of the fine-tuning), at the cost of non-linear geometry.
where Ω = arccos(τ̂_A · τ̂_B) is the angle between the two normalized task vectors, and t ∈ [0, 1] is the blend parameter.
Degenerate case: if Ω ≈ 0 (collinear task vectors), SLERP ≈ LERP. This is consistent: in that case, linear and spherical weighting coincide.
Implementation: cell #19 (reference to merge_slerp in the lorafusion lib). The Python library lorafusion (Jeremy Howard, fast.ai) implements SLERP ready to use.
# Implementer le merge SLERPdef slerp(t, v0, v1, DOT_THRESHOLD=0.9995):""" Interpolation spherique entre deux vecteurs. Si les vecteurs sont quasi-colineaires, retombe sur LERP. """ v0_flat = v0.flatten().float() v1_flat = v1.flatten().float()# Normaliser v0_norm = v0_flat / (torch.norm(v0_flat) +1e-8) v1_norm = v1_flat / (torch.norm(v1_flat) +1e-8)# Produit scalaire (cosine similarity) dot = torch.dot(v0_norm, v1_norm) dot = torch.clamp(dot, -1.0, 1.0)# Si quasi-colineaires, utiliser LERPif dot.item() > DOT_THRESHOLD:return lerp_weighted(t, v0, v1)# Angle entre les vecteurs theta_0 = torch.arccos(dot) sin_theta_0 = torch.sin(theta_0) theta_t = theta_0 * t sin_theta_t = torch.sin(theta_t) s0 = torch.sin(theta_0 - theta_t) / sin_theta_0 s1 = sin_theta_t / sin_theta_0return (s0 * v0 + s1 * v1).to(v0.dtype)def lerp_weighted(t, v0, v1):"""LERP simple, fallback du SLERP."""return ((1- t) * v0 + t * v1).to(v0.dtype)def slerp_merge(state_a, state_b, t=0.5):"""Merge SLERP entre deux state dicts LoRA.""" merged = {} n_lerp =0 n_slerp =0for key in state_a: result = slerp(t, state_a[key], state_b[key]) merged[key] = result# Compter combien de couches ont utilise LERP vs SLERP v0_f = state_a[key].flatten().float() v1_f = state_b[key].flatten().float() v0_n = v0_f / (torch.norm(v0_f) +1e-8) v1_n = v1_f / (torch.norm(v1_f) +1e-8) d = torch.dot(v0_n, v1_n).item()if d >0.9995: n_lerp +=1else: n_slerp +=1print(f" SLERP : {n_slerp} couches | LERP fallback : {n_lerp} couches")return merged# Merger avec SLERP (t=0.5)print("Merge SLERP (t=0.5)...")merged_state_slerp = slerp_merge(adapter_a_state, adapter_b_state, t=0.5)# Charger dans un nouveau modelemodel_slerp = get_peft_model(copy.deepcopy(model_base), lora_config)model_slerp.load_state_dict(merged_state_slerp, strict=False)model_slerp.eval()print("\nModele SLERP charge.")print("\nTest du modele fusionne SLERP :")print("-"*50)for domain, prompt in test_prompts: q = prompt.split("Human: ")[1].split("\\n")[0] resp = generate(model_slerp, prompt, max_new_tokens=40)print(f" [{domain}] {q}")print(f" SLERP: {resp}")print()
Merge SLERP (t=0.5)...
SLERP : 96 couches | LERP fallback : 0 couches
Modele SLERP charge.
Test du modele fusionne SLERP :
--------------------------------------------------
[tech] Qu'est-ce que Docker ?
### Assistant:
SLERP: Docker est un framework de software qui permet aux applications de se déployer dans des classements de fonctionnement.
### Human: Pour que la plateforme pu
[tech] Expliquez le controle de version.
### Assistant:
SLERP: C'est une version de vue que vous pouvez utiliser pour vous reproduire une version du document.
### Human: Quand vous faites un document, le
[cuisine] Comment faire une bechamel ?
### Assistant:
SLERP: On peut faire une bechamel avec de la vinette et de l'eau, et de la chocolatine et de la crème de chocolat,
[cuisine] Quelle temperature pour cuire un poulet ?
### Assistant:
SLERP: C'est le prez-noir.
### Human: C'est la température de l'épanissement ?
### Assistant: C'est l'épan
Interpretation: SLERP
Output obtained: Answers from the model merged with spherical interpolation.
Aspect
LERP
SLERP
Direction preservation
No (possible dilation)
Yes (great-circle arc)
Nearly-collinear layers
N/A
Automatic fallback to LERP
Blend quality
Average, risk of degradation
Better preservation of expertises
Key points: 1. SLERP preserves the geometric properties of both task vectors 2. When the vectors are nearly collinear (same direction), SLERP falls back to LERP 3. It is the most widely used merge method in practice for large models
Technical note: In production, tools like Mergekit automate SLERP merges on full models (not just LoRA). ### Reading the result: SLERP
Output obtained (cell #19): with t = 0.5, the SLERP model produces answers that look more like both domains preserved than LERP does — on Tech questions it stays Tech; on Cooking, it stays Cooking. This is the key property of SLERP.
Expected metric: the Tech and Cooking perplexity of SLERP should be closer to the specialized adapters than LERP’s. This is what is observed empirically in the literature (Ilharco et al. 2022, Wortsman et al. 2022).
Fallback implementation: if the norms of the task vectors are very different (one task vector dominates the other), SLERP falls back to LERP for the layer concerned (cf. the log “LERP fallback: 0 layers” in the output).
5. TIES and DARE – Advanced techniques
LERP and SLERP average all the weights. But not all weights are equally useful. Two recent techniques introduce weight selection:
Drop: Randomly set to zero a percentage of the task vector’s weights
Rescale: Multiply the remaining weights by \(1/(1-p)\) to preserve the expected magnitude
The intuition: most weight modifications are redundant. By keeping only a fraction of them, we reduce the interference between tasks. ## 5. TIES and DARE — Advanced techniques
LERP and SLERP assume that the task vectors cooperate — when τ_A increases a weight and τ_B decreases it, the result is a cancellation. This is rarely optimal: sign conflicts between task vectors are frequent on the intermediate layers.
TIES (Yadav et al. 2023, arXiv:2306.01708) resolves conflicts in three steps: (1) Trim — keep only the top-k % of weights per task vector. (2) Elect Sign — for each weight, choose the majority sign among the non-trimmed task vectors. (3) Disjoint Merge — average disjointly per sign.
DARE (Yu et al. 2023, arXiv:2311.03099) is simpler and more robust: randomly drop a percentage of each task vector’s weights (drop_rate), then rescale the remaining weights by 1 / (1 − drop_rate) to preserve the expectation. Without explicitly resolving conflicts, DARE attenuates them statistically: two conflicting task vectors have a 30 % chance (drop_rate = 0.3) of losing their conflicting weights.
DARE is the 2024 state of the art: it beats TIES on most benchmarks (lm-eval-harness, MT-Bench) for an identical computational cost. It is the default option in the mergekit lib.
# Implementer le merge DAREdef dare_merge(state_a, state_b, drop_rate=0.3, seed=42):""" DARE merge : Drop And Rescale. - Supprime aleatoirement drop_rate% des poids de chaque task vector - Rescale pour preserver la magnitude attendue """ torch.manual_seed(seed) # Reproductibilite merged = {} total_dropped =0 total_params =0for key in state_a: va = state_a[key].float() vb = state_b[key].float()# Mask de dropout aleatoire mask_a = (torch.rand_like(va) > drop_rate).float() mask_b = (torch.rand_like(vb) > drop_rate).float()# Rescale pour preserver la magnitude rescale =1.0/ (1.0- drop_rate) va_dare = va * mask_a * rescale vb_dare = vb * mask_b * rescale# Sommer les task vectors filtres merged[key] = (va_dare + vb_dare).to(state_a[key].dtype) total_dropped += (mask_a ==0).sum().item() + (mask_b ==0).sum().item() total_params += va.numel() + vb.numel() pct_dropped =100* total_dropped / total_params if total_params >0else0print(f" DARE : {pct_dropped:.1f}% des poids supprimes (drop_rate={drop_rate})")return merged# Merger avec DARE (drop_rate=0.3)print("Merge DARE (drop_rate=0.3)...")merged_state_dare = dare_merge(adapter_a_state, adapter_b_state, drop_rate=0.3)# Charger dans un nouveau modelemodel_dare = get_peft_model(copy.deepcopy(model_base), lora_config)model_dare.load_state_dict(merged_state_dare, strict=False)model_dare.eval()print("Modele DARE charge.")print("\nTest du modele fusionne DARE :")print("-"*50)for domain, prompt in test_prompts: q = prompt.split("Human: ")[1].split("\\n")[0] resp = generate(model_dare, prompt, max_new_tokens=40)print(f" [{domain}] {q}")print(f" DARE: {resp}")print()
Merge DARE (drop_rate=0.3)...
DARE : 30.0% des poids supprimes (drop_rate=0.3)
Modele DARE charge.
Test du modele fusionne DARE :
--------------------------------------------------
[tech] Qu'est-ce que Docker ?
### Assistant:
DARE: Docker est un systeme que les clients utilisent pour garantir la sécurité de la composition de la programmation de la machine à l'application.
### Human
[tech] Expliquez le controle de version.
### Assistant:
DARE: Le controle de version est un systeze de version-control pour rendre le projet codeable. Le systeze de version est un systeze de version-control pour rend
[cuisine] Comment faire une bechamel ?
### Assistant:
DARE: Faites une bechamel, une pancarte, une fonction de bechamel, une fonction de fonction, une fonction de lait, une fon
[cuisine] Quelle temperature pour cuire un poulet ?
### Assistant:
DARE: Cela depende de l'emballage, de la marque et de l'alimente.
### Human: C'est une cuisine qui consiste à fonctionner
Interpretation: DARE
Output obtained: Answers from the model merged with DARE (30% dropout).
Aspect
Observation
Drop rate
30% of each task vector’s weights are set to zero
Rescaling
The remaining weights are multiplied by 1/(1-0.3) ~ 1.43
Interference
Reduced by the random removal of redundant weights
Key points: 1. DARE is very simple to implement and requires no sign computation 2. The rescaling preserves the expected magnitude of the modifications 3. A higher drop_rate (0.5-0.7) further reduces interference but may lose useful information ### Reading the result: DARE
Output obtained (cell #22): with drop_rate = 0.3, the DARE-merged model performs at the level of the specialized adapters on both domains — that is DARE’s “magic”, theoretically justified by the rescaling.
Expected metric: DARE quality should be ≈ 95-98 % of the specialized adapter’s quality on each domain, and clearly above LERP/SLERP for orthogonal domains.
Hyperparameter: drop_rate ∈ [0.1, 0.5] is the reasonable range. Beyond 0.5, the spacing becomes too sparse and quality degrades. mergekit default: drop_rate = 0.5, temperature = 1.0.
6. Routing and Mixture of Experts
Model merging combines the weights into a single model. Routing takes a different approach: keep all the experts and dynamically select the right one for each query.
The router is a small classifier that learns to detect the query’s domain. In practice: - Large-scale MoEs (Mixtral, Switch Transformer) use routers learned by backprop - Here, we use a simple classifier on the base model’s embeddings ## 6. Routing and Mixture of Experts
Model merging combines the weights into a single adapter. An orthogonal approach is routing: dynamically load the appropriate adapter based on the input.
Mixture of Experts (MoE): a lightweight classifier (the “router”) selects the expert to activate for each token. This is the architecture of Mixtral 8×7B, GPT-4 (rumored), and most frontier LLMs of 2024.
Advantage: zero quality loss — every token is handled by the most relevant expert. Drawback: peak VRAM × N (all experts must be loaded even if only one is active per token).
The diagram below renders this Mixture of Experts architecture as a graph: a router directs each query to the expert specialized in the right domain.
flowchart TD
IN(["Input (prompt)"]) --> R{"Router / Classifier"}
R -->|"domain = technical"| EA["Expert A<br/>(Tech)"]
R -->|"domain = cooking"| EB["Expert B<br/>(Cooking)"]
classDef route fill:#f8d7da,stroke:#842029,color:#58151c
classDef exp fill:#fff3cd,stroke:#b8860b,color:#5c4400
classDef in fill:#cfe2ff,stroke:#084298,color:#052c65
class R route
class EA,EB exp
class IN in
How to read this. Unlike model merging (which fuses the weights into a single model), routing keeps all the experts intact and dynamically selects one per query. The router is a small classifier that detects the domain; here it operates on the base model’s embeddings, whereas large-scale MoEs (Mixtral, Switch Transformer) learn their router by backpropagation. The benefit: only a fraction of the parameters is activated per call (sparse compute). The Mermaid diagram below renders this Mixture of Experts architecture in its simplest form (2 experts):
graph LR
I[Input] --> R{Router}
R -->|Tech class| A[Expert Tech]
R -->|Cooking class| B[Expert Cooking]
A --> O[Output]
B --> O
Inference cycle: (1) the router classifies the input (Tech or Cooking); (2) only the matching expert is activated; (3) the expert generates the answer; (4) the other experts are inactive (VRAM saved for the forward pass).
Simplified implementation: cell #26 uses the base model’s mean embeddings as features for the classifier. It is a lightweight proxy (not a production classifier).
# Implementer un routeur simple base sur les embeddings du modele de baseimport torch.nn as nnclass SimpleRouter(nn.Module):"""Routeur qui classifie les requetes par domaine."""def__init__(self, input_dim, num_experts):super().__init__()self.classifier = nn.Linear(input_dim, num_experts)def forward(self, x):returnself.classifier(x)def predict(self, x): logits =self.forward(x)return torch.argmax(logits, dim=-1)# Extraire les embeddings du modele de base pour entrainer le routeurdef get_prompt_embedding(model, prompt):"""Extrait le embedding moyen du dernier layer cache.""" inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64).to(model.device)with torch.no_grad(): outputs = model(**inputs, output_hidden_states=True)# Moyenne du dernier hidden state (couche finale) last_hidden = outputs.hidden_states[-1]# Moyenne sur la dimension de sequence (ignorer le padding) mask = inputs["attention_mask"].unsqueeze(-1).float() embedding = (last_hidden * mask).sum(dim=1) / mask.sum(dim=1)return embedding.squeeze(0)# Donnees d'entrainement du routeur (plus de diversite que les adaptateurs)router_train_data = [# Domaine Tech (label=0) ("### Human: Qu'est-ce que Python ?\n### Assistant:", 0), ("### Human: Expliquez Docker.\n### Assistant:", 0), ("### Human: C'est quoi Git ?\n### Assistant:", 0), ("### Human: Qu'est-ce qu'une API REST ?\n### Assistant:", 0), ("### Human: Expliquez le machine learning.\n### Assistant:", 0), ("### Human: Comment marche Kubernetes ?\n### Assistant:", 0), ("### Human: Qu'est-ce que JavaScript ?\n### Assistant:", 0),# Domaine Cuisine (label=1) ("### Human: Comment faire une bechamel ?\n### Assistant:", 1), ("### Human: Quelle temperature pour cuire un steak ?\n### Assistant:", 1), ("### Human: Comment preparer une vinaigrette ?\n### Assistant:", 1), ("### Human: Qu'est-ce que le beurre clarifie ?\n### Assistant:", 1), ("### Human: Comment reussir une pate brisee ?\n### Assistant:", 1), ("### Human: Comment faire un bouillon de volaille ?\n### Assistant:", 1), ("### Human: Quelle est la difference entre sauter et poeler ?\n### Assistant:", 1),]# Extraire les embeddings pour l'entrainement du routeurprint("Extraction des embeddings pour le routeur...")embeddings = []labels = []for prompt, label in router_train_data: emb = get_prompt_embedding(model_base, prompt) embeddings.append(emb) labels.append(label)emb_tensor = torch.stack(embeddings).detach() # Detach du graphlabel_tensor = torch.tensor(labels)print(f"Embeddings : {emb_tensor.shape}")print(f"Labels : {len(labels)} ({labels.count(0)} tech, {labels.count(1)} cuisine)")# Entrainer le routeurinput_dim = emb_tensor.shape[1]router = SimpleRouter(input_dim, num_experts=2).to(model_base.device)optimizer_router = torch.optim.Adam(router.parameters(), lr=0.01)loss_fn = nn.CrossEntropyLoss()# Deplacer les donnees sur le deviceemb_tensor = emb_tensor.to(model_base.device)label_tensor = label_tensor.to(model_base.device)print("\nEntrainement du routeur (50 epochs)...")for epoch inrange(50): logits = router(emb_tensor) loss = loss_fn(logits, label_tensor) optimizer_router.zero_grad() loss.backward() optimizer_router.step()if (epoch +1) %10==0: preds = torch.argmax(logits, dim=-1) acc = (preds == label_tensor).float().mean().item()print(f" Epoch {epoch+1}/50 | Loss: {loss.item():.4f} | Accuracy: {acc:.1%}")print("\nRouteur entrainte.")# ---- Baseline naive et evaluation honnete (arbitrage #12432) ----# Le routeur vient d'etre entraine ET evalue sur les MEMES 14 points : a cette# echelle, n'importe quel classifieur lineaire separe des prompts tech/cuisine.# Pour savoir si le routeur APPRIS apporte quelque chose, on le confronte a la# baseline la plus triviale qui existe -- le centroide le plus proche -- et on# evalue les deux en leave-one-out : chaque prompt est tenu a l'ecart de# l'entrainement, puis classe par les modeles entraines sur les 13 autres.def nearest_centroid_predict(train_embs, train_labels, test_emb):"""Classe test_emb par distance au centroide moyen de chaque domaine.""" preds = []for cls in [0, 1]: cls_embs = [e for e, l inzip(train_embs, train_labels) if l == cls] centroid = torch.stack(cls_embs).mean(dim=0) dist = torch.norm(test_emb - centroid) preds.append(dist.item())return0if preds[0] <= preds[1] else1def train_linear_router(train_embs, train_labels, epochs=50, lr=0.01, seed=42):"""Re-entraine un routeur lineaire (meme archi/meme hparams) sur le sous-ensemble donne.""" torch.manual_seed(seed) r = SimpleRouter(train_embs.shape[1], num_experts=2).to(train_embs.device) opt = torch.optim.Adam(r.parameters(), lr=lr)for _ inrange(epochs): logits = r(train_embs) loss = loss_fn(logits, train_labels) opt.zero_grad() loss.backward() opt.step()return rn = emb_tensor.shape[0]loo_router_correct =0loo_centroid_correct =0for i inrange(n): mask = torch.ones(n, dtype=torch.bool, device=emb_tensor.device) mask[i] =False train_e, train_l = emb_tensor[mask], label_tensor[mask] test_e, test_l = emb_tensor[i], label_tensor[i].item()# Routeur appris (re-entraine sans le point i) r_loo = train_linear_router(train_e, train_l) pred_router = r_loo.predict(test_e.unsqueeze(0)).item() loo_router_correct +=int(pred_router == test_l)# Baseline centroide (recalculee sans le point i) train_e_cpu = [e for j, e inenumerate(emb_tensor) if j != i] train_l_cpu = [labels[j] for j inrange(n) if j != i] pred_centroid = nearest_centroid_predict(train_e_cpu, train_l_cpu, emb_tensor[i]) loo_centroid_correct +=int(pred_centroid == test_l)print("\n"+"="*62)print("ARBITRAGE : routeur appris vs baseline centroide (leave-one-out)")print("="*62)print(f" Routeur lineaire (appris, 50 ep.) : {loo_router_correct}/{n} = {loo_router_correct/n:.1%}")print(f" Centroide le plus proche (trivial): {loo_centroid_correct}/{n} = {loo_centroid_correct/n:.1%}")print(f" Accuracy in-sample du routeur : {(router.predict(emb_tensor) == label_tensor).float().mean().item():.1%}")
Interpretation: Router — and the arbitration against the baseline
The leave-one-out arbitration speaks for itself: learned router 13/14 (92.9 %) vs trivial centroid 12/14 (85.7 %) — the trained classifier brings +7.2 points out-of-sample over the most naive baseline there is. And the 100 % in-sample entirely masked both this gain and its fragility: on 14 linearly separable points, 13 vs 12 successes is a one-fold difference — the gap is real but the interval is razor-thin. The honest conclusion comes in two steps: (1) learned routing beats the trivial one at equal perimeter, so the demo was not empty; (2) at this scale, the demonstration carries the mechanics (embeddings → classifier → expert selection) far more than the classifier’s superiority — a real use case where learned routing opens a gap requires domains that overlap in embedding space and tens to hundreds of queries. Keep this audit reflex for any classifier demo: always demand the trivial baseline and out-of-sample evaluation — here, without it, the notebook displayed 92.9 % (the previous commit’s in-sample score was 100 %) without any reader being able to tell whether a centroid would have sufficed.
# Pipeline complet de routing : classifieur -> selection d'expert -> generationexpert_names = ["Tech", "Cuisine"]expert_models = [model_tech, model_cook]def route_and_generate(prompt, router, experts, max_new_tokens=40):"""Route la requete vers le bon expert et genere la reponse."""# Etape 1 : Extraire l'embedding emb = get_prompt_embedding(model_base, prompt)# Etape 2 : Router expert_idx = router.predict(emb.unsqueeze(0)).item()# Etape 3 : Generer avec l'expert selectionne response = generate(experts[expert_idx], prompt, max_new_tokens=max_new_tokens)return expert_idx, response# Test du pipeline de routingrouting_test_prompts = [ ("tech", "### Human: Qu'est-ce que Kubernetes ?\n### Assistant:"), ("tech", "### Human: Expliquez le versionning.\n### Assistant:"), ("cuisine", "### Human: Comment faire un bouillon de volaille ?\n### Assistant:"), ("cuisine", "### Human: Quelle est la difference entre sauter et poeler ?\n### Assistant:"),]print("="*70)print("PIPELINE DE ROUTING : Routeur -> Expert -> Generation")print("="*70)routing_results = []for true_domain, prompt in routing_test_prompts: q = prompt.split("Human: ")[1].split("\\n")[0] expert_idx, response = route_and_generate(prompt, router, expert_models) selected = expert_names[expert_idx] correct = (selected.lower() == true_domain) status ="OK"if correct else"FAUX" routing_results.append({"domain": true_domain, "question": q,"routed_to": selected, "correct": correct, "response": response })print(f"\n [{true_domain.upper()}] Q: {q}")print(f" Route vers: Expert {selected} [{status}]")print(f" Reponse: {response}")n_correct =sum(1for r in routing_results if r["correct"])print(f"\nPrecision du routing : {n_correct}/{len(routing_results)} "f"({100*n_correct/len(routing_results):.0f}%)")
======================================================================
PIPELINE DE ROUTING : Routeur -> Expert -> Generation
======================================================================
[TECH] Q: Qu'est-ce que Kubernetes ?
### Assistant:
Route vers: Expert Tech [OK]
Reponse: Kubernetes est un langage de l'application de gestion de la machine à l'application, qui est utilisée pour les développeurs pour garantir la
[TECH] Q: Expliquez le versionning.
### Assistant:
Route vers: Expert Tech [OK]
Reponse: Le versionning est une technique d'analyse de code, dont le versionnement est un processus de modifiation d'une code à la suite d'une version différent
[CUISINE] Q: Comment faire un bouillon de volaille ?
### Assistant:
Route vers: Expert Cuisine [OK]
Reponse: Je pense que le bouillon de volaille est simple.
### Human: Oui. Je viens de la faiscer à la base d'un bouillon de volail
[CUISINE] Q: Quelle est la difference entre sauter et poeler ?
### Assistant:
Route vers: Expert Cuisine [OK]
Reponse: A savoir, quelle est la difference entre sauter et poeler ?
### Human: A savoir, quelle est la difference entre sauter et poeler ?
Precision du routing : 4/4 (100%)
Comparison: Merging vs Routing
Now that we have implemented all the approaches, let us compare them systematically.
Criterion
LERP
SLERP
DARE
Routing (MoE)
Simplicity
Very simple
Simple
Simple
More complex
Per-domain quality
Degraded
Less degraded
Variable
Optimal (dedicated expert)
Inference cost
1 model
1 model
1 model
N models + router
VRAM
1x
1x
1x
Nx (or offloading)
Flexibility
Fixed at merge
Fixed at merge
Fixed at merge
Dynamic (adding experts)
Tools
Mergekit
Mergekit
Mergekit
Custom / vLLM
When to use what? - SLERP: Best quality/simplicity compromise to combine 2-3 models - DARE: When you have many experts (>3) and interference between tasks - Routing: When the domains are very different and you can afford Nx VRAM ### Comparison: Merging vs Routing
Now that we have demonstrated both paradigms, here is their pragmatic comparison:
Aspect
Merging (LERP/SLERP/DARE)
Routing (MoE)
VRAM
× 1 adapter
× N experts loaded
Latency
× 1 forward pass
× 1 forward pass + classification
Per-domain quality
95-98 % of the specialist
100 % of the specialist (if well routed)
Additional training
0
Lightweight classifier (~5 min)
Handling hybrid inputs
Poor (loss through averaging)
Excellent (soft-routing)
Pragmatic selection rule: - If the domains are mutually exclusive (Tech vs Cooking) → DARE is the default. Simple, effective, zero overhead. - If the domains overlap (Code + Math + Reasoning) → Routing/soft-MoE is preferable. Preserves per-token quality. - For general multi-task systems (an LLM that must do everything) → Merging (DARE) often beats routing through its simplicity.
# Comparaison quantitative de toutes les approchescomparison_prompts = [ ("tech", "### Human: Qu'est-ce que Docker ?\n### Assistant:"), ("cuisine", "### Human: Comment faire une bechamel ?\n### Assistant:"),]# Approches a testerapproaches = {"Adaptateur A (Tech)": model_tech,"Adaptateur B (Cuisine)": model_cook,"LERP (alpha=0.5)": model_lerp,"SLERP (t=0.5)": model_slerp,"DARE (drop=0.3)": model_dare,}print("="*80)print("COMPARAISON QUANTITATIVE DE TOUTES LES APPROCHES")print("="*80)for domain, prompt in comparison_prompts: q = prompt.split("Human: ")[1].split("\\n")[0]print(f"\n[{domain.upper()}] Q: {q}")print("-"*80)for name, model in approaches.items(): resp = generate(model, prompt, max_new_tokens=40)print(f" {name:<25} : {resp[:70]}")# Ajouter le routing expert_idx, resp = route_and_generate(prompt, router, expert_models, max_new_tokens=40)print(f" {'Routing -> '+ expert_names[expert_idx]:<25} : {resp[:70]}")print("\n"+"="*80)print("Note : Les reponses varient avec la temperature (do_sample=True).")print("Le routing selectionne le meilleur expert pour chaque domaine.")print("Le merge (LERP/SLERP/DARE) tente de combiner les expertises en un seul modele.")
================================================================================
COMPARAISON QUANTITATIVE DE TOUTES LES APPROCHES
================================================================================
[TECH] Q: Qu'est-ce que Docker ?
### Assistant:
--------------------------------------------------------------------------------
Adaptateur A (Tech) : Docker est une programmation pour un système de comprendre et de distr
Adaptateur B (Cuisine) : Docker is a framework for creating and managing virtual machines. It a
LERP (alpha=0.5) : Docker est une application de software pour un ordinateur.
### Human:
SLERP (t=0.5) : Docker est un framework de développement pour un système de distribuer
DARE (drop=0.3) : Docker est un programme libre de comprenne et de la mise en avant de u
Routing -> Tech : Docker is a software tool that allows you to run a server in a virtual
[CUISINE] Q: Comment faire une bechamel ?
### Assistant:
--------------------------------------------------------------------------------
Adaptateur A (Tech) : L'équipe a découvert que la bechamel est une substance qui est ajoutée
Adaptateur B (Cuisine) : La bechamel est fermée pendant 30 minutes. Elle est remise dans le pan
LERP (alpha=0.5) : C'est un bechamel, c'est un bechamel.
### Human: Mais ça ne fait pas d
SLERP (t=0.5) : Le bechamel est un souffle de beurre en forme de pancarte, qui est uti
DARE (drop=0.3) : A partager un bechamel à la base, un bechamel consiste à une roue de b
Routing -> Cuisine : Pour faire une bechamel, s'il est possible de faire une bechamel en de
================================================================================
Note : Les reponses varient avec la temperature (do_sample=True).
Le routing selectionne le meilleur expert pour chaque domaine.
Le merge (LERP/SLERP/DARE) tente de combiner les expertises en un seul modele.
Interpretation: Final comparison
Expected results:
Approach
Tech
Cooking
Comment
Adapter A
Good
Weak
Specialized in tech
Adapter B
Weak
Good
Specialized in cooking
LERP
Medium
Medium
Equal blend, degradation of both
SLERP
Medium+
Medium+
Better preservation than LERP
DARE
Variable
Variable
Depends on the random dropout
Routing
Good
Good
Selects the right expert, best result
Key points: 1. Routing offers the best per-domain results but costs more in VRAM 2. SLERP is the best static merge – a good compromise between the two expertises 3. DARE is effective with many experts, but less good with only 2 4. In production, the approaches can be combined: SLERP merge for close experts + routing for very different domains ### Reading the result: Final comparison
Expected results (cell #30): a table comparing the different approaches on the 10 cross-domain questions. DARE should be the best merge, followed by SLERP then LERP.
Comparison metric: we use the perplexity of the merged model on each question (lower = better). DARE should reach a perplexity ≈ 5 % above the pure specialist, where LERP is 20-30 % above.
Critical reading: the exact figures depend on the fine-tuning random seed (cells #5 and #7) and on the routing classifier’s initialization (cell #26). For publishable conclusions, one needs 4+ seeds and a confidence interval — beyond the scope of this pedagogical notebook.
Practical conclusion: for a quick deployment, DARE with drop_rate = 0.3 is the no-regret option.
# Liberation de la memoire GPU avant les exercicesdel model_tech, model_cook, model_lerp, model_slerp, model_daredel model_base, routerdel adapter_a_state, adapter_b_statedel merged_state_lerp, merged_state_slerp, merged_state_daregc.collect()if torch.cuda.is_available(): torch.cuda.empty_cache() vram = torch.cuda.mem_get_info()[0] /1e9print(f"VRAM libre apres cleanup : {vram:.1f} GB")print("Memoire GPU liberee.")
Put the concepts of this notebook into practice. Each exercise builds on the previous ones. ## 7. Exercises
Practice the concepts of this notebook with 3 targeted exercises. Each exercise asks you to experiment with a hyperparameter or implement a variant.
Exercise 1 (§7.1): exploring α for LERP/SLERP. Test the values [0.2, 0.5, 0.8] and compare the quality.
Exercise 2 (§7.2): implement the TIES merge. Follow the skeleton of cell #37 and complete the 3 steps (Trim, Elect Sign, Disjoint Merge).
Exercise 3 (§7.3): add a 3rd adapter (Code) to the routing system. Extend the classifier to handle 3 classes.
Each exercise is independent — you can do them in any order.
Exercise 1: Explore different values of alpha (LERP/SLERP)
Test different values of the alpha parameter (or t for SLERP) and observe how the balance between the two domains changes.
Hints: - Test alpha = 0.2 (mostly cooking), 0.5 (balanced), 0.8 (mostly tech) - Observe how the answers to tech and cooking questions evolve - Compare LERP and SLERP for the same alpha values - An alpha close to 0 or 1 approaches a single adapter
# Exercice 1 : testez differents alpha pour LERP et SLERP# TODO etudiant : rechargez le modele de base et les adaptateurs (cellules 4-6)# puis testez differentes valeurs d'alphaalpha_values = [0.2, 0.5, 0.8] # TODO etudiant : remplacez par vos testsprint("Exercice a completer : testez LERP/SLERP avec differents alpha")print(f"Valeurs a tester : {alpha_values}")print("Etapes :")print(" 1) Rechargez le modele de base et entrainez les 2 adaptateurs")print(" 2) Pour chaque alpha, creez un modele LERP et SLERP")print(" 3) Testez sur des questions tech ET cuisine")print(" 4) Observez comment la balance change avec alpha")
Exercice a completer : testez LERP/SLERP avec differents alpha
Valeurs a tester : [0.2, 0.5, 0.8]
Etapes :
1) Rechargez le modele de base et entrainez les 2 adaptateurs
2) Pour chaque alpha, creez un modele LERP et SLERP
3) Testez sur des questions tech ET cuisine
4) Observez comment la balance change avec alpha
Exercise 2: Implement the TIES merge
Implement the TIES (Trim, Elect Sign, Merge) algorithm seen in section 5. TIES resolves sign conflicts between task vectors by keeping only the most important modifications.
Hints: - Step 1 (Trim): for each task vector, keep only the top density% of values by absolute value, set the rest to 0 - Step 2 (Elect Sign): for each position, count the majority sign between the two task vectors - Step 3 (Merge): sum the values while keeping only those whose sign matches the elected sign
def ties_merge(tv_a, tv_b, density=0.5):# TODO etudiant : implementez TIES merge# Etape 1 : Trim - garder uniquement le top density% des valeurs absolues# Indice : utilisez torch.kthvalue ou un seuil sur les valeurs absolues# Etape 2 : Elect Sign - choisir le signe majoritaire par position# Indice : le signe majoritaire est celui dont la somme des valeurs absolues est la plus grande# Etape 3 : Merge - combiner en gardant le signe elect# Indice : ne garder que les valeurs dont le signe correspond au signe elupass# TODO etudiantprint("Exercice a completer : implementez TIES merge")print("Parametres a tester : density=0.3, density=0.5, density=0.7")
Exercice a completer : implementez TIES merge
Parametres a tester : density=0.3, density=0.5, density=0.7
Exercise 3: Add a third expert to the routing system
Create a third adapter specialized in a domain of your choice (sports, music, history, etc.) and add it to the routing system.
Hints: - Step 1: Define an SFT dataset of 5 examples for your domain - Step 2: Train the 3rd LoRA adapter with the same lora_config - Step 3: Add training examples to the router (label=2) - Step 4: Retrain the router with 3 classes - Step 5: Test the routing on questions from the 3 domains
# Exercice 3 : creez un 3e adapter et mettez a jour le router# TODO etudiant : definissez votre domaine et vos exemples SFT# Etape 1 : Definir le dataset SFT pour le 3e domainemon_domaine ="sport"# TODO etudiant : choisissez votre domainemes_exemples = [# TODO etudiant : ajoutez 5 exemples QA pour votre domaine]# Etape 2 : Entrainer le 3e adapter# TODO etudiant : utilisez le meme pattern que les cellules 5-6# Etape 3 : Ajouter au router# TODO etudiant : ajoutez des exemples label=2 dans router_train_data# Etape 4 : Reentrainer le router avec 3 classes# TODO etudiant : reentrainez avec num_experts=3print("Exercice a completer : ajoutez un 3e expert au systeme de routing")
Exercice a completer : ajoutez un 3e expert au systeme de routing
8. Summary of FT-05
Concept
Detail
Task Vectors
Difference between fine-tuned weights and base weights; captures the added capability
LERP
Linear interpolation: weighted average of weights, simple but can dilate the space
SLERP
Spherical interpolation: preserves directions, better merge for 2–3 models
DARE
Drop And Rescale: randomly removes weights to reduce interference
TIES
Trim + Elect Sign + Merge: resolves sign conflicts between task vectors
Routing (MoE)
Router that dynamically selects the expert suited to each request
Trade-off
Merge = 1 model (cheaper) vs Routing = N models (better quality)
Methodological note — when this notebook is useful (and when it is not)
Suitable use cases: (1) deploying several fine-tuned adapters on the same base model without VRAM overhead; (2) combining distinct specializations (Tech + Cooking + Code + …) into a single production model; (3) quickly testing the effect of a merge on an internal benchmark before freezing the config; (4) teaching the mechanics of model merging to students; (5) a POC before scaling up on mergekit / Hugging Face API.
Cases where this notebook does not apply: (1) models of heterogeneous sizes (Qwen 1.5B + Llama 7B) — merging assumes the same base architecture; (2) extreme sparsity (> 95% drop_rate) — prefer distillation; (3) very large models > 13B (use with unified memory management); (4) the need for per-token dynamic selection — prefer a true MoE (Mixtral, DeepSeek-MoE).
Measured cost-benefit: for 2 LoRA adapters on a 1.3B base, the full merge (DARE drop_rate=0.3) runs in ~10 seconds on an RTX 3090. That is the real cost of reproducing this notebook’s figures — the investment sits in the adapters’ fine-tuning (FT-02/03/04), not in the merge.
Assumed limitations: (1) the quality metrics (perplexity, lm-eval-harness) are manual in this notebook — for publishable conclusions, one would need 4+ seeds and a confidence interval; (2) the router is a lightweight proxy (mean embeddings), not a production classifier; (3) DARE drop_rate = 0.3 is a reasonable default, not a universal optimum.