RL-6 : GRPO — Group Relative Policy Optimization pour le Trading

Série RL for Trading (#1461) | RL-1 Q-Learning · RL-2 PPO · RL-3 Reward Shaping · RL-4 Multi-Asset · RL-5 Overlay · RL-6 GRPO

Objectifs

• Implémenter GRPO (Group Relative Policy Optimization), l’algorithme clé de DeepSeek-R1
• GRPO élimine le besoin d’un critic séparé en utilisant des récompenses relatives intra-groupe
• L’avantage est calculé par normalisation au sein d’un groupe de G trajectoires pour le même état
• Appliquer au trading multi-actif sur le panier anti-biais 24 symbols
• Validation multi-seed (0/1/7/42) avec edge ≥ 2σ

Pourquoi GRPO ?

Aspect	PPO	GRPO
Value function	Oui (Critic network)	Non (basé sur le groupe)
Avantage	GAE (λ-returns)	Normalisation intra-groupe
Complexité	2 réseaux	1 seul réseau
Stabilité	Clip + entropy	Clip + KL + baseline groupe

L’hypothèse GRPO : la relative performance d’une action par rapport aux alternatives au même instant est un signal plus robuste qu’une estimation absolue de valeur.

Durée estimée : ~6 min (Papermill) | Prérequis : PyTorch, panier L1/L2

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
from pathlib import Path
import warnings
warnings.filterwarnings("ignore")

# GRPO hyperparameters
SEEDS = [0, 1, 7, 42]
LOOKBACK = 20
N_EPISODES = 120
EPISODE_LEN = 252
HIDDEN_DIM = 128
LR = 3e-4
CLIP_EPS = 0.2
KL_COEF = 0.05
ENTROPY_COEF = 0.01
GROUP_SIZE = 8          # G trajectories per state for advantage estimation
FEE_BPS = 10
BATCH_SIZE_EP = 64

DATA_PATH = Path("../datasets/panier/panier_close_all.csv")
print(f"PyTorch {torch.__version__}, CUDA={torch.cuda.is_available()}")
print(f"GRPO config: group_size={GROUP_SIZE}, kl_coef={KL_COEF}, episodes={N_EPISODES}")

# Determinisme GPU (issue #16795) : une graine seule ne garantit rien --
# cuDNN choisit ses algorithmes de facon non deterministe par defaut.
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
import os
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
print(f"Determinisme: use_deterministic_algorithms=True, cudnn.deterministic=True")

PyTorch 2.6.0+cu124, CUDA=True
GRPO config: group_size=8, kl_coef=0.05, episodes=120
Determinisme: use_deterministic_algorithms=True, cudnn.deterministic=True

1. Chargement du panier

Même panier 24-actifs que RL-04/RL-05.

df_raw = pd.read_csv(DATA_PATH, index_col=0, parse_dates=True)
df_raw.index = pd.DatetimeIndex(df_raw.index)
df_raw = df_raw.sort_index()

exclude = [c for c in df_raw.columns if df_raw[c].notna().sum() < len(df_raw) * 0.5]
df = df_raw.drop(columns=exclude)
df = df.ffill().dropna()

n_assets = df.shape[1]
symbols = list(df.columns)
returns = df.pct_change().dropna()

print(f"Panier: {n_assets} assets, {len(returns)} business days")
print(f"Range: {returns.index.min().date()} to {returns.index.max().date()}")

Panier: 24 assets, 3096 business days
Range: 2017-11-10 to 2026-05-02

2. Environnement Portfolio (réutilisé de RL-04)

Allocation via Dirichlet (toujours valide : positif, somme=1).

class PortfolioEnv:
    """Multi-asset portfolio with Dirichlet policy."""

    def __init__(self, returns_df, lookback=20, episode_len=252, fee_bps=10):
        self.returns = returns_df.values
        self.n_assets = returns_df.shape[1]
        self.lookback = lookback
        self.episode_len = episode_len
        self.fee = fee_bps / 10000
        self._idx = None
        self._prev_weights = None

    @property
    def state_dim(self):
        return self.lookback * self.n_assets

    @property
    def action_dim(self):
        return self.n_assets

    def reset(self, start_idx=None):
        max_start = len(self.returns) - self.lookback - self.episode_len - 1
        if start_idx is None:
            start_idx = np.random.randint(self.lookback, max(max_start, self.lookback + 1))
        self._idx = start_idx
        self._prev_weights = np.full(self.n_assets, 1.0 / self.n_assets)
        return self._get_state()

    def _get_state(self):
        window = self.returns[self._idx - self.lookback:self._idx]
        std = window.std(axis=0, keepdims=True) + 1e-8
        return (window / std).flatten()

    def step(self, weights):
        self._idx += 1
        done = self._idx >= len(self.returns) - 1
        
        day_ret = self.returns[self._idx]
        port_ret = np.dot(weights, day_ret)
        bh_ret = day_ret.mean()
        
        turnover = np.abs(weights - self._prev_weights).sum()
        fee_cost = turnover * self.fee * 0.5
        port_ret -= fee_cost
        self._prev_weights = weights.copy()
        
        reward = (port_ret - bh_ret) * 100
        info = {"port_ret": port_ret, "bh_ret": bh_ret}
        return self._get_state() if not done else np.zeros(self.state_dim), reward, done, info

3. Policy Network (Actor seul — pas de Critic)

GRPO n’a besoin que d’un policy network. L’avantage est estimé via les récompenses relatives au sein du groupe.

L’Actor utilise Softplus + concentration de base pour la distribution Dirichlet (poids toujours valides).

class GRPOPolicy(nn.Module):
    """Policy-only network for GRPO. Outputs Dirichlet concentrations."""

    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )
        self.concentration_head = nn.Linear(hidden_dim, action_dim)
        self.base_concentration = nn.Parameter(torch.ones(action_dim))

    def forward(self, state):
        feat = self.backbone(state)
        raw = self.concentration_head(feat)
        concentrations = torch.nn.functional.softplus(raw) + self.base_concentration + 0.1
        return concentrations


def sample_dirichlet(concentrations, n_samples=1):
    """Sample portfolio weights from Dirichlet distribution."""
    if n_samples == 1:
        dist = torch.distributions.Dirichlet(concentrations)
        return dist.sample(), dist
    dist = torch.distributions.Dirichlet(concentrations)
    return dist.sample((n_samples,)), dist


def dirichlet_log_prob(dist, weights):
    """Log probability under Dirichlet."""
    return dist.log_prob(weights)

4. GRPO Update

L’idée centrale de GRPO : 1. Pour chaque état, échantillonner G trajectoires (group) 2. Calculer l’avantage par normalisation intra-groupe : A_i = (R_i - mean(R)) / std(R) 3. Pas besoin de Critic — la baseline est implicite dans le groupe 4. PPO-clip + penalty KL vs old policy

def grpo_update(policy, optimizer, states, actions_group, rewards_group, old_log_probs, device,
                clip_eps=0.2, kl_coef=0.05, entropy_coef=0.01):
    """
    GRPO update using group-relative advantages.
    
    states: (batch, state_dim)
    actions_group: (batch, G, action_dim) - G samples per state
    rewards_group: (batch, G) - reward for each group member
    old_log_probs: (batch, G) - log probs under old policy
    """
    batch_size, G, action_dim = actions_group.shape
    
    # Compute group-relative advantages (key GRPO idea)
    mean_r = rewards_group.mean(dim=1, keepdim=True)
    std_r = rewards_group.std(dim=1, keepdim=True) + 1e-8
    advantages = (rewards_group - mean_r) / std_r  # (batch, G)
    
    # Flatten for processing
    states_exp = states.unsqueeze(1).expand(-1, G, -1).reshape(-1, states.shape[-1])
    actions_flat = actions_group.reshape(-1, action_dim)
    adv_flat = advantages.reshape(-1)
    old_lp_flat = old_log_probs.reshape(-1)
    
    # New policy log probs
    concentrations = policy(states_exp)
    dist = torch.distributions.Dirichlet(concentrations)
    new_log_probs = dist.log_prob(actions_flat)
    entropy = dist.entropy().mean()
    
    # PPO-style clipped objective
    ratio = torch.exp(new_log_probs - old_lp_flat)
    surr1 = ratio * adv_flat
    surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv_flat
    actor_loss = -torch.min(surr1, surr2).mean()
    
    # KL penalty (vs old policy)
    kl_penalty = kl_coef * (old_lp_flat - new_log_probs).mean()
    
    # Entropy bonus
    loss = actor_loss + kl_penalty - entropy_coef * entropy
    
    optimizer.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(policy.parameters(), 0.5)
    optimizer.step()
    
    return loss.item()

5. Boucle d’entraînement GRPO multi-seed

Chaque épisode : on échantillonne G trajectoires en parallèle, on calcule les avantages relatifs, on met à jour la policy.

def train_grpo_one_seed(returns_df, seed, device="cpu"):
    """Train GRPO for one seed."""
    torch.manual_seed(seed)
    np.random.seed(seed)
    
    env = PortfolioEnv(returns_df, lookback=LOOKBACK, episode_len=EPISODE_LEN, fee_bps=FEE_BPS)
    policy = GRPOPolicy(env.state_dim, env.action_dim, HIDDEN_DIM).to(device)
    optimizer = optim.Adam(policy.parameters(), lr=LR)
    
    losses = []
    for ep in range(N_EPISODES):
        # Collect G trajectories (group)
        group_states = []
        group_actions = []
        group_rewards = []
        group_log_probs = []
        
        for g in range(GROUP_SIZE):
            states_ep = []
            actions_ep = []
            rewards_ep = []
            log_probs_ep = []
            
            state = env.reset()
            for step in range(env.episode_len):
                s = torch.FloatTensor(state).unsqueeze(0).to(device)
                with torch.no_grad():
                    conc = policy(s)
                    dist = torch.distributions.Dirichlet(conc)
                    weights = dist.sample()
                    log_prob = dist.log_prob(weights)
                
                action_np = weights.squeeze(0).cpu().numpy()
                next_state, reward, done, _ = env.step(action_np)
                
                states_ep.append(state)
                actions_ep.append(action_np)
                rewards_ep.append(reward)
                log_probs_ep.append(log_prob.item())
                
                state = next_state
                if done:
                    break
            
            group_states.append(states_ep)
            group_actions.append(actions_ep)
            group_rewards.append(rewards_ep)
            group_log_probs.append(log_probs_ep)
        
        # Align episode lengths (use minimum)
        min_len = min(len(r) for r in group_rewards)
        
        # Build batch for GRPO update
        n_steps = min_len
        if n_steps < 10:
            continue
        
        batch_states = []
        batch_actions = []
        batch_rewards = []
        batch_old_lp = []
        
        for t in range(0, n_steps, max(1, n_steps // 20)):  # subsample ~20 steps
            states_t = np.array([gs[t] for gs in group_states])
            actions_t = np.array([ga[t] for ga in group_actions])
            rewards_t = np.array([gr[t] for gr in group_rewards])
            lp_t = np.array([glp[t] for glp in group_log_probs])
            
            batch_states.append(states_t)   # (G, state_dim)
            batch_actions.append(actions_t) # (G, action_dim)
            batch_rewards.append(rewards_t) # (G,)
            batch_old_lp.append(lp_t)       # (G,)
        
        if not batch_states:
            continue
        
        # Stack and reshape for GRPO update
        all_states = torch.FloatTensor(np.concatenate(batch_states, axis=0)).to(device)
        all_actions = torch.FloatTensor(np.concatenate(batch_actions, axis=0)).to(device)
        all_rewards = torch.FloatTensor(np.concatenate(batch_rewards, axis=0)).to(device)
        all_old_lp = torch.FloatTensor(np.concatenate(batch_old_lp, axis=0)).to(device)
        
        # Simple batch update (no group structure needed for single-step advantages)
        concentrations = policy(all_states)
        dist = torch.distributions.Dirichlet(concentrations)
        new_lp = dist.log_prob(all_actions)
        
        # Group-relative advantage: normalize rewards across batch
        adv = (all_rewards - all_rewards.mean()) / (all_rewards.std() + 1e-8)
        
        ratio = torch.exp(new_lp - all_old_lp)
        surr1 = ratio * adv
        surr2 = torch.clamp(ratio, 1 - CLIP_EPS, 1 + CLIP_EPS) * adv
        actor_loss = -torch.min(surr1, surr2).mean()
        kl_pen = KL_COEF * (all_old_lp - new_lp).mean()
        entropy = dist.entropy().mean()
        loss = actor_loss + kl_pen - ENTROPY_COEF * entropy
        
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(policy.parameters(), 0.5)
        optimizer.step()
        losses.append(loss.item())
    
    # OOS evaluation
    oos_start = len(returns_df) - 500
    oos_rets_port = []
    oos_rets_bh = []
    state = env.reset(start_idx=oos_start)
    policy.eval()
    with torch.no_grad():
        for t in range(min(500, len(returns_df) - oos_start - 1)):
            s = torch.FloatTensor(state).unsqueeze(0).to(device)
            conc = policy(s)
            weights = conc / conc.sum()  # deterministic: use normalized concentrations
            action_np = weights.squeeze(0).cpu().numpy()
            next_state, reward, done, info = env.step(action_np)
            oos_rets_port.append(info["port_ret"])
            oos_rets_bh.append(info["bh_ret"])
            state = next_state
            if done:
                break
    
    def sharpe(r):
        return float(np.mean(r) / (np.std(r, ddof=1) + 1e-8) * np.sqrt(252))
    
    port_arr = np.array(oos_rets_port)
    bh_arr = np.array(oos_rets_bh)
    
    return {
        "seed": seed,
        "sharpe_port": sharpe(port_arr),
        "sharpe_bh": sharpe(bh_arr),
        "delta_sharpe": sharpe(port_arr) - sharpe(bh_arr),
        "mean_ret_port": float(np.mean(port_arr)) * 252,
        "mean_ret_bh": float(np.mean(bh_arr)) * 252,
        "n_oos": len(port_arr),
        "final_loss": losses[-1] if losses else float("nan"),
        "n_episodes": len(losses)
    }

# Run all seeds
device = "cuda" if torch.cuda.is_available() else "cpu"
results = []
for seed in SEEDS:
    r = train_grpo_one_seed(returns, seed, device)
    results.append(r)
    print(f"Seed {seed}: Sharpe port={r['sharpe_port']:.3f}, BH={r['sharpe_bh']:.3f}, "
          f"delta={r['delta_sharpe']:+.3f}, episodes={r['n_episodes']}")

print(f"\nDone: {len(results)} seeds on {device}")

Seed 0: Sharpe port=0.714, BH=0.732, delta=-0.018, episodes=120
Seed 1: Sharpe port=0.698, BH=0.732, delta=-0.034, episodes=120
Seed 7: Sharpe port=0.711, BH=0.732, delta=-0.021, episodes=120
Seed 42: Sharpe port=0.725, BH=0.732, delta=-0.008, episodes=120

Done: 4 seeds on cuda

6. Résultats et verdict GRPO

print("=" * 70)
print("VERDICT -- QC-Py-RL-06: GRPO (Group Relative Policy Optimization)")
print("=" * 70)

deltas = [r["delta_sharpe"] for r in results]
mean_delta = np.mean(deltas)
std_delta = np.std(deltas, ddof=1) if len(deltas) > 1 else float("nan")
sigma_edge = mean_delta / std_delta if std_delta > 1e-9 else float("nan")
n_positive = sum(1 for d in deltas if d > 0)

mean_port = np.mean([r["sharpe_port"] for r in results])
mean_bh = np.mean([r["sharpe_bh"] for r in results])

print(f"GRPO Portfolio Sharpe (mean): {mean_port:+.3f}")
print(f"Equal-Weight Sharpe:          {mean_bh:.3f}")
print(f"Delta mean:                   {mean_delta:+.3f}")
print(f"Sigma edge:                   {sigma_edge:+.2f}")
print(f"Seeds positive vs EW:         {n_positive}/{len(results)}")
print()

if sigma_edge >= 2.0 and n_positive >= 3:
    verdict = "BEATS"
elif n_positive == 0:
    verdict = "NO BEATS"
else:
    verdict = "INCONCLUSIVE"

print(f">>> VERDICT: {verdict} <<<")
print()
if verdict == "NO BEATS":
    print("GRPO (sans critic, avantage relatif intra-groupe) ne depasse pas")
    print("l'allocation equal-weight passive sur le panier anti-biais.")
elif verdict == "INCONCLUSIVE":
    print("Signal partiel mais non robuste cross-seed.")
else:
    print("GRPO capture un signal exploitable — premier BEATS de la serie !")
print()

print("=" * 70)
print("RL Series Progress (#1461)")
print("=" * 70)
series = [
    ("RL-1. Q-Learning Tabulaire", "NO BEATS", "PR #1581"),
    ("RL-2. PPO (numpy)", "NO BEATS", "PR #1583"),
    ("RL-3. Reward Shaping (PyTorch)", "NO BEATS", "PR #1584"),
    ("RL-4. Multi-Asset PPO", "NO BEATS", "PR #1585"),
    ("RL-5. Tactical Overlay", "NO BEATS", "PR #1587"),
    ("RL-6. GRPO (Group Relative)", verdict, "This PR"),
]
for name, v, pr in series:
    print(f"  {name:40s} | {v:15s} | {pr}")

======================================================================
VERDICT -- QC-Py-RL-06: GRPO (Group Relative Policy Optimization)
======================================================================
GRPO Portfolio Sharpe (mean): +0.712
Equal-Weight Sharpe:          0.732
Delta mean:                   -0.020
Sigma edge:                   -1.86
Seeds positive vs EW:         0/4

>>> VERDICT: NO BEATS <<<

GRPO (sans critic, avantage relatif intra-groupe) ne depasse pas
l'allocation equal-weight passive sur le panier anti-biais.

======================================================================
RL Series Progress (#1461)
======================================================================
  RL-1. Q-Learning Tabulaire               | NO BEATS        | PR #1581
  RL-2. PPO (numpy)                        | NO BEATS        | PR #1583
  RL-3. Reward Shaping (PyTorch)           | NO BEATS        | PR #1584
  RL-4. Multi-Asset PPO                    | NO BEATS        | PR #1585
  RL-5. Tactical Overlay                   | NO BEATS        | PR #1587
  RL-6. GRPO (Group Relative)              | NO BEATS        | This PR