RL-5 : Tactical Overlay — RL sur B&H

Série RL for Trading (#1461) | RL-1 Q-Learning · RL-2 PPO · RL-3 Reward Shaping · RL-4 Multi-Asset · RL-5 Tactical Overlay

Objectifs

  • Utiliser un agent RL comme couche tactique au-dessus d’un portefeuille B&H equal-weight
  • L’agent décide d’un biais de sur/sous-pondération par actif (pas des poids absolus)
  • Comparer : B&H seul vs B&H + RL overlay (long-only)
  • Validation multi-seed (0/1/7/42) avec edge ≥ 2σ

Principe

Au lieu de remplacer l’allocation B&H, l’agent RL ajuste marginalement chaque position : - Action ∈ [-1, +1] par actif (biais tactique) - Poids final = poids_B&H × (1 + action × overlay_scale) - Si overlay_scale=0.3, l’agent peut surpondérer de 30% ou sous-pondérer de 30%

L’hypothèse : un petit ajustement RL ajouté à B&H peut capturer des opportunités tactiques sans détruire la diversification du portefeuille passif.

Durée estimée : ~5 min (Papermill) | Prérequis : PyTorch, panier L1/L2

import numpy as np
import pandas as pd
import os
os.environ.setdefault('CUBLAS_WORKSPACE_CONFIG', ':4096:8')  # avant import torch : workspace deterministe CuBLAS
import torch
import torch.nn as nn
import torch.optim as optim
from pathlib import Path
import warnings
warnings.filterwarnings("ignore")

# Determinisme explicite : les seeds par-run (cellule d'entrainement) ne suffisent pas seuls
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True, warn_only=True)

SEEDS = [0, 1, 7, 42]
LOOKBACK = 20
N_EPISODES = 150
EPISODE_LEN = 252
HIDDEN_DIM = 128
LR = 3e-4
GAMMA = 0.99
GAE_LAMBDA = 0.95
CLIP_EPS = 0.2
ENTROPY_COEF = 0.01
OVERLAY_SCALE = 0.3
FEE_BPS = 10
BATCH_SIZE_EP = 64

DATA_PATH = Path("../datasets/panier/panier_close_all.csv")
print(f"PyTorch {torch.__version__}, CUDA={torch.cuda.is_available()}")
print(f"Config: overlay_scale={OVERLAY_SCALE}, fee_bps={FEE_BPS}, episodes={N_EPISODES}")
PyTorch 2.6.0+cu124, CUDA=True
Config: overlay_scale=0.3, fee_bps=10, episodes=150

1. Chargement du panier anti-biais

Même panier 24-actifs que RL-04 (excluant XLC pour sparsity).

df_raw = pd.read_csv(DATA_PATH, index_col=0, parse_dates=True)
df_raw.index = pd.DatetimeIndex(df_raw.index)
df_raw = df_raw.sort_index()

# Exclude XLC (sparse, <50% coverage)
exclude = [c for c in df_raw.columns if df_raw[c].notna().sum() < len(df_raw) * 0.5]
df = df_raw.drop(columns=exclude)
df = df.ffill().dropna()

n_assets = df.shape[1]
symbols = list(df.columns)
returns = df.pct_change().dropna()

print(f"Panier: {n_assets} assets, {len(returns)} business days")
print(f"Range: {returns.index.min().date()} to {returns.index.max().date()}")
print(f"Assets: {symbols[:6]}... + {n_assets-6} more")
Panier: 25 assets, 3116 business days
Range: 2017-11-10 to 2026-05-22
Assets: ['SPY', 'RSP', 'IWM', 'XLF', 'XLK', 'XLV']... + 19 more

2. Environnement Tactical Overlay

L’environnement wrappe un portefeuille B&H equal-weight. L’agent RL observe les rendements récents et produit un biais par actif.

Composant Description
State Rendements normalisés LOOKBACK × N_ASSETS
Action Biais ∈ [-1, +1] par actif (tanh-squashed)
Poids final w_i = (1/N) × (1 + action_i × OVERLAY_SCALE)
Reward Rendement différentiel (overlay - B&H) 
class TacticalOverlayEnv:
    """B&H equal-weight portfolio with RL tactical overlay adjustments."""

    def __init__(self, returns_df, lookback=20, episode_len=252, overlay_scale=0.3, fee_bps=10):
        self.returns = returns_df.values
        self.dates = returns_df.index
        self.n_assets = returns_df.shape[1]
        self.lookback = lookback
        self.episode_len = episode_len
        self.overlay_scale = overlay_scale
        self.fee = fee_bps / 10000
        self.base_weight = 1.0 / self.n_assets
        self._idx = None
        self._prev_action = None

    @property
    def state_dim(self):
        return self.lookback * self.n_assets

    @property
    def action_dim(self):
        return self.n_assets

    def reset(self, start_idx=None):
        max_start = len(self.returns) - self.lookback - self.episode_len - 1
        if start_idx is None:
            start_idx = np.random.randint(self.lookback, max(max_start, self.lookback + 1))
        self._idx = start_idx
        self._prev_action = np.zeros(self.n_assets)
        return self._get_state()

    def _get_state(self):
        window = self.returns[self._idx - self.lookback:self._idx]
        std = window.std(axis=0, keepdims=True) + 1e-8
        normalized = window / std
        return normalized.flatten()

    def step(self, action):
        self._idx += 1
        done = self._idx >= len(self.returns) - 1

        # Compute overlay weights
        bias = action * self.overlay_scale
        weights = np.full(self.n_assets, self.base_weight) * (1.0 + bias)
        weights = np.clip(weights, 0, None)
        w_sum = weights.sum()
        if w_sum > 0:
            weights /= w_sum
        else:
            weights = np.full(self.n_assets, self.base_weight)

        # Portfolio return
        day_ret = self.returns[self._idx]
        port_ret = np.dot(weights, day_ret)

        # B&H equal-weight return
        bh_ret = day_ret.mean()

        # Transaction cost on weight changes
        turnover = np.abs(weights - self._prev_action).sum()
        fee_cost = turnover * self.fee * 0.5
        port_ret -= fee_cost

        self._prev_action = weights.copy()

        # Reward = differential return
        reward = (port_ret - bh_ret) * 100

        info = {"port_ret": port_ret, "bh_ret": bh_ret, "turnover": turnover}
        return self._get_state() if not done else np.zeros(self.state_dim), reward, done, info

3. Réseau Actor-Critic

Actor produit un biais ∈ [-1, +1] par actif via tanh. Critic estime la valeur de l’état.

class TacticalActorCritic(nn.Module):
    """Actor-Critic for tactical overlay. Actor outputs tanh-squashed biases."""

    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )
        self.actor_mean = nn.Linear(hidden_dim, action_dim)
        self.actor_log_std = nn.Parameter(torch.zeros(action_dim))
        self.critic = nn.Linear(hidden_dim, 1)

    def forward(self, state):
        feat = self.backbone(state)
        mean = torch.tanh(self.actor_mean(feat))
        std = torch.exp(self.actor_log_std.clamp(-2, 0))
        value = self.critic(feat).squeeze(-1)
        return mean, std, value


def collect_rollout(env, model, device):
    """Collect one episode of experience."""
    states, actions, rewards, log_probs, values, dones = [], [], [], [], [], []
    state = env.reset()

    for _ in range(env.episode_len):
        s = torch.FloatTensor(state).unsqueeze(0).to(device)
        with torch.no_grad():
            mean, std, value = model(s)
        dist = torch.distributions.Normal(mean.squeeze(0), std.squeeze(0))
        action = dist.sample()
        log_prob = dist.log_prob(action).sum()

        action_np = action.cpu().numpy()
        next_state, reward, done, _ = env.step(action_np)

        states.append(state)
        actions.append(action_np)
        rewards.append(reward)
        log_probs.append(log_prob.item())
        values.append(value.item())
        dones.append(done)

        state = next_state
        if done:
            break

    return {
        "states": np.array(states),
        "actions": np.array(actions),
        "rewards": np.array(rewards),
        "log_probs": np.array(log_probs),
        "values": np.array(values),
        "dones": np.array(dones)
    }

4. Entraînement PPO

PPO clip + GAE + entropy bonus. Même pipeline que RL-03/RL-04.

def compute_gae(rewards, values, dones, gamma=0.99, lam=0.95):
    """Generalized Advantage Estimation."""
    advantages = []
    gae = 0
    values_list = list(values) + [0]
    for t in reversed(range(len(rewards))):
        delta = rewards[t] + gamma * values_list[t+1] * (1 - dones[t]) - values_list[t]
        gae = delta + gamma * lam * (1 - dones[t]) * gae
        advantages.insert(0, gae)
    return np.array(advantages)


def ppo_update(model, optimizer, rollout, device, clip_eps=0.2, entropy_coef=0.01):
    """One PPO update epoch on collected rollout."""
    states = torch.FloatTensor(rollout["states"]).to(device)
    actions = torch.FloatTensor(rollout["actions"]).to(device)
    old_log_probs = torch.FloatTensor(rollout["log_probs"]).to(device)
    advantages = torch.FloatTensor(rollout["advantages"]).to(device)
    returns = torch.FloatTensor(rollout["returns"]).to(device)

    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

    total_loss = 0
    n_batches = 0
    indices = np.arange(len(states))
    np.random.shuffle(indices)

    for start in range(0, len(states), BATCH_SIZE_EP):
        idx = indices[start:start+BATCH_SIZE_EP]
        batch_s = states[idx]
        batch_a = actions[idx]
        batch_old_lp = old_log_probs[idx]
        batch_adv = advantages[idx]
        batch_ret = returns[idx]

        mean, std, value = model(batch_s)
        dist = torch.distributions.Normal(mean, std)
        new_log_prob = dist.log_prob(batch_a).sum(dim=-1)
        entropy = dist.entropy().sum(dim=-1).mean()

        ratio = torch.exp(new_log_prob - batch_old_lp)
        surr1 = ratio * batch_adv
        surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * batch_adv
        actor_loss = -torch.min(surr1, surr2).mean()
        critic_loss = nn.MSELoss()(value, batch_ret)
        loss = actor_loss + 0.5 * critic_loss - entropy_coef * entropy

        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        n_batches += 1

    return total_loss / max(n_batches, 1)

5. Boucle d’entraînement multi-seed

4 seeds × 150 épisodes. Sharpe OOS sur la dernière portion du dataset.

def train_one_seed(returns_df, seed, device="cpu"):
    """Train tactical overlay PPO for one seed, return OOS metrics."""
    torch.manual_seed(seed)
    np.random.seed(seed)

    env = TacticalOverlayEnv(
        returns_df, lookback=LOOKBACK, episode_len=EPISODE_LEN,
        overlay_scale=OVERLAY_SCALE, fee_bps=FEE_BPS
    )
    model = TacticalActorCritic(env.state_dim, env.action_dim, HIDDEN_DIM).to(device)
    optimizer = optim.Adam(model.parameters(), lr=LR)

    losses = []
    for ep in range(N_EPISODES):
        rollout = collect_rollout(env, model, device)
        advantages = compute_gae(rollout["rewards"], rollout["values"], rollout["dones"], GAMMA, GAE_LAMBDA)
        rollout["advantages"] = advantages
        rollout["returns"] = advantages + np.array(rollout["values"])
        loss = ppo_update(model, optimizer, rollout, device, CLIP_EPS, ENTROPY_COEF)
        losses.append(loss)

    # OOS evaluation: last 500 days
    oos_start = len(returns_df) - 500
    oos_rets_port = []
    oos_rets_bh = []
    state = env.reset(start_idx=oos_start)
    model.eval()
    with torch.no_grad():
        for t in range(min(500, len(returns_df) - oos_start - 1)):
            s = torch.FloatTensor(state).unsqueeze(0).to(device)
            mean, std, _ = model(s)
            action = mean.squeeze(0).cpu().numpy()
            next_state, reward, done, info = env.step(action)
            oos_rets_port.append(info["port_ret"])
            oos_rets_bh.append(info["bh_ret"])
            state = next_state
            if done:
                break

    port_arr = np.array(oos_rets_port)
    bh_arr = np.array(oos_rets_bh)

    def sharpe(r):
        return float(np.mean(r) / (np.std(r, ddof=1) + 1e-8) * np.sqrt(252))

    return {
        "seed": seed,
        "sharpe_port": sharpe(port_arr),
        "sharpe_bh": sharpe(bh_arr),
        "delta_sharpe": sharpe(port_arr) - sharpe(bh_arr),
        "mean_ret_port": float(np.mean(port_arr)) * 252,
        "mean_ret_bh": float(np.mean(bh_arr)) * 252,
        "n_oos": len(port_arr),
        "final_loss": losses[-1] if losses else float("nan")
    }

# Run all seeds
device = "cuda" if torch.cuda.is_available() else "cpu"
results = []
for seed in SEEDS:
    r = train_one_seed(returns, seed, device)
    results.append(r)
    print(f"Seed {seed}: Sharpe port={r['sharpe_port']:.3f}, BH={r['sharpe_bh']:.3f}, "
          f"delta={r['delta_sharpe']:+.3f}, OOS={r['n_oos']} days")

print(f"\nDone: {len(results)} seeds on {device}")
Seed 0: Sharpe port=1.020, BH=1.080, delta=-0.059, OOS=499 days
Seed 1: Sharpe port=1.069, BH=1.080, delta=-0.011, OOS=499 days
Seed 7: Sharpe port=1.144, BH=1.080, delta=+0.064, OOS=499 days
Seed 42: Sharpe port=1.031, BH=1.080, delta=-0.048, OOS=499 days

Done: 4 seeds on cuda

6. Résultats et verdict

Edge ≥ 2σ cross-seed pour BEATS. Baseline = B&H equal-weight.

print("=" * 70)
print("VERDICT -- QC-Py-RL-05: Tactical Overlay on B&H")
print("=" * 70)

deltas = [r["delta_sharpe"] for r in results]
mean_delta = np.mean(deltas)
std_delta = np.std(deltas, ddof=1) if len(deltas) > 1 else float("nan")
sigma_edge = mean_delta / std_delta if std_delta > 1e-9 else float("nan")
n_positive = sum(1 for d in deltas if d > 0)

mean_port_sharpe = np.mean([r["sharpe_port"] for r in results])
mean_bh_sharpe = np.mean([r["sharpe_bh"] for r in results])

print(f"RL+Overlay Sharpe (mean):  {mean_port_sharpe:+.3f}")
print(f"B&H Equal-Weight Sharpe:    {mean_bh_sharpe:.3f}")
print(f"Delta mean:                 {mean_delta:+.3f}")
print(f"Sigma edge:                 {sigma_edge:+.2f}")
print(f"Seeds positive vs B&H:      {n_positive}/{len(results)}")
print()

if sigma_edge >= 2.0 and n_positive >= 3:
    verdict = "BEATS"
elif n_positive == 0:
    verdict = "NO BEATS"
else:
    verdict = "INCONCLUSIVE"

print(f">>> VERDICT: {verdict} <<<")
print()
if verdict == "NO BEATS":
    print("L'overlay tactique RL ne capture pas d'opportunite exploitable.")
    print("Le B&H equal-weight reste superieur meme avec ajustement marginal.")
elif verdict == "INCONCLUSIVE":
    print("Resultats mixtes. L'overlay montre un signal partiel mais non robuste.")
else:
    print("L'overlay tactique capture un signal exploitable apres couts.")
print()

print("=" * 70)
print("RL Series Progress (#1461)")
print("=" * 70)
series = [
    ("RL-1. Q-Learning Tabulaire", "NO BEATS", "PR #1581"),
    ("RL-2. PPO (numpy)", "NO BEATS", "PR #1583"),
    ("RL-3. Reward Shaping (PyTorch)", "NO BEATS", "PR #1584"),
    ("RL-4. Multi-Asset PPO Portfolio", "NO BEATS", "PR #1585"),
    ("RL-5. Tactical Overlay + B&H", verdict, "This PR"),
]
for name, v, pr in series:
    print(f"  {name:40s} | {v:15s} | {pr}")
======================================================================
VERDICT -- QC-Py-RL-05: Tactical Overlay on B&H
======================================================================
RL+Overlay Sharpe (mean):  +1.066
B&H Equal-Weight Sharpe:    1.080
Delta mean:                 -0.014
Sigma edge:                 -0.24
Seeds positive vs B&H:      1/4

>>> VERDICT: INCONCLUSIVE <<<

Resultats mixtes. L'overlay montre un signal partiel mais non robuste.

======================================================================
RL Series Progress (#1461)
======================================================================
  RL-1. Q-Learning Tabulaire               | NO BEATS        | PR #1581
  RL-2. PPO (numpy)                        | NO BEATS        | PR #1583
  RL-3. Reward Shaping (PyTorch)           | NO BEATS        | PR #1584
  RL-4. Multi-Asset PPO Portfolio          | NO BEATS        | PR #1585
  RL-5. Tactical Overlay + B&H             | INCONCLUSIVE    | This PR
Retour au sommet