Nifty 50 Reinforcement-Learning Trader: a plain-English explainer (v2)

Written for Satya, 5 Oct 2026. Status: final for v2. The v2 run (full 2M-step training, then a 4-fold walk-forward) started at 10:49 IST and finished at 11:16 IST. Every number here comes from the project's code and result files (results/full_training_2000k_results.json, results/walk_forward_results.json), or from the v2 change notes. No numbers are made up. Full v2 results with equity charts: v2 results page.

0. The short version

• What it is. A computer "agent" looks at Nifty 50 once an hour and picks one of three actions: stay flat, go long, or go short. When it enters a trade, it also chooses how wide the stop-loss is and how far away the target is. It teaches itself by trial and error on historical data. This is called reinforcement learning (RL).
• What we know so far.
• The first short training run (80k steps, delivered 4 Oct) lost 6.40% on the sealed 2025-07-10 to 2026-10-01 test period.
• Nifty buy-and-hold lost 11.69% over the same period. The agent lost less than the market but still lost money.
• A hyperparameter search then picked a winning configuration, large_batch.
• What changed in v2.
• Costs: trading costs now follow the real NSE rules by date. STT and lot size changed several times between 2022 and 2026.
• Model selection: the sealed test now scores the best-validation checkpoint. The last-step model is reported separately to show overfitting.
• Walk-forward: it uses the search winner.
• Features: two useless forex "session" features were dropped.
• Speed: 8 simulators now run in parallel, about 6× faster.
• v2 results (finished 11:16 IST). See the v2 results page.
• Sealed test (2025-07-10 to 2026-10-01), best-val checkpoint: −19.11% return, max drawdown −21.39%, 567 trades, win rate 41.45%, profit factor 0.80, Sharpe −1.82. Buy-and-hold was −11.69%, so the agent was −7.43 pp behind the market.
• Last-step model (same test): −22.77%, drawdown −24.58%, 952 trades, PF 0.87, −11.08 pp vs buy-and-hold. Worse than the best-val checkpoint, which confirms the late overfitting.
• Walk-forward (4 × 200k, best-val checkpoint per fold): 1 of 4 folds made money and 2 of 4 beat buy-and-hold.
  • Fold 1 (2023-01-23 to 2023-04-20): +1.12% vs buy-and-hold −2.73%
  • Fold 2 (2024-03-19 to 2024-06-14): −0.43% vs buy-and-hold +7.13%
  • Fold 3 (2025-05-19 to 2025-08-06): −4.04% vs buy-and-hold −1.73%
  • Fold 4 (2026-07-13 to 2026-10-01): −4.27% vs buy-and-hold −7.39%
• An older 2M run (old costs) was stopped at about 1.46M steps. It is not a result.
• Honest bottom line. No trading edge has been shown. With realistic date-aware costs, the v2 agent lost more than buy-and-hold on the sealed test and was inconsistent across walk-forward folds. Earlier logs showed strong overfitting: the model gets very good on the training years while getting worse on validation (§9). Section 9 lists what the final numbers must show before anyone takes this seriously.

1. Overview: what the project is and where it came from

Origin. The code is adapted from Ziad Francis's open-source project Reinforcement_Trading_Part_2. That project was built for forex-style data. This project points it at Nifty 50 1-minute data and adds:

• a Nifty data loader (nifty_data.py)
• regime flags for trend, range, high-vol and low-vol (regime_detection.py)
• a date-aware Indian F&O cost model (broker_costs.py)
• a hyperparameter search and walk-forward testing (hyperparam_search.py, walk_forward.py)
• a driver script for the GPU runs (run_extended_training.py)

Reinforcement learning in one paragraph. You never tell the agent "this is a good trade." It simply tries things inside a market simulator. After each step, the simulator gives back a reward: a score that goes up when equity goes up and down when equity falls. Over millions of steps, the agent adjusts itself to collect more reward. Think of a new trader paper-trading years of charts at high speed, with only the P&L for feedback.

The algorithm: PPO. The learning method is Proximal Policy Optimization (PPO) from the standard stable-baselines3 library. "Proximal" means each update may change the agent's behaviour only a little (clip_range = 0.1), which keeps learning stable. The agent's "brain" is a small neural network called an MLP (multi-layer perceptron): a few layers of numbers that turn the market snapshot into action probabilities. The winning setup uses two hidden layers of 256 neurons each.

The pipeline at a glance

2. The data

How the hourly bars are built. Each bar is labelled by its close time on clock-hour boundaries. That gives seven bars a day: 10:00, 11:00, 12:00, 13:00, 14:00, 15:00, and a final bar labelled 16:00. The 10:00 bar covers only 09:15–10:00 (45 minutes), and the last bar covers only 15:00–15:30 (30 minutes).

The first ~250 hourly bars (about Jan–Feb 2022) are dropped as warm-up. Indicators like the 200-period EMA need history before they mean anything. That's why every training window starts on 2022-02-22.

Why check stops and targets on 1-minute bars? An hourly candle can show that both your stop and your target were touched, but it can't say which came first. So the simulator walks through the 60 one-minute candles inside the hour, in order. If a single 1-minute candle touches both the stop and the target, the simulator assumes the stop was hit first. This rule is deliberately pessimistic (conservative), so it cannot flatter the results.

What is NOT in the data. The agent sees only Nifty's own price history. Specifically, there is:

• no order flow, bid/ask or depth
• no market breadth (advance/decline)
• no open interest or options data
• no VIX
• no FII/DII flows
• no news

Nothing was invented to fill these gaps. The index "volume" column isn't used by any feature.

3. The features: what the agent "sees"

A feature is one number the agent looks at when deciding. All features are causal: each uses only data up to and including the current bar's close, never anything later.

Most are ATR-normalised, meaning divided by the Average True Range (ATR, the typical bar range over the last 14 bars). "Close is 2 ATRs above the 20 EMA" then means the same thing at Nifty 17,000 and at Nifty 26,000, which lets one model work across years.

23 price/volatility features (from repo/features.py):

What was removed:

• v2: the two forex session flags session_newyork and session_london_ny_overlap. They are always 0 during NSE hours, so they carried no information.
• An earlier audit: four near-duplicate features (correlation 0.96–1.00): ema50_slope5_atr, rsi_centered, roc5_atr and body_atr.

4 regime flags (from regime_detection.py), each 0 or 1:

• is_trend: this bar moved more than 1 ATR close-to-close
• is_range: this bar moved less than 0.5 ATR
• is_high_vol: current ATR is in the top 25% of the last 50 bars
• is_low_vol: current ATR is in the bottom 25% of the last 50 bars

That makes 27 market features. Missing values (NaN) are filled with 0.

The simulator adds 6 "my position" numbers:

• whether I'm long, short or flat
• open P&L in R
• how long I've been in the trade
• distance to target in ATRs
• distance to stop in ATRs
• planned reward-to-risk

So each decision uses 33 numbers. Before they reach the network, they are rescaled to roughly mean 0 and spread 1, then clipped at ±10. A wrapper called VecNormalize does this using running averages from the training data, often just called the "vecnorm" stats.

4. How the agent decides and trades

4.1 The choices (the "action space")

Every hour, the agent outputs three choices at once. Technically, this is a Gymnasium MultiDiscrete([3, 3, 4]) action.

R means "one unit of risk": the amount you lose if the stop is hit. A 2R target pays twice what the stop costs.

Position sizing. Each trade risks 0.5% of current equity. Starting equity is 10,000, an abstract account unit rather than literally rupees, so the first trade risks 50. Position size is risk ÷ stop distance: a wider stop means a smaller position. The simulator allows fractional sizes rather than whole Nifty lots.

4.2 A worked example: one hour

The prices are illustrative, to show the mechanics. They are not from a real trade log.

1. 11:00 IST bar closes. Nifty closes at 25,000 and the hourly ATR(14) is 50 points. Equity is 10,000 and the agent is flat. The 33 numbers above are computed from data up to 11:00, and nothing later.
2. The agent picks [long, SL 1.0×ATR, TP 2R].
3. Entry. The simulator buys at the close plus half the spread plus slippage, for example 25,000 + 0.25 + 0.5 = 25,000.75.
   • Stop distance = 1.0 × 50 = 50 points, so SL = 24,950.75.
   • Target = 2 × 50 = 100 points, so TP = 25,100.75.
   • Risk = 0.5% × 10,000 = 50, so size = 50 ÷ 50 = 1.0 unit.
4. 11:01 to 12:00. The simulator checks each 1-minute candle in order:
   • If a candle's low reaches the SL, the trade closes at the stop: −1R before costs.
   • If a candle's high reaches the TP, the trade closes at the target: +2R before costs.
   • If one candle touches both, it counts as a stop (the pessimistic rule).
5. Costs. Slippage and spread worsen the fills. Charges such as STT, exchange fees, stamp duty and GST are deducted using the rates in force on that trade's date (§5).
6. Reward for the hour = change in equity ÷ the 0.5% risk unit, which is roughly "R gained or lost". While a trade is open, there are two tiny extra terms:
   • plus 1% of the open P&L in R, marked at the bar close the agent already saw
   • minus 0.00002 for holding
7. 12:00 bar closes. If the trade is still open, the agent decides again:
   • Same direction: the bracket stays.
   • Flat: the trade closes at the 12:00 close.
   • Opposite direction: the trade closes and reverses.

4.3 What "one step" and "2 million steps" mean

One step is one hourly decision. During training, the agent plays episodes of 2,048 hours (roughly 1.2 years), each starting from a random point in the training data. v2 runs 8 of these simulators in parallel.

The full run's training window has 4,877 hourly bars. So 2,000,000 steps is about 410 passes over the same history, which is a lot. That's why overfitting is the main risk (§9).

5. Trading costs: date-aware model, Zerodha/Upstox style

broker_costs.py now charges each trade at the rates in force on its date, taken from NSE circulars. These are still assumptions for a typical discount broker, not exact broker bills.

Rates that change by date

Rates held constant

Worked examples. Each is 1 lot and a +100-point round trip, including slippage:

The cost of a round trip has nearly tripled in points over the data period, mostly because of STT. Much of the sealed test window (2025-07-10 to 2026-10-01) is in the most expensive regime. That makes the test harder, and more realistic, than before.

For comparison:

• The 80k baseline (§8.1) used an old rough model of 3.5 points per trade.
• The hyperparameter search (§8.2) used an earlier flat model of about 5.9 points.

Note: I haven't re-read the v2 simulator code line by line. This section describes the cost schedule and the worked examples from broker_costs.py, not every detail of how the simulator applies them.

6. How the training was run

6.1 Words used in this section

• Train / validation / test. History is cut into time blocks, in order:
  • Train: the agent learns from it.
  • Validation: used to compare settings and pick checkpoints. The agent never learns from it directly, but our choices are guided by it.
  • Test: locked away (sealed) until the very end, to give one honest out-of-sample number.
• Embargo. A 50-bar gap (about 7 trading days) between blocks. It stops indicators and trades from one block spilling into the next.
• Hyperparameters. Settings of the learning process itself:
  • learning rate: how big each adjustment step is
  • batch size: how many experiences are averaged per adjustment
  • network size: how many "neurons" the MLP has
• Checkpoint. A saved snapshot of the agent partway through training, together with its matching vecnorm stats.

6.2 Stage 1: hyperparameter search (DONE)

• 8 configurations: baseline, low_lr, high_lr, small_batch, large_batch, deep_net, wide_net, shallow_net.
• Each was trained separately on 2 time folds for 50,000 steps each, so 16 runs in total.

How the winner was chosen (hyperparam_search.py, get_best_config):

1. Score each config by its average validation excess return vs buy-and-hold across the two folds, in percentage points (pp). Excess return is the agent's return minus what simply holding Nifty returned over the same window.
2. Pick the highest average.

There is no extra eligibility filter at this stage. Drawdown and trade count are recorded but don't affect the ranking.

Winner: large_batch. Learning rate 0.0003, batch 256, network [256, 256], rollout 2,048 steps. It averaged +6.39 pp of excess, against +5.80 pp for second-placed shallow_net. The full table is in §8.2.

The search was run with the earlier code. Costs were a flat ~5.9 points, and each run's validation score came from its last-step 50k model. It was not re-run for v2. v2 reuses its winner (--use_best_hyperparams).

6.3 Checkpoint choice inside every run (the "consistency score")

During each run, the code regularly pauses to score the agent on two slices:

• the last ~761 bars of the training data
• the validation block

For each slice it computes quality = cumulative reward − 1.0 × max drawdown %. The checkpoint's score is the weaker of the two (min(q_train, q_val)).

A checkpoint can become "best" only if it is eligible: positive reward on both slices and at least 5 trades on each. This stops a do-nothing agent (zero trades, zero drawdown) from winning. The best eligible checkpoint is saved to best_model/ with its own vecnorm stats.

v2 change: the sealed test (and each walk-forward fold's test) now scores this best-validation checkpoint with its matching vecnorm. The model from the very last training step is evaluated separately, so the gap between the two shows how much overfitting happened late in training.

6.4 Stage 2: full training, 2,000,000 steps (v2, DONE)

What overfitting looked like in the earlier run. These are checkpoint scores from the stopped old-cost run. This is not a result; it is shown only to explain the pattern. Values are cumulative reward in R units: 1 R ≈ 0.5% of equity, so +20 R is very roughly +10%.

The training score kept climbing while validation fell. That's textbook overfitting: the agent memorises 2022–2024 instead of learning something that carries over to 2025. It's exactly why v2 now tests the best-validation checkpoint rather than the last step.

6.5 Stage 3: walk-forward test (v2, DONE)

Walk-forward means repeating train → validate → test on several stretches of history instead of just one. It shows whether results hold up across different market conditions.

• Budget: 800,000 steps split over 4 folds, 200,000 steps each.
• Settings: the large_batch winner (--use_best_hyperparams). Each fold's test scores that fold's own best-validation checkpoint.

• Fold layout (walk_forward.py): the 7,967 bars are cut into 4 consecutive, non-overlapping blocks of about 1,990 bars, roughly 14 months each. Within each block:

  • the first 60% is training
  • the next 20% is validation
  • the last 20% (about 400 bars, roughly 2.5–3 months) is the out-of-sample test

  Each part is separated by a 50-bar embargo. - Output: per-fold test return, drawdown, trade count and excess vs buy-and-hold go to results/walk_forward_results.json. Fold test windows and results are in §8.3.

6.6 Timeline (all times IST)

6.7 GPU use and parallel environments

PPO with a small MLP doesn't need much GPU. Each cycle has two phases:

1. Collect experience. The simulator steps through hours and scans 1-minute candles on the CPU.
2. Update the network. The GPU does a few quick passes over the collected samples, in a short burst.

With one simulator, the GPU sat idle most of the time (~323 steps/s). v2 runs 8 simulators in parallel (SubprocVecEnv, one process each). This lifted speed to 1,900–2,200 steps/s (about 6×), so a 2M-step run now takes roughly 15–20 minutes of training instead of about 1¾ hours. The GPU is still not the bottleneck. That's expected for this kind of model.

7. Anti-leakage design: how the project avoids cheating

Leakage, or look-ahead bias, is when a backtest accidentally uses information that wouldn't have been available at the time. It's the most common reason backtests look great and live trading doesn't.

What the project does about it:

1. Causal features. Every indicator uses only past and current bars: EMAs, rolling windows, and shift(k) for past returns. repo/leakage_checks.py has a guard that recomputes features after appending future rows and fails if any past value changes.
2. Decide at the bar close, execute afterwards. The agent decides at the hourly close it has already seen, and the trade plays out on the following 1-minute candles. The open-P&L part of the reward uses only that known close.
3. Pessimistic intrabar rule. If the stop and target are touched in the same minute, the stop counts first.
4. Chronological splits with embargoes. Train comes before validation, which comes before test, with 50-bar gaps between them. Nothing is shuffled across time.
5. Sealed test, one model chosen without looking at it. The test window is evaluated once, at the end. The model scored on it is chosen by validation only (the best-val checkpoint), and the choice is fixed before the test is opened. The search folds' own test windows were never touched.
6. Normalisation travels with the model. Each checkpoint is saved with its own vecnorm stats, frozen during evaluation, so validation and test data never influence them.

Still open: the search overlaps the test window. The hyperparameter search's fold 2 used 2024-06 to 2025-10 for training and 2025-10-31 to 2026-04-08 for validation. Both overlap the full run's sealed test window. No agent traded the test window while the final model was being trained. But the choice of large_batch was partly informed by how configs did in months that fall inside that test window. That's mild selection leakage. The walk-forward, and later live paper-trading, are the cleaner checks.

8. Results so far

8.1 Baseline: 80k steps on CPU (delivered 4 Oct)

What the measures mean:

• Max drawdown: the worst peak-to-trough fall in equity.
• Win rate: the share of trades that made money.
• Profit factor (PF): gross profits ÷ gross losses. Below 1.0 means losing overall.
• Sharpe-like ratio: return per unit of volatility, annualised.

Reading it. The agent lost less than the market in a falling year, but it still lost money. With PF 0.95 over 554 trades, the 42.8% win rate didn't cover costs.

This run used the old rough 3.5-point cost model. Under v2's date-aware costs (about 8–16 points per trade in the test window), the same trading would have done worse. So the v2 numbers will be compared against a tougher bar.

8.2 Hyperparameter search results (DONE, earlier code)

Source: results/hyperparam_search_results.json. These are validation numbers only; the test windows stayed sealed. Configs are sorted by score.

Buy-and-hold over the validation windows returned +10.34% in fold 1 (a rising market) and −7.04% in fold 2 (a falling market). Each run was 50,000 steps, seed 42, with the earlier flat cost model and 29 features.

What to take from it:

• Fold 1 (rising market) was hard. Only large_batch beat buy-and-hold, and only by +0.77 pp. Six of the eight configs lost money while Nifty rose 10%.
• Fold 2 (falling market) was easy to "beat". Nifty fell 7%, so an agent that never traded would score about +7 pp excess. Most of every config's score comes from this fold.
• The top two are close (+6.39 vs +5.80 pp). With only 2 folds and 1 random seed, the gap could easily be luck. large_batch is a reasonable choice, not a proven one.
• Overfitting was already visible at 50k steps. On fold 1, six configs scored +14 to +36 R on the training slice but negative on validation.

8.3 The v2 full run and walk-forward

Sealed test, 2025-07-10 to 2026-10-01 (2,129 hourly bars), date-aware costs:

Walk-forward, 4 folds × 200k steps, large_batch, each fold's best-val checkpoint:

Against the §9 checklist:

1. Positive sealed-test return: no (−19.11%).
2. Consistent across folds: no. Only fold 1 made money, and only folds 1 and 4 beat buy-and-hold. Every fold looked good on validation (+2.4% to +6.9%) but fell back on test, which is the same overfitting pattern seen in training.
3. Profit factor above 1.1: no (0.80).
4. Drawdown no worse than buy-and-hold: no (−21.39% vs −15.72%).
5. Enough trades: yes (567).
6. Robustness across seeds: not tested (one seed).

The best-val checkpoint did better than the last-step model (−19.11% vs −22.77%), so picking by validation helped a bit, but it didn't turn the result positive. The 80k baseline's −6.40% used a rough flat 3.5-point cost and isn't directly comparable; the v2 costs are higher (about 6 to 14 points per round trip depending on date).

9. How to read the final numbers when they arrive

Why train, validation and test are kept separate.

• Train results mostly show how well the agent memorised. On their own they mean almost nothing.
• Validation is better, but we've looked at it to choose settings and checkpoints, so it's a bit optimistic.
• The sealed test is the closest thing to "what would have happened live". That's the number that counts.

Best-val checkpoint vs last-step model. v2 reports both.

• The best-val checkpoint is the official test result.
• The last-step model shows what happens if you just keep training.

If the last-step model is much worse on the test, the run overfit late, as the earlier logs suggested it would. If the two are similar, training was more stable than feared.

Overfitting is when a model fits noise in the training years rather than repeatable behaviour. The warning sign is a growing gap between train and validation: train returns climbing while validation stays flat or falls.

Why buy-and-hold is the benchmark. If an agent can't beat simply holding Nifty, there's no reason to run it. But in a falling year, "beating" buy-and-hold can just mean staying flat a lot. So look at both the absolute return (did it make money?) and the excess.

What would count as a real edge. All of these together, after the date-aware costs:

1. A positive sealed-test return, not just "lost less than Nifty".
2. Consistency across walk-forward folds. The agent should be positive (or at least beat buy-and-hold) in most folds, including up, down and sideways markets. One great fold and three bad ones is not an edge.
3. Profit factor > 1.1 on test. A PF between 1.0 and 1.1 is too thin to survive real-world execution.
4. Acceptable drawdown: for example, no worse than buy-and-hold over the same window, and tolerable for a system risking 0.5% per trade.
5. Enough trades (hundreds) for the win rate and PF to mean something statistically.
6. Robustness: similar results across a few different random seeds.

Nothing is proven. Both the 80k baseline and the v2 2M run lost money on the sealed test, and v2 lost more than buy-and-hold.

10. Limitations and missing data

• Price-only inputs. No order flow, breadth, OI, options, VIX, flows or news. The agent can only find patterns in Nifty's own past prices, which is the hardest place to find an edge.
• Index, not a tradable contract. The agent trains on spot Nifty, but you would trade futures. Futures basis, rollovers, expiry effects and gaps aren't modelled. Overnight gaps are only partly captured: a stop can be hit at the first candle after a gap, but it fills at the stop price, which is optimistic for real gaps.
• Fills at exact stop/target prices plus a fixed 0.5-point slippage. Real stop fills in fast markets can be worse.
• Fractional, risk-based sizing rather than whole lots. Real accounts face lot granularity, especially with lots of 65–75 units.
• Costs are still assumptions. The exchange-charge rate (0.00183%) is applied to all dates because its historical path is unconfirmed. Brokerage assumes a ₹20 discount broker.
• Small dataset for RL. About 4,900 training hours. Millions of steps means hundreds of passes over the same history, which invites memorisation.
• One random seed throughout. RL results can swing a lot from seed to seed.
• The search was short and used the earlier setup. It ran 2 folds × 50k steps with the old flat costs, 29 features and last-step scoring. The best setting under those conditions may not be the best for a 2M run under v2.
• Walk-forward folds are short and separate. Each fold trains on only about 1,150 hours (about 8 months) of its own block, not on all earlier history. At 200k steps, that is about 175 passes per fold.

11. Open issues

What's left from the code review after the v2 fixes:

1. The search overlaps the sealed test window. Search fold 2 (training 2024-06 to 2025-10, validation 2025-10-31 to 2026-04-08) overlaps the full run's test window (2025-07-10 to 2026-10-01). See §7.
2. The GitHub PR code differs from the laptop code. The PR branch is cursor/nifty-rl-trading-results-b2f0, under aiprojects/rl-trading-nifty/ (PRs 13 and 14). It doesn't match the v2 code actually running on the laptop. The PR should be updated so the repo reflects what produced the results.

Fixed in v2:

• the test now scores the best-val checkpoint with its matching vecnorm
• walk-forward uses large_batch
• costs are date-aware (STT, lot size)
• the two dead session features were dropped
• environments run in parallel

12. What's done and next steps

Done:

• ☑ v2 full 2M run finished at 11:07 IST (§8.3).
• ☑ v2 walk-forward 4 × 200k finished at 11:16 IST (§8.3).
• ☑ Results published on the v2 results page.

Suggested next steps (for Satya to decide):

1. Read the results using the checklist in §9. Compare the best-val checkpoint with the last-step model, and look at every walk-forward fold, not just the average.
2. Fight overfitting if the gap is large:
   • fewer steps or early stopping
   • a smaller network or stronger regularisation
   • several random seeds, averaged
3. Re-run the hyperparameter search under v2: date-aware costs, 27 features, best-val scoring. Use folds that don't touch the sealed test window, to close the selection-leakage gap. With 8 environments, the search is now cheap.
4. Better inputs, probably the biggest lever: OI, India VIX, futures basis, breadth and options-derived signals, all strictly causal.
5. Fairer walk-forward: anchored or expanding training windows, and more folds.
6. Sync the GitHub PR with the laptop v2 code.
7. Only after a positive, consistent, cost-robust result: paper-trade live for a few weeks before any real money.

Appendix: file map (laptop, /home/satya/rl-trading-nifty/)

Self-contained page (no external assets). Same content as EXPLAINER.md. Version 2 (final), updated 5 Oct 2026 after the v2 run finished at 11:16 IST. v2 results page · 80k baseline results page.