import pandas as pd,numpy as np,datetime as dt,html,json
L='/workspace/research/buyer-method-audit'; R=lambda f:pd.read_csv(f'{L}/results/{f}')
now=dt.datetime.now(dt.timezone(dt.timedelta(hours=5,minutes=30))).strftime('%Y-%m-%d %H:%M IST')
def tb(df,fl=2): return df.to_html(index=False,float_format=lambda x:f'{x:,.{fl}f}',border=0,na_rep='–')
css=open('/workspace/research/flow-forensics-2026-10/index.html').read().split('<style>')[1].split('</style>')[0]
CH=json.load(open(f'{L}/results/frozen_choices.json'))
F=R('flaw_cost_exit_pooled.csv'); fp=F.pivot(index='ex',columns='scen',values='pnl')
P=R('primary_results.csv'); PT=P[['cand','ex','split','n','mean','median','win','avg_win','avg_loss','payoff','rs_per_trade','rs_total','maxdd_rs','worst_month','worst_month_rs','random_mean','random_pctile','t_p_one','q_bh']]
SP=R('spread_by_tod.csv'); SD=R('spread_by_dte.csv'); SO=R('spread_by_offset.csv')
Y=R('chosen_by_year.csv'); G=R('grid_all_exits_scen.csv'); Gp=G[G.scen=='REAL'].pivot_table(index=['cand','ex'],columns='split',values=['n','mean']).round(2); Gp.columns=[f'{a}_{b}' for a,b in Gp.columns]; Gp=Gp.reset_index()
TOD=R('random_by_tod.csv'); TODp=TOD.pivot(index='hm',columns='ex',values='pnl').reset_index()
WV=R('random_with_vs_counter.csv'); SPX=R('spot_proxy_by_year.csv'); SPX=SPX[SPX.ex.isin(['E1','E4'])]
DI=R('deep_itm_orb_E1_summary.csv'); DI=DI[DI.scen=='REAL']
names={'A':'O30 >150 continuation','B15':'ORB15 with-gap','B30':'ORB30 with-gap','C':'Rule B early-low rebound (CE)','D':'First-hour VWAP pullback','E':'Trend-day filter'}
PT.insert(1,'name',PT.cand.map(names))
te=P[P.split=='test'].set_index('cand')
H=f"""<!DOCTYPE html><html><head><meta charset=utf-8><meta name=viewport content='width=device-width,initial-scale=1'><title>Audit of our option-buyer backtest method — what we were doing wrong, and the re-run</title><style>{css}</style></head><body>
<h1>Are our backtests biased against option buying? Audit, fixes and a clean re-run</h1>
<p class=note>Built {now}. Research only, not advice. Read-only data, no orders. Asked 26-Oct-2026 17:11 IST: “you are not able to find 1 strategy in option buying which is profitable, you are doing something wrong, there are thousands of strategies and you say all of them lose.”</p>
<p class=note><b>Update 26-Oct-2026 (asked 18:06 IST):</b> “trail the profit at 50% after 1:2” re-run → <a href='#trail50'>section 6</a>. It does not change the verdict.</p>
<div class=verdict><b>Short answer.</b><ol>
<li><b>Yes, we got one big thing wrong: costs.</b> Earlier pages (recoveries, lows, call-wall, pre-expiry, last-30, overnight) charged <b>3 pts slippage per side</b> on ATM weekly options. The real Upstox order book (31,096 snapshots, 11 sessions, Oct-2026) shows a <b>median ATM/±1 bid-ask spread of 0.25 pt</b> (90th percentile 0.40). Top-of-book size is about 9–10 lots. A realistic fill is ≈0.5 pt per side. That is <b>about 2.0 pts (≈₹133/lot of 65) per 2-order round trip</b> including all charges, not ≈7. Fixing it improves every result by <b>≈+4.9 pts/trade</b>. The famous “random entries lose −9” falls to about −2.5.</li>
<li><b>The other suspected flaws were checked and are small.</b> Buyer-style exits (structural spot stop, no scale-out, hold to 15:14, 2R/4R targets, ATR-like trail) do <b>not</b> beat the old scale-out on average. Pooled across 804 trades: trail E4 {fp.loc['E4','REAL']:+.2f}, old E0 {fp.loc['E0','REAL']:+.2f}, 2R {fp.loc['E2','REAL']:+.2f}, 4R {fp.loc['E3','REAL']:+.2f}, hold-to-close {fp.loc['E1','REAL']:+.2f} pts. With-trend beats counter-trend by only ≈0.2–0.7 pt. First-hour entries are no cheaper than mid-day. Strike choice (ITM/ATM/OTM) moves results by ±1 pt. No lookahead was found in the old code.</li>
<li><b>Even after the fixes, none of the 6 pre-declared candidates is profitable out of sample.</b> In 2026, 5 of 6 are negative. The trend-day filter is {te.loc['E','mean']:+.2f} pts/trade (n={te.loc['E','n']:.0f}, p={te.loc['E','t_p_one']:.2f}, BH q=1). None survives BH. Most sit at or below random same-time entries with the same exits.</li>
<li><b>Where the edge really is:</b> ORB with-gap plus a structural stop held to 15:14 makes <b>+5.9 (ORB15) / +8.1 (ORB30) Nifty points per trade on spot, 2022–2026, positive in every year</b> (ORB30 p=0.04 before multiple-testing correction). The option buyer keeps only ≈25–50% of that move. Theta and the intraday IV drift eat the rest, then about 2 pts of cost, so the trades end at −1 to −5 pts. <b>The problem is not that we test the wrong patterns. Intraday spot edges of 5–10 points are smaller than what a long option bleeds over the hours needed to collect them.</b></li>
<li><b>A lead, not a strategy (found after looking at 2026, so it is post-hoc):</b> the put side of ORB with-gap (gap-down day, breaks the opening range down, 2–6 strikes ITM, structural stop, hold to 15:14) made +5.6 to +7.4 pts/trade in 2026 (n≈50) but about 0 in 2024–25 (−3.5 to +2.6). It goes on a forward-watch list, not a trade list.</li></ol></div>
<h2 id=wrong>1. What we were doing wrong (measured)</h2>
<table><tr><th>#</th><th>Suspected flaw</th><th>Evidence in our code</th><th>Verdict</th><th>Measured impact</th></tr>
<tr><td>1</td><td><b>Slippage 3 pts/side</b></td><td><code>SLIP=3.0</code> in recov/tradelib.py, lows/part2_options.py, callwall/cwlib.py, pre-expiry/backtest_pre.py, last30/options_pnl.py, overnight/backtest.py. (nifty-setups-backtest used 0.3–0.5, which is fine.)</td><td><b>CONFIRMED, the main flaw</b></td><td>OLD→REAL: E0 {fp.loc['E0','OLD']:+.2f}→{fp.loc['E0','REAL']:+.2f}; E1 {fp.loc['E1','OLD']:+.2f}→{fp.loc['E1','REAL']:+.2f} pts/trade, i.e. <b>+4.9 pts/trade</b> (≈₹320/lot). Stress at 1 pt/side costs another −1.0.</td></tr>
<tr><td>1b</td><td>STT</td><td>Old code used 0.10% on all sells. It has been 0.15% since 1-Apr-2026 (Finance Act 2026, NSE circular).</td><td>Confirmed, tiny</td><td>≈−0.07 pt/trade in 2026 (it made old 2026 results slightly too good)</td></tr>
<tr><td>2</td><td>Premium stop −20 + scale-out cut the right tail</td><td>E0 in tradelib/cwlib: stop −20, 40% at +20, 30% at +40, 60-min cap</td><td><b>Not confirmed</b></td><td>Pooled REAL: E0 {fp.loc['E0','REAL']:+.2f} vs structural stop + hold to 15:14 {fp.loc['E1','REAL']:+.2f}, 4R {fp.loc['E3','REAL']:+.2f}, 2R {fp.loc['E2','REAL']:+.2f}, trail {fp.loc['E4','REAL']:+.2f}. Longer holds raise the payoff ratio (1.4–2.7) but win rates drop to 20–35% and theta is paid for longer. Only the trail is better, by ≈+0.4.</td></tr>
<tr><td>3</td><td>Entries were countertrend or late; first hour ignored</td><td>recov and lows were rebound fades. callwall was a mid-day cross.</td><td>Partly true, small</td><td>Random entries: with-trend {WV.with_trend.mean():+.2f} vs counter {WV.counter.mean():+.2f} (first hour {WV[WV.hm.isin(['09:30','09:45','10:15'])].with_trend.mean():+.2f} vs {WV[WV.hm.isin(['09:30','09:45','10:15'])].counter.mean():+.2f}). First-hour random entries {TOD[TOD.hm.isin(['09:20','09:30','09:45'])].pnl.mean():+.2f} vs later {TOD[~TOD.hm.isin(['09:20','09:30','09:45'])].pnl.mean():+.2f}, so no free first-hour edge. All 6 candidates here are with-trend and first-hour.</td></tr>
<tr><td>4a</td><td>Lookahead / same-bar entry</td><td>All old option sims fill at the <i>next</i> minute's open. Stops fill at min(open, stop).</td><td>Not found</td><td>0. One small optimism: recov 'spotstop' exits at the same bar's close.</td></tr>
<tr><td>4b</td><td>Wrong strike</td><td>ATM only in recov/callwall</td><td>Minor</td><td>ITM1/ITM2/OTM1 vs ATM: −0.1 to +0.6 pt on average. 0DTE is worse (−0.5 to −1.1).</td></tr>
<tr><td>4c</td><td>Spot close vs executable option price</td><td>Old trades already used real 1-min option bars</td><td>Not a flaw</td><td>–</td></tr>
<tr><td>4d</td><td>Too many variants / BH kills everything</td><td>callwall picked 1 of 440 combos per side. Last30 had dozens of variants.</td><td>Real issue (it cuts both ways)</td><td>Here the family is fixed at 6 tests, declared before 2026 was opened. Nothing is positive enough for BH to even matter: the best raw p is {te.t_p_one.min():.2f}.</td></tr>
<tr><td>4e</td><td>60-min hold cap</td><td>recov/callwall used 60 min (runner 240)</td><td>Tested via E1–E4 (hold to 15:14)</td><td>Holding longer did not help on average (see #2)</td></tr>
<tr><td>4f</td><td>Sessions excluded / missing data</td><td>Spot 15:15–15:29 frozen since 2026-08-03 is treated as missing. Options are real to 15:40.</td><td>Handled</td><td>The time exit is at the 15:15 option open, so the frozen window is never used.</td></tr></table>
<img src='charts/cost_vs_exit.png' style='max-width:100%'>
<h2 id=cost>2. Corrected cost per trade</h2>
<p>Bid-ask from the Upstox ATM-feed snapshots (strikes ATM±2, CE+PE, 08–26 Oct 2026). Spread in option points:</p>{tb(SP,3)}{tb(SD,3)}{tb(SO,3)}
<p><b>Charges</b> (2-order round trip, ATM premium ≈145, lot 65): brokerage ₹40, STT 0.15% of sell premium ≈₹14, NSE txn 0.03503% ≈₹7, GST 18% on brokerage+txn ≈₹8.5, SEBI and stamp <₹1. Total ≈₹68 = <b>1.04 pts</b>. Scale-out exits use 3–4 orders: 1.43–1.96 pts.
<b>Slippage</b>: 0.5 pt/side used as REAL, against a measured half-spread of ≈0.13. That leaves room for a tick and for faster minutes. Stress case 1.0 pt/side.
<b>Total REAL ≈ 2.0 pts/trade (₹133/lot)</b> versus the old ≈7.0–8.0. Caveat: the spreads come from calm-to-normal Oct-2026 sessions. Gap opens and news minutes can be wider.</p>
<h2 id=rerun>3. Re-run: 6 pre-declared candidates (<a href='PREREG.md'>PREREG.md</a>, choices frozen in <a href='results/frozen_choices.json'>frozen_choices.json</a> at {CH['_fixed_at_ist']} before any 2026 number)</h2>
<p class=note>Instrument: ATM, nearest weekly with DTE≥1, CE for long / PE for short. Entry at the next minute's option open after a spot-bar-close signal. Exits are decided on spot and executed at the next option open. Time exit at the 15:15 option open. R = structural stop distance clipped to 15–70 spot pts. Exit set E0 legacy / E1 structural stop + time / E2 2R / E3 4R / E4 BE at 1R then trail 1R. The exit is chosen per candidate on TRAIN (Oct-2024..Dec-2025), then tested once on 2026 (to 23-Oct). REAL cost. Random = same-time, same-direction, same-R entries on random days of the same period with the same exit; percentile = rank of the candidate mean among 2,000 random same-size sets. ₹ per lot uses lot 75 in 2024–25 and 65 in 2026.</p>
{tb(PT)}<img src='charts/candidates.png' style='max-width:100%'>
<h3>Per year (chosen exit, REAL cost)</h3>{tb(Y,1)}<p><a href='results/chosen_by_month.csv'>per-month CSV</a></p>
<h3>Secondary: every exit, REAL cost (exploratory, not in the family)</h3>{tb(Gp)}
<p>The trend-day filter (E) with the trail E4 is +18.4 pts in 2026, but −9.8 in train. It was the worst-ranked choice on train, so taking it now would be fitting to 2026. Not claimed.</p>
<h2 id=edge>4. Why buyers still lose: the spot edge exists, the option eats it</h2>
<p>Spot-only proxy (index points, no costs, same signal and exit; 2022–23 has no futures volume, so VWAP-based D/E start in Oct-2024):</p>{tb(SPX,1)}
<p>ORB with-gap held to 15:14 (E1): spot points vs what the option buyer keeps after REAL cost, by strikes ITM:</p>{tb(DI)}<img src='charts/spot_vs_option.png' style='max-width:100%'>
<p>By direction (all periods): on short signals spot gains +13.3 pts, the ATM put gains +3.2 before costs, and the 6-ITM put +6.6. On long signals spot loses −3.9 and calls lose −6. Over 3–5 hours a long weekly option gives back roughly 3–6 pts to theta and the falling intraday IV. That is the same size as the spot edge.</p>
<h3>Random ATM buys by entry time (REAL cost)</h3>{tb(TODp)}<img src='charts/random_tod.png' style='max-width:100%'>
"""+open(f'{L}/section_trail50.html').read()+f"""<h2 id=gaps>5. Data, limits, gaps</h2><ul>
<li>Options: Upstox expired-instrument 1-min chains, panels from 21-Oct-2024 (the 1–18 Oct 2024 days are not in the panel cache) to 23-Oct-2026. 90 of 12,150 simulated rows had no option bar at the entry minute and were skipped, never filled. Some days have thin strike coverage (e.g. 2024-12-20).</li>
<li>No option volume in the panel. VWAP = spot typical price weighted by Nifty futures volume, available from Oct-2024 only.</li>
<li>Bid-ask comes from 11 sessions of Oct-2026 only. 2024–25 spreads are assumed similar.</li>
<li>Sample sizes: O30 has 17 train and 9 test trades, which is too few for any conclusion. Others have 28–133.</li>
<li>Exits trigger on 1-min spot bars and fill at the next option open. Real intra-minute stop fills could be better or worse.</li>
<li>Option selling and futures are out of scope.</li></ul>
<p>Files: <a href='results/'>results</a> · <a href='scripts/'>scripts</a> · <a href='charts/'>charts</a> · <a href='PREREG.md'>pre-registration</a>. Related: <a href='/rl-trading/research/intraday-recoveries/'>intraday recoveries</a>, <a href='/rl-trading/research/flow-forensics-2026-10/'>flow forensics</a>, <a href='/rl-trading/rules/opening-30m-move/'>opening-30m rule</a>.</p>
<script src="/assets/helpbot/helpbot.js" defer></script></body></html>"""
open(f'{L}/index.html','w').write(H); print('written',len(H))