What makes these problems challenging is that they combine several difficult ingredients: uncertainty, dynamics, interdependence, and optimization over time. The good news is that there exists a unified framework that allows us to model—and eventually solve—these problems, regardless of the specific application.

The core principle is simple but non-negotiable:

👉 Model first, then solve it.

Skipping modeling is like trying to build a skyscraper without a blueprint. Algorithms—no matter how powerful—are useless unless they are grounded in a clear description of states, decisions, uncertainty, and objectives.

The Anatomy of a Sequential Decision Problem

At the heart of this framework are six key building blocks.

1. State

The state captures everything we need to know at time t to make a good decision going forward. It is a compressed representation of history: instead of remembering the entire sequence of past events, we summarize it in a state variable.

• Physical state: Tangible variables that describe the environment. Example: current inventory level.

• Information state: Observable signals such as prices, costs, or competitor actions.

• Belief state: Probabilistic estimates of hidden or uncertain factors. For example, if we do not directly observe demand, our belief about demand follows a distribution updated over time.

💡 Key Point: A well-defined state includes all relevant information but excludes irrelevant history. Latent variables—hidden drivers of uncertainty—do influence outcomes but are not part of the state if they cannot be observed or inferred.

2. Decisions and Policies

A decision x_t is the action taken at time t.

A policy π is a rule that maps states to actions:

• In inventory, the decision is “how much to order.”

• In pricing, it is “what price to set.”

• In ride-sharing, it might be “where to send a driver.”

Policies can be deterministic (“always reorder to level ss”) or stochastic (“randomize prices according to a distribution”).

3. Exogenous Information

Uncertainty enters after a decision is made. This is the exogenous information:

• Customer demand revealed tomorrow.

• Competitor’s pricing next week.

• A machine breaking down unexpectedly.

The exogenous process evolves according to some probability law, which may be partially known (model-based) or entirely unknown (model-free).

4. Transition Function

The transition function glues states, decisions, and uncertainty together:

It describes how today’s choice and tomorrow’s randomness determine the next state.

• In inventory:

• In pricing: tomorrow’s market share depends on today’s price, competitor’s price, and consumer preferences.

This function is the “physics” of the system: it dictates how reality unfolds.

5. Contribution (Reward)

Each time step brings an immediate contribution:

This is often a profit (revenues minus costs), but it could also be utility, service quality, or negative costs.

Example (inventory):

6. Objective Function

Finally, everything ties together in the objective: maximize cumulative contributions.

This expectation accounts for uncertainty, since outcomes depend on random exogenous information.

Example 1: Inventory Management

Imagine managing a single product:

• State: inventory level, costs, and prices.

• Decision: how much to order today.

• Exogenous information: demand tomorrow.

• Transition: tomorrow’s stock = today’s stock + order – demand.

• Contribution: profit (sales revenue minus costs).

• Objective: maximize profit over the season.

This simple setup is a canonical example used in both operations research and reinforcement learning.

Modeling the Solution Process

Before solving, we must model carefully. The structured process is:

1. Narrative description: Write the story in plain words.

2. Identify core elements: List states, decisions, exogenous inputs, transitions, and contributions.

3. Model in the universal framework: Formalize in words + equations.

4. Uncertainty model: Specify distributions (e.g., demand ~ Poisson, prices ~ Gaussian).

5. Design the policy: Choose policy family—lookup, heuristic, optimization, or RL.

6. Evaluate the policy: Test with simulation or historical data.

Example 2: Marketplace Pricing

• Narrative: A seller sets item prices over time while competing with others.

• Core elements:

  • State: current prices, inventory, competitor’s last price.

  • Decision: today’s price.

  • Exogenous: tomorrow’s demand, competitor’s response.

  • Contribution: daily profit.

  • Objective: maximize cumulative profit.

• Uncertainty model: Demand could follow a Poisson distribution; competitor’s moves could be modeled as a stochastic process.

The challenge here is learning a pricing policy that balances exploration (trying new prices) and exploitation (maximizing profit).

Model-Based vs Model-Free Approaches

Model-Based

If we know the transition law or transition probabilities, we can write the problem as a Markov Decision Process (MDP).

• Methods:

  • Backward recursion (finite horizon).

  • Value iteration (computing optimal value functions).

  • Policy iteration (alternating between policy evaluation and improvement).

  • Linear programming formulations.

Here, the transition probabilities combine both the deterministic dynamics and exogenous randomness.

Model-Free

When we don’t know the transition probabilities, we can learn by experience: reinforcement learning (RL).

• Tabular RL: For small state spaces, we maintain explicit tables of state–action values (Q-learning, SARSA).

• Function approximation: For large state spaces, we approximate value functions or policies with neural networks (Deep RL).

💡 Subtle point: If we don’t know just the transition probabilities but we do know the distribution of exogenous variables, we often still classify the approach as “model-free,” since computing probabilities directly is infeasible.

The Curses of Sequential Decisions

Sequential decision problems face four major “curses”:

1. State space curse: Too many possible states (e.g., multi-product inventory).

2. Decision space curse: Too many possible actions.

3. Outcome space curse: Too many random outcomes of uncertainty.

4. Modeling curse: Transition probabilities are extremely difficult to calculate.

👉 Key insight: It’s usually easier to model the distribution of exogenous information than the full transition probabilities.

Designing Policies

Policies are the heart of solving sequential decision problems.

Policy Function Approximations (PFAs)

• Direct mappings from states to actions.

• Implemented via lookup tables, parametric functions, or nonparametric models.

• Work best with small action spaces or when good heuristics exist.

• Example: the classic order-up-to policy in inventory (always replenish up to level ss).

Limitations: PFAs struggle with continuous/high-dimensional decisions and long horizons.

Cost Function Approximations (CFAs)

• Use parameterized optimization models.

• Modify the contribution function and constraints to make the problem tractable.

• Ideal for large-scale problems such as airline scheduling, where exact optimization is impossible.

Lookahead Approximations

Policies can also be designed by looking ahead into the future and approximating the consequences of today’s decisions.

1. Value Function Approximations (VFAs):
   Approximate the long-term value of being in a state. Based on Bellman’s equation:

   VFAs are at the core of dynamic programming and reinforcement learning.

2. Direct Lookahead Approximations (DLAs):
   Directly approximate the downstream value:

   • Limit the horizon.

   • Use deterministic approximations.

   • Sample random outcomes (Monte Carlo).

   • Use simple rollout policies.

   ⚖️ Rule of thumb:

   • DLAs work well when uncertainty is limited.

   • VFAs are more effective under high uncertainty, as they better capture future risk.

Closing Thoughts

Sequential decision problems—whether in operations, economics, or AI—are all variations of the same blueprint. The framework unifies them:

• State: compressed history.

• Decision: action chosen.

• Exogenous info: randomness revealed.

• Transition: system evolution.

• Contribution: immediate payoff.

• Objective: long-term optimization.

Designing policies is the art; solving them is the science. But neither matters unless we model first. Once the problem is structured, the right algorithm naturally follows.

👉 In short: Model first, then solve it. Always.

• The HJB equation is the backbone of optimal decision-making under uncertainty.

• In finance, it unifies problems like portfolio choice, execution/hedging, and inventory/risk control.

• Two canonical problems we’ll walk through:

  1. Merton portfolio (closed form).

  2. Optimal execution (Almgren–Chriss), both closed form and numerical.

• Complete, runnable Python included.

Welcome to the Series

Welcome to our new series on stochastic control and everything connected to stochastic operations in finance.

Our goal is simple: to build, step by step, a practical toolkit that spans from the Hamilton–Jacobi–Bellman equation all the way to modern applications in trading, risk management, and market making.

We’ll mix theory, worked examples, and fully runnable Python. Some topics will feel advanced, others more hands-on — let us know if the pace is too fast or too slow, and we’ll tune the level.

We decided to start here, with the HJB equation, because it quietly underpins so many decisions a quant faces every day. From portfolio choice to execution algorithms, it’s the hidden blueprint.

Why HJB belongs on the trading desk

At first glance, the Hamilton–Jacobi–Bellman (HJB) equation sounds like it belongs in a physics textbook, not a trading playbook. But in reality, it is one of the most powerful tools for making decisions in uncertain environments.

The core question is very intuitive:

Given what I know right now, and given that the future is uncertain, how should I act to maximize my objective?

In finance, that objective could mean:

• Maximizing long-run portfolio growth.

• Minimizing costs while liquidating a large position.

• Hedging in a way that balances cost against risk.

The HJB translates these goals into mathematics. You describe how the market evolves, define your objective, and the HJB tells you how the value of being in a given state evolves. Solve it, and you get the optimal policy.

The HJB in plain English

Think of your wealth, portfolio, or inventory as a dynamic system that moves randomly over time. The HJB says:

• Write down the expected payoff from following the best strategy.

• Ask how this payoff changes as time moves forward.

• Balance immediate rewards against the future benefit of being in a better (or worse) position.

The result is a partial differential equation (PDE). Solving it is like having a GPS for decision-making under uncertainty.

Example I — The Merton Portfolio Problem

Setup:

• One safe asset grows at rate r.

• One risky asset (a stock) grows with drift μ and volatility σ.

• At each moment, you choose the fraction π of wealth in the risky asset.

The optimal fraction is:

where γ is risk aversion.

Interpretation:

• Higher expected excess return → invest more.

• Higher volatility → invest less.

• More risk-averse → invest less.

This is the classic Merton rule: a clean balance of reward versus risk.

Example II — Optimal Execution (Almgren–Chriss model)

Suppose you need to sell a large block of shares over a trading horizon [0, T].

Trade-off:

• Sell too fast → high market impact costs.

• Sell too slow → risk that prices move against you.

HJB setup:

• State: remaining inventory q.

• Control: sell rate u(t).

• Costs:

  • Temporary impact ∝ u².

  • Risk penalty ∝ q².

The optimal strategy is:

where κ depends on volatility, risk aversion, and impact cost.

Interpretation:

• Sell faster as the deadline approaches.

• Higher volatility or higher risk aversion → trade more aggressively.

Why numerical solvers matter

In toy problems we got neat closed forms. In reality:

• Volatility depends on prices.

• Impact depends on order book depth.

• Portfolios include multiple correlated assets.

No closed form! But HJB still applies — we just solve it numerically.

Code — Merton Portfolio

import numpy as np

r = 0.02
mu = 0.08
sigma = 0.20
gamma = 5.0

pi_star = (mu - r) / (gamma * sigma**2)
print(f"Merton optimal risky weight: {pi_star:.3f}")

Code — Tiny Numerical HJB Solver

import numpy as np

eta = 1.0
sigma = 0.3
gamma = 5.0
T = 1.0

q_max = 1.0
M = 200
N = 1000

q = np.linspace(-q_max, q_max, 2*M+1)
V = np.zeros_like(q)
V_new = np.zeros_like(q)
dq = q[1] - q[0]
dt = T / N

def grad(V):
    g = np.zeros_like(V)
    g[1:-1] = (V[2:] - V[:-2]) / (2*dq)
    return g

for _ in range(N):
    g = grad(V)
    rhs = (g**2)/(4*eta) - 0.5*gamma*sigma**2 * q**2
    V_new[:] = V - dt*rhs
    V, V_new = V_new, V

u = grad(V) / (2*eta)
print("Sample control at q=0.5:", np.interp(0.5, q, u))

Key Takeaways

• HJB is the blueprint for stochastic optimization.

• In portfolio choice: yields the Merton allocation.

• In execution: gives an optimal liquidation curve.

• In reality: approximate numerically or with RL, but the principle is the same.

Further Reading

• Merton (1969): Lifetime Portfolio Selection under Uncertainty.

• Almgren & Chriss (2000): Optimal Execution of Portfolio Transactions.

• Texts on stochastic control, dynamic programming, or reinforcement learning.

Wrapping Up — and What’s Next

Today we saw how HJB connects classic problems like portfolio choice and optimal execution, bridging closed-form solutions with tiny numerical solvers. But this is just the start of the journey.

In the coming posts of this series, we’ll dive into:

• Optimal execution with alpha signals — how to blend predictive drift into liquidation schedules.

• Mean–variance hedging in incomplete markets — handling risk when perfect replication is impossible.

• Risk-sensitive control with exponential utility — what changes when robustness is built into preferences.

• HJB and Reinforcement Learning — the deep connection between PDEs and modern RL algorithms.

• Continuous-time market making (Avellaneda–Stoikov) — HJB at the heart of quoting and inventory management.

Stay tuned: we’ll keep weaving theory, intuition, and hands-on examples so each idea connects directly back to real trading and quantitative finance practice.

Modern data science is entranced by prediction. From deep learning to ensemble models, from transformers to sequential fancy models, enormous resources are devoted to anticipating the next value in a time series, whether it's price, return, volatility, or any other financial indicator. Forecasting has become the dominant framework through which we interpret the future.

But this fixation is misguided. Prediction is not the core problem. The way we formulate the problem, how we define uncertainty, structure the data, and align our models with market dynamics determines more than any architecture or hyperparameter ever will.

We argue that in complex, adaptive systems such as financial markets, especially those that are highly liquid and efficient, simpler models that are theoretically coherent often outperform their complex counterparts. The differentiator is not predictive firepower. It's epistemic discipline.

What follows is a framework rooted in three foundational principles: and an empirical case study demonstrating their validity.

In a world addicted to prediction, true advantage lies in adaptation not foresight.

We live in a culture that worships control. Strategic roadmaps, quarterly forecasts, probabilistic dashboards, all constructed to manufacture an illusion of certainty. The prevailing belief is that if we can see what’s coming, we can prepare for it.

But what if this paradigm is not only flawed, but fundamentally misaligned with how complex systems, markets, organizations, even ecosystems, actually function? What if survival, not prediction, is the real metric of success?

Our work is grounded in a radically different premise: that resilience outperforms foresight. We don’t build models to forecast the future, we build them to withstand it.

This begins with three principles most systems, and most strategists, fail to internalize.

1. Embrace Uncertainty: It’s Not a Bug, It’s the Terrain

Forecasting is often a sophisticated form of self-deception. Anyone who has attempted to predict returns or volatility in a complex financial system knows the limits of such endeavors. Even when predictive signals appear statistically valid, their operational utility tends to be negligible or transient. More often than not, those signals emerge from flawed backtests, erroneous assumptions, or overfit code.

Hundreds of fintech startups have attempted to build machine learning pipelines using hundreds of features to predict volatility, Sharpe ratios, or other performance metrics. Yet very few, if any, have delivered tangible improvements in live trading strategies. Most are marketing operations wrapped in technical jargon.

The obsession with prediction obscures a fundamental truth: uncertainty is not a nuisance variable. It is information.

There are two kinds of uncertainty we care about:

• Epistemic uncertainty arises from incomplete knowledge. It is reducible with more data, better models, or sharper measurement. Techniques such as Bayesian inference or model ensembling can help quantify it.

• Aleatoric uncertainty stems from the inherent randomness of the world. It is irreducible. No amount of data or computation can eliminate it. Acknowledging its presence is not weakness; it is rigor.

Designing for uncertainty requires letting go of deterministic thinking. It requires models that absorb volatility rather than collapse under it. In environments dominated by complexity and feedback loops, adaptability beats accuracy—every single time.

2. Method Over Outcome: Truth Is a Process, Not a Performance

The scientific mindset does not chase outcomes; it tests mechanisms. A model that performs well but cannot be explained, falsified, or replicated is not a model, it’s a liability.

Overfitting is not just a technical problem; it's an epistemological failure. It arises when we prioritize short-term performance over long-term integrity, when we value correlation over causality, and when we mistake noise for signal.

We do not trust intuition. We do not rely on heuristics. If a process cannot be formalized, measured, and subjected to stress, it cannot be trusted. Robust strategies are born from falsifiability, not faith.

Rigor is not optional. It is the foundation. And in a world where everyone is optimizing for metrics, the differentiator is not the model—but the method behind it.

3. Iteration Is the Only Proof: Learn from the Friction

Every model is a hypothesis. Every strategy is a bet. And every deployment is a confrontation with reality.

In complex systems, failure is not an anomaly, it is feedback. We do not iterate to improve performance metrics; we iterate to break assumptions. Assumptions are the most dangerous form of technical debt, and they compound invisibly until they collapse under real-world stress.

We do not seek confirmation. We seek contradiction. That’s how we grow.

Iteration, properly understood, is not tweaking parameters. It’s epistemic combat. It's how we refine our hypotheses in the face of adversarial conditions.

Burn the Map, Build the Compass

If you're building anything that must outlast the next investment cycle, you need more than predictions. You need principles! You need tools that help you navigate, not forecast.

Throw out the crystal ball. Burn the map.

And begin walking into uncertainty with a compass grounded in epistemic humility, statistical rigor, and methodological resilience.

You're not here to control the storm.

You're here to become stormproof.

Final Note: Simplicity Is an Underrated Superpower

Let me close with a caution: there is a pervasive myth in the machine learning community that more complexity equals better results. That access to state-of-the-art frameworks: Keras, PyTorch, TensorFlow, translates into strategic edge.

It does not.

Complex models often offer the illusion of competence. But they are only as good as the problem formulation behind them. Without a clear objective function, clean data structure, and sound causal assumptions, no architecture, no matter how deep or wide, an deliver sustained results.

Simple models, designed with precision, often outperform deep learning systems that are poorly specified.

The power lies not in the tool, but in the clarity of the problem.

Empirical Illustration: Clarity Over Complexity

These principles are not theoretical abstractions, they manifest clearly in empirical practice. A recent thesis on the predictive modeling of Nasdaq-100 futures (NQ) provides a compelling demonstration. The study systematically benchmarked a wide range of models, from classical econometric systems to advanced probabilistic sequential deep learning architectures, on their ability to forecast both returns and volatility across distinct intraday market sessions [ICT’s Killzone].

Findings That Challenge Prevailing Assumptions

• The Kalman Filter, a state-space model rooted in probabilistic inference, achieved the best overall performance in return prediction.

• The Score-Driven model, designed for time-varying volatility and capable of handling excess kurtosis and non-Gaussianity, outperformed all others in volatility forecasting.

• Notably, both models operated on univariate time series inputs, yet surpassed deep learning architectures such as Transformers trained on expansive multivariate datasets.

The reason? These models are constructed with epistemic discipline. They do not ignore uncertainty—they explicitly model it. Both the Kalman and Score-Driven frameworks embed epistemic and aleatoric uncertainty within their structure, treating uncertainty not as a nuisance, but as core information. This is the defining advantage: a principled formulation that respects the data-generating process, rather than a brute-force search for correlation.

Contextual Performance and Market Structure

The models' success is further amplified by the structural properties of the market itself. Nasdaq-100 futures, as highly liquid and information-efficient instruments, embody the conditions under which the Efficient Market Hypothesis (EMH) is most valid. In such contexts, the marginal utility of complex feature engineering and high-capacity models quickly decays. Simpler models, aligned with theoretical and structural priors, prove more robust and adaptive.

Moreover, the adaptive characteristics of models like N4SID-Kalman and Score-Driven resonate strongly with the Adaptive Market Hypothesis (AMH). These models update parameters dynamically in response to shifting regimes, volatility clustering, or abrupt structural changes—without requiring complete retraining or loss of interpretability. They are not just predictive—they are structurally resilient.

Deep Learning: Not to Be Discarded, but to Be Refined

It would be a mistake to interpret this as a dismissal of deep learning. In fact, the thesis demonstrates a thoughtful and commendable implementation: Monte Carlo Dropout was employed to quantify the uncertainty surrounding deep learning predictions. This allowed not only for point estimates, but for distributional inference, moving neural networks beyond deterministic forecasts toward a more Bayesian, uncertainty-aware framework.

This is crucial. The problem is not deep learning per se, but the lack of principled formulation. Too often, neural networks are deployed as “feature soups,” where hundreds of variables are thrown into an architecture without a clear hypothesis, structure, or understanding of the target dynamics. The result is opacity and overfitting.

In this study, by contrast, deep models were designed with epistemic intent, and the Transformer architecture, while not outperforming simpler models, excelled during high-volatility, high-kurtosis sessions, demonstrating its ability to capture long-range dependencies in structurally turbulent regimes.

In summary, this case study offers empirical affirmation of this article’s central claim: it is not predictive power that matters, but the methodological intelligence with which a model is constructed. When uncertainty is embraced as signal—not noise—and when models are formulated to reflect market structure, simplicity becomes a strategic advantage.

—GMAE452