Timing of Back-Test#
We show the runtime of a typical single-period optimization back-test.
This is similar to what was show in figure 7.8 of the paper.
Many elements matter in determining how fast a back-test can be run; here we present a few (size of risk model, choice of numerical solver and CVXPY flags, …) but many more are relevant and understanding of those comes only with (deep) expertise in optimization software (and computer systems).
One interesting feature of Cvxportfolio is that it enables for automatic caching of some expensive numerical procedures; one of them is estimation of large covariance matrices. Here we show the execution time difference when running the same back-test twice. The first time covariance matrices are estimated and saved on disk, the second time they are loaded. This especially matters when doing hyper-parameter optimization (the expensive calculation is only done once).
Finally, we show that cvxportfolio.result.BacktestResult does a good
job accounting for and reporting the time spent doing a back-test and in its
various components. You can expect it will do even more granular reporting in
future releases.
The results shown below are obtained on a Linux workstation with the latest versions, at the time of writing, of most libraries.
Note
To reproduce what is shown here you should make sure that the first
time this script is run there are no covariance matrices already saved
for the historical market data used here. If you run it from scratch, that
is OK, but if you re-run this script it will pick up the covariance
matrices already estimated. There is currently (Cvxportfolio 1.3) no
easy way to remove caches other than manually deleting files in
~/cvxportfolio_data, which you can always safely do.
import time
import matplotlib.pyplot as plt
import pandas as pd
import cvxportfolio as cvx
# same choice as in the paper
from .universes import SP500 as UNIVERSE
# changing these may have some effect on the solver time, but small
GAMMA_RISK = 1.
GAMMA_TRADE = 1.
GAMMA_HOLD = 1.
# the solve time grows (approximately) linearly with this. 15 is the same
# number we had in the paper examples
NUM_RISK_FACTORS = 15
# if you change this to 2 (quadratic model) the resulting program is a QP
# and can be solved faster
TCOST_EXPONENT = 1.5
# you can add any constraint or objective
# term to see how it affects execution time
policy = cvx.SinglePeriodOptimization(
objective = cvx.ReturnsForecast()
- GAMMA_RISK * cvx.FactorModelCovariance(
num_factors=NUM_RISK_FACTORS)
- GAMMA_TRADE * cvx.StocksTransactionCost(exponent=TCOST_EXPONENT)
- GAMMA_HOLD * cvx.StocksHoldingCost(),
constraints = [
cvx.LeverageLimit(3),
],
# You can select any CVXPY-interfaced solver here to see how it affects
# execution time of your particular program. Different solvers apply
# different roundings and other numerical heuristics; their solutions
# may also have (small) differences in other back-test statistics, such
# as Sharpe Ratio. This solver is the default one for this
# type of programs, as of CVXPY 1.5.0. Other freely available solvers
# that work well for this type of programs are for example CVXOPT, ECOS
# and SCS. There are numerous commercial ones as well. See the CVXPY
# documentation for the full list
# https://www.cvxpy.org/tutorial/solvers/index.html
solver='CLARABEL',
# this is a CVXPY compilation flag, it is recommended for large
# optimization programs (like this one) but not for small ones
ignore_dpp=True,
# you can add any other cvxpy.Problem.solve option
# here, see https://www.cvxpy.org/tutorial/advanced/index.html
)
# this downloads data for all the sp500
simulator = cvx.StockMarketSimulator(UNIVERSE)
# we repeat two times to see the difference due to estimation and saving
# of covariance matrices (the first run), and loading them from disk the
# second time
figures = {}
for run in ['first', 'second']:
# execution and timing, 5 years backtest
s = time.time()
result = simulator.backtest(
policy,
start_time=pd.Timestamp.today() - pd.Timedelta(f'{365.24*5}d'))
print('\n\n' + run.upper() + ' RUN')
print('BACK-TEST TOOK:', time.time() - s)
print(
'SIMULATOR + POLICY TIMES:',
result.simulator_times.sum() + result.policy_times.sum())
print(
'AVERAGE TIME PER ITERATION:',
result.simulator_times.mean() + result.policy_times.mean())
print('RESULT:')
print(result)
# plot; this method was introduced in Cvxportfolio 1.3.0
figures[run] = result.times_plot()
This is the output printed to screen when executing this script. You can see
the difference in timing between the two runs (all other statistics are
identical) because in the first run covariance matrices are estimated and
saved on disk, and the second time they are loaded. You can also see that
the time accounting done by cvxportfolio.simulator.MarketSimulator
is very accurate, and coincides (with a tiny difference) with what is reported
by Python’s time here in the script.
Updating data.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
FIRST RUN
BACK-TEST TOOK: 260.66519236564636
SIMULATOR + POLICY TIMES: 260.6166265010834
AVERAGE TIME PER ITERATION: 0.20716742965109966
RESULT:
#################################################################
Universe size 501
Initial timestamp 2019-05-13 13:30:00+00:00
Final timestamp 2024-05-10 13:30:00+00:00
Number of periods 1259
Initial value (USDOLLAR) 1.000e+06
Final value (USDOLLAR) 4.112e+06
Profit (USDOLLAR) 3.112e+06
Avg. return (annualized) 59.4%
Volatility (annualized) 77.9%
Avg. excess return (annualized) 57.4%
Avg. active return (annualized) 57.4%
Excess volatility (annualized) 77.9%
Active volatility (annualized) 77.9%
Avg. growth rate (annualized) 28.3%
Avg. excess growth rate (annualized) 26.3%
Avg. active growth rate (annualized) 26.3%
Avg. StocksTransactionCost 0bp
Max. StocksTransactionCost 9bp
Avg. StocksHoldingCost 1bp
Max. StocksHoldingCost 5bp
Sharpe ratio 0.74
Information ratio 0.74
Avg. drawdown -48.2%
Min. drawdown -88.3%
Avg. leverage 299.9%
Max. leverage 399.6%
Avg. turnover 3.9%
Max. turnover 150.0%
Avg. policy time 0.150s
Avg. simulator time 0.057s
Of which: market data 0.018s
Of which: result 0.015s
Total time 260.617s
#################################################################
SECOND RUN
BACK-TEST TOOK: 149.64058017730713
SIMULATOR + POLICY TIMES: 149.59178948402405
AVERAGE TIME PER ITERATION: 0.11891239227664868
RESULT:
#################################################################
Universe size 501
Initial timestamp 2019-05-13 13:30:00+00:00
Final timestamp 2024-05-10 13:30:00+00:00
Number of periods 1259
Initial value (USDOLLAR) 1.000e+06
Final value (USDOLLAR) 4.112e+06
Profit (USDOLLAR) 3.112e+06
Avg. return (annualized) 59.4%
Volatility (annualized) 77.9%
Avg. excess return (annualized) 57.4%
Avg. active return (annualized) 57.4%
Excess volatility (annualized) 77.9%
Active volatility (annualized) 77.9%
Avg. growth rate (annualized) 28.3%
Avg. excess growth rate (annualized) 26.3%
Avg. active growth rate (annualized) 26.3%
Avg. StocksTransactionCost 0bp
Max. StocksTransactionCost 9bp
Avg. StocksHoldingCost 1bp
Max. StocksHoldingCost 5bp
Sharpe ratio 0.74
Information ratio 0.74
Avg. drawdown -48.2%
Min. drawdown -88.3%
Avg. leverage 299.9%
Max. leverage 399.6%
Avg. turnover 3.9%
Max. turnover 150.0%
Avg. policy time 0.066s
Avg. simulator time 0.053s
Of which: market data 0.017s
Of which: result 0.015s
Total time 149.592s
#################################################################
And these are the figures that are plotted. The first run, with longer
time spent in the policy (which does the covariance estimation,
in the cvxportfolio.forecast.HistoricalFactorizedCovariance
object).
This figure is made by the cvxportfolio.result.BacktestResult.times_plot() method.#
And the second run, when covariances are loaded from disk. The time taken to load them is accounted for in the simulator times (there are a few spikes there, saved covariances are loaded at each change of universe).
This figure is made by the cvxportfolio.result.BacktestResult.times_plot() method.#