AIを使って不労所得の道へ(9)~quantstatsを使ってFX強化学習の記録をポートフォリオ分析してみた~

AIを使って不労所得

はじめに

素人でも戦略的に取引したい

株やFXの知識がないため,戦略的な取引ができない素人が多いと思います.私もその一人です.
そこで,強化学習の力を借りて戦略的に取引をしようと思います.

AIncomeプロジェクト

AIで所得(Income)を得るため「AIncome」と名付けました.

前回までの取り組み

前回は強化学習とそれをシミュレーションするOpenAI Gymの説明をしました*1.さらには,ランダムで行動を決定して売り買いしてみたところ,ランダムでも利益が出るような結果になっていました*1.また,ランダム投資からPPO2へ深層学習強化学習のモデル投入*2やTensorboardで可視化*3,手数料の考慮*4などを取り組んでいました.
さらに,学習回数を10倍に増やしたりしてみましたところ,処理時間が大幅に増えてきました*5ので,並列化の知識を付け*6,FXのデータに適用してみました*7

今回の取り組みの概要

強化学習で行ったFXの取引を分析してみます.

Google Drive setting

from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
%cd /content/drive/MyDrive/AIncome
/content/drive/MyDrive/AIncome
!pwd
/content/drive/MyDrive/AIncome

作業フォルダ

!mkdir AutoTrade09
mkdir: cannot create directory ‘AutoTrade09’: File exists
%cd  AutoTrade09
/content/drive/MyDrive/AIncome/AutoTrade09
!pwd
/content/drive/MyDrive/AIncome/AutoTrade09

必要パッケージのインストール

"""
#!pip install gym[all] -U
!pip install "gym==0.19.0"
!pip install stable-baselines[mpi]
#!pip install tensorflow==1.14.0
!pip install tensorflow-gpu==1.14.0
!pip install pyqt5
!pip install imageio
!pip install gym-anytrading
"""
!pip install "gym==0.19.0"
!pip install stable-baselines[mpi]
!pip uninstall -y tensorflow-gpu
!pip uninstall -y tensorflow
!pip install tensorflow-gpu==1.14.0
!pip install gym-anytrading

!pip install QuantStats
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: gym==0.19.0 in /usr/local/lib/python3.7/dist-packages (0.19.0)
Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from gym==0.19.0) (1.21.6)
Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym==0.19.0) (1.5.0)
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: stable-baselines[mpi] in /usr/local/lib/python3.7/dist-packages (2.10.2)
Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.1.0)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (4.6.0.66)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.7.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.21.6)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.3.5)
Requirement already satisfied: cloudpickle>=0.5.5 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (1.5.0)
Requirement already satisfied: gym[atari,classic_control]>=0.11 in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (0.19.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (3.2.2)
Requirement already satisfied: mpi4py in /usr/local/lib/python3.7/dist-packages (from stable-baselines[mpi]) (3.1.3)
Requirement already satisfied: pyglet>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (1.5.26)
Requirement already satisfied: atari-py==0.2.6 in /usr/local/lib/python3.7/dist-packages (from gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (0.2.6)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from atari-py==0.2.6->gym[atari,classic_control]>=0.11->stable-baselines[mpi]) (1.15.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (2.8.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (3.0.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines[mpi]) (1.4.4)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->stable-baselines[mpi]) (4.1.1)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines[mpi]) (2022.2.1)
Found existing installation: tensorflow-gpu 1.14.0
Uninstalling tensorflow-gpu-1.14.0:
  Successfully uninstalled tensorflow-gpu-1.14.0
WARNING: Skipping tensorflow as it is not installed.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting tensorflow-gpu==1.14.0
  Using cached tensorflow_gpu-1.14.0-cp37-cp37m-manylinux1_x86_64.whl (377.1 MB)
Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.21.6)
Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.14.0)
Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.14.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.1.2)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (0.8.1)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.15.0)
Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (0.2.0)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.47.0)
Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.2.0)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (0.5.3)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.1.0)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.0.8)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (0.37.1)
Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (1.14.1)
Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu==1.14.0) (3.17.3)
Requirement already satisfied: h5py in /usr/local/lib/python3.7/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==1.14.0) (3.1.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (1.0.1)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (57.4.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (3.4.1)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (4.12.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (4.1.1)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<1.15.0,>=1.14.0->tensorflow-gpu==1.14.0) (3.8.1)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py->keras-applications>=1.0.6->tensorflow-gpu==1.14.0) (1.5.2)
Installing collected packages: tensorflow-gpu
Successfully installed tensorflow-gpu-1.14.0

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: gym-anytrading in /usr/local/lib/python3.7/dist-packages (1.3.1)
Requirement already satisfied: numpy>=1.16.4 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (1.21.6)
Requirement already satisfied: matplotlib>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (3.2.2)
Requirement already satisfied: pandas>=0.24.2 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (1.3.5)
Requirement already satisfied: gym>=0.12.5 in /usr/local/lib/python3.7/dist-packages (from gym-anytrading) (0.19.0)
Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym>=0.12.5->gym-anytrading) (1.5.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (0.11.0)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (2.8.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.1.1->gym-anytrading) (1.4.4)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib>=3.1.1->gym-anytrading) (4.1.1)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.2->gym-anytrading) (2022.2.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib>=3.1.1->gym-anytrading) (1.15.0)
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: QuantStats in /usr/local/lib/python3.7/dist-packages (0.0.59)
Requirement already satisfied: python-dateutil>=2.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (2.8.2)
Requirement already satisfied: tabulate>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (0.8.10)
Requirement already satisfied: matplotlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (3.2.2)
Requirement already satisfied: yfinance>=0.1.70 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (0.1.74)
Requirement already satisfied: scipy>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (1.7.3)
Requirement already satisfied: seaborn>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (0.11.2)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (1.21.6)
Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.7/dist-packages (from QuantStats) (1.3.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->QuantStats) (3.0.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->QuantStats) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=3.0.0->QuantStats) (1.4.4)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib>=3.0.0->QuantStats) (4.1.1)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->QuantStats) (2022.2.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.0->QuantStats) (1.15.0)
Requirement already satisfied: lxml>=4.5.1 in /usr/local/lib/python3.7/dist-packages (from yfinance>=0.1.70->QuantStats) (4.9.1)
Requirement already satisfied: requests>=2.26 in /usr/local/lib/python3.7/dist-packages (from yfinance>=0.1.70->QuantStats) (2.28.1)
Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from yfinance>=0.1.70->QuantStats) (0.0.11)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.26->yfinance>=0.1.70->QuantStats) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.26->yfinance>=0.1.70->QuantStats) (1.24.3)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.26->yfinance>=0.1.70->QuantStats) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.26->yfinance>=0.1.70->QuantStats) (2022.6.15)
!pip uninstall tensorboard-plugin-wit --yes
WARNING: Skipping tensorboard-plugin-wit as it is not installed.

インポート

"""

import gym_anytrading

from gym_anytrading.envs import TradingEnv, ForexEnv, StocksEnv, Actions, Positions
from gym_anytrading.datasets import FOREX_EURUSD_1H_ASK, STOCKS_GOOGL
from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines import PPO2
from stable_baselines import ACKTR
from stable_baselines.bench import Monitor
from stable_baselines.common import set_global_seeds
"""
import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import gym

import gym_anytrading
from gym_anytrading.envs import TradingEnv, ForexEnv, StocksEnv, Actions, Positions
from gym_anytrading.datasets import FOREX_EURUSD_1H_ASK, STOCKS_GOOGL

from stable_baselines.bench import Monitor

from stable_baselines.common.vec_env import DummyVecEnv
from stable_baselines.common.policies import MlpPolicy
from stable_baselines.common import make_vec_env
#from stable_baselines import PPO2
from stable_baselines import ACKTR
from stable_baselines import A2C

import quantstats as qs

設定


# ログフォルダの生成
log_dir = './logs/'
os.makedirs(log_dir, exist_ok=True)

# train data
idx1 = 100
idx2 = 5000

# test data
idx3 = 6000

window_size = 100

trade_fee = 0

環境の生成


def my_process_data(env):
    start = env.frame_bound[0] - env.window_size
    end = env.frame_bound[1]
    prices = env.df.loc[:, 'Low'].to_numpy()[start:end]
    signal_features = env.df.loc[:, ['Close', 'Open', 'High', 'Low']].to_numpy()[start:end]
    return prices, signal_features

class MyForexEnv(ForexEnv):
    _process_data = my_process_data

"""
# df = gym_anytrading.datasets.STOCKS_GOOGL.copy()
df = FOREX_EURUSD_1H_ASK.copy()
print(df.head())
window_size = 10
start_index = window_size
end_index = len(df)

def env_maker():
  return gym.make(
    'stocks-v0',
    df = df,
    window_size = window_size,
    frame_bound = (start_index, end_index)
  )
"""

env_marker2 = lambda:  MyForexEnv(df=df, window_size=window_size, frame_bound=(idx1, idx2))
env_marker2.trade_fee = trade_fee
# env = DummyVecEnv([env_maker for _ in range(10)])
# env = DummyVecEnv([env_maker for _ in range(1)])
env = DummyVecEnv([env_marker2 for _ in range(1)])

#env = Monitor(env, log_dir, allow_early_resets=True)

Train trade

policy_kwargs = dict(net_arch=[64, 'lstm', dict(vf=[128, 128, 128], pi=[64, 64])])
model = PPO2('MlpLstmPolicy', env, verbose=1, policy_kwargs=policy_kwargs, nminibatches=1, tensorboard_log=log_dir)
model.learn(total_timesteps=100000)
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

Wrapping the env in a DummyVecEnv.
WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'
n_updates : 781
total_timesteps : 100000

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:509: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.bool_ here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

ストリーミング出力は最後の 5000 行に切り捨てられました。
| serial_timesteps   | 57344           |
| time_elapsed       | 300             |
| total_timesteps    | 57344           |
| value_loss         | 719.87354       |
----------------------------------------
0
----------------------------------------
| approxkl           | 5.552556e-06    |
| clipfrac           | 0.0             |
| episode_reward     | [-632.99996714] |
| explained_variance | 2.81e-05        |
| fps                | 231             |
| n_updates          | 449             |
| policy_entropy     | 0.69281244      |
| policy_loss        | -0.00014998415  |
| serial_timesteps   | 57472           |
| time_elapsed       | 301             |
| total_timesteps    | 57472           |
| value_loss         | 401.4166        |
----------------------------------------
0
----------------------------------------
| approxkl           | 0.00045888976   |
| clipfrac           | 0.0             |
| episode_reward     | [-643.29997808] |
| explained_variance | 2.68e-06        |
| fps                | 228             |
| n_updates          | 450             |
| policy_entropy     | 0.69295144      |
| policy_loss        | -0.0013548336   |
| serial_timesteps   | 57600           |
| time_elapsed       | 301             |
| total_timesteps    | 57600           |
| value_loss         | 1238.9834       |
----------------------------------------
0

.....

0
----------------------------------------
| approxkl           | 1.1865658e-05   |
| clipfrac           | 0.0             |
| episode_reward     | [-469.40000837] |
| explained_variance | 6.97e-06        |
| fps                | 218             |
| n_updates          | 781             |
| policy_entropy     | 0.13281491      |
| policy_loss        | 0.00047062582   |
| serial_timesteps   | 99968           |
| time_elapsed       | 519             |
| total_timesteps    | 99968           |
| value_loss         | 142.19855       |
----------------------------------------

<__main__.PPO2 at 0x7fdf370c6250>

Save model

model.save("ppo2_forex-v0_lstm")

Load model

model = PPO2.load("ppo2_forex-v0_lstm")
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

Loading a model without an environment, this model cannot be trained until it has a valid environment.
WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutgoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: converting >: AttributeError: module 'gast' has no attribute 'Index'

"""
# multiprocess environment
# env = make_vec_env('forex-v0', n_envs=400)
env.trade_fee = trade_fee

model = PPO2(MlpPolicy, env, tensorboard_log=log_dir)
# model = PPO2('MlpPolicy', env, tensorboard_log=log_dir)
# model = ACKTR('MlpPolicy', env, tensorboard_log=log_dir)
# model = A2C('MlpPolicy', env, tensorboard_log=log_dir)

model.learn(total_timesteps=2500000)
model.save("ppo2_forex-v0")

# del model # remove to demonstrate saving and loading

model = PPO2.load("ppo2_forex-v0")
"""
'\n# multiprocess environment\n# env = make_vec_env(\'forex-v0\', n_envs=400)\nenv.trade_fee = trade_fee\n\nmodel = PPO2(MlpPolicy, env, tensorboard_log=log_dir)\n# model = PPO2(\'MlpPolicy\', env, tensorboard_log=log_dir)\n# model = ACKTR(\'MlpPolicy\', env, tensorboard_log=log_dir)\n# model = A2C(\'MlpPolicy\', env, tensorboard_log=log_dir)\n\nmodel.learn(total_timesteps=2500000)\nmodel.save("ppo2_forex-v0")\n\n# del model # remove to demonstrate saving and loading\n\nmodel = PPO2.load("ppo2_forex-v0")\n'

Test trade

import numpy as np

# env = DummyVecEnv([env_maker for _ in range(10)])
# env = env_maker()
env = env_marker2()
observation = env.reset()

while True:
    observation = observation[np.newaxis, ...]
    #observation = np.tile(observation, (10, 1, 1))

    # ----------------------------
    # observation
    #
    # print(observation)
    # print(observation.shape)

    # action = env.action_space.sample()
    action, _states = model.predict(observation)
    # print(action.shape)
    observation, reward, done, info = env.step(action)

    print(done)
    # env.render()
    if done:
        print("info:", info)
        break
False
False
False

....

True
info: {'total_reward': -851.4999999999702, 'total_profit': 0.7702014657332269, 'position': 0}

Analysis Using quantstats

print(env)
print(len(env.history['total_profit']))
print(len(df.index))
print(len(df.index[idx1:idx2]))

4899
6225
4900
qs.extend_pandas()

net_worth = pd.Series(env.history['total_profit'], index=df.index[idx1+1:idx2])
returns = net_worth.pct_change().iloc[1:]

qs.reports.full(returns)
qs.reports.html(returns, output='a2c_quantstats.html')
    [Performance Metrics]

                               Strategy
    -------------------------  ----------
    Start Period               2017-06-01
    End Period                 2017-10-19
    Risk-Free Rate             0.0%
    Time in Market             13.0%

    Cumulative Return          -22.98%
    CAGR﹪                     -49.38%

    Sharpe                     -0.94
    Prob. Sharpe Ratio         0.0%
    Smart Sharpe               -0.94
    Sortino                    -1.14
    Smart Sortino              -1.14
    Sortino/√2                 -0.81
    Smart Sortino/√2           -0.8
    Omega                      0.59

    Max Drawdown               -23.2%
    Longest DD Days            60
    Volatility (ann.)          1.41%
    Calmar                     -2.13
    Skew                       -3.78
    Kurtosis                   72.91

    Expected Daily %           -0.01%
    Expected Monthly %         -2.15%
    Expected Yearly %          -22.98%
    Kelly Criterion            -27.89%
    Risk of Ruin               0.0%
    Daily Value-at-Risk        -0.15%
    Expected Shortfall (cVaR)  -0.15%

    Max Consecutive Wins       1
    Max Consecutive Losses     1
    Gain/Pain Ratio            -0.56
    Gain/Pain (1M)             -0.99

    Payoff Ratio               0.89
    Profit Factor              0.59
    Common Sense Ratio         0.01
    CPC Index                  0.21
    Tail Ratio                 0.02
    Outlier Win Ratio          29.53
    Outlier Loss Ratio         2.19

    MTD                        -3.72%
    3M                         -6.79%
    6M                         -16.44%
    YTD                        -22.98%
    1Y                         -22.98%
    3Y (ann.)                  -49.38%
    5Y (ann.)                  -49.38%
    10Y (ann.)                 -49.38%
    All-time (ann.)            -49.38%

    Best Day                   0.82%
    Worst Day                  -1.67%
    Best Month                 0.35%
    Worst Month                -5.4%
    Best Year                  -22.98%
    Worst Year                 -22.98%

    Avg. Drawdown              -12.02%
    Avg. Drawdown Days         8
    Recovery Factor            -0.99
    Ulcer Index                0.14
    Serenity Index             -0.01

    Avg. Up Month              0.35%
    Avg. Down Month            -2.36%
    Win Days %                 39.74%
    Win Month %                8.33%
    Win Quarter %              0.0%
    Win Year %                 0.0%

    [5 Worst Drawdowns]

        Start       Valley      End           Days    Max Drawdown    99% Max Drawdown
    --  ----------  ----------  ----------  ------  --------------  ------------------
     1  2017-12-01  2017-08-09  2017-10-19     -44          -23.20              -23.11
     2  2017-09-01  2017-10-01  2017-11-01      60           -0.83               -0.75

    [Strategy Visualization]
    via Matplotlib

Plot Results

plt.figure(figsize=(30, 10))
env.render_all()
plt.show()

tensorboard

# Load the TensorBoard notebook extension
%load_ext tensorboard
The tensorboard extension is already loaded. To reload it, use:
  %reload_ext tensorboard
%tensorboard --logdir logs/
Reusing TensorBoard on port 6006 (pid 3451), started 1:20:52 ago. (Use '!kill 3451' to kill it.)

参考サイト

コメント

タイトルとURLをコピーしました