CTYPTO/freqtrade
1
0
Fork 0
mirror of https://github.com/freqtrade/freqtrade.git synced 2025-12-05 23:41:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Reorganise multipage doc

Browse Source
This commit is contained in:
th0rntwig
2022-09-17 17:43:39 +02:00
parent ede282392f
commit c210d6614c
10 changed files with 359 additions and 392 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
docs/freqai-data-handling.md
View File

@@ -1,31 +1,23 @@
# Data handling
`FreqAI` aims to organize prediction data, model files, and meta data in a way that automates crash resilient reloading and simplifies post-processing. Data is organized into `user_data_dir/models/` and contains all the data associated with the trainings and backtests. This file structure is heavily controlled and inferenced by the `FreqaiDataKitchen()` and should therefore not be manually modified.
`FreqAI` aims to organize model files, prediction data, and meta data in a way that simplifies post-processing and enhances crash recililence by automatic data reloading. The data is saved in a file structure,`user_data_dir/models/`, which contains all the data associated with the trainings and backtests. The `FreqaiDataKitchen()` relies heavily on the file structure for proper training and inferencing and should therefore not be manually modified.
## File structure
The file structure is automatically generated based on the user set `identifier` in the configuration file. The following structure shows where the data is stored for post processing:
The file structure is automatically generated based on the model `identifier` set by the user in the [config](freqai-configuration.md#setting-up-the-configuration-file). The following structure shows where the data is stored for post processing:
* `config_*.json`
* a copy of the user submitted configuration file
* `historic_predictions.pkl`
* all historic predictions generated during the lifetime of the `identifier` live deployment. These are also used to reload the model after a crash or a config change. A backup file is always held incase of corruption on the main file - FreqAI automatically detects corruption and replaces the corrupted file with the backup.
* `pair_dictionary.json`
* contains the training queue as well as the location of the most recently trained model on disk.
* `sub-train-*_TIMESTAMP`
* a folder containing all the files associated with a single model, such as:
* `*_metadata.json`
* metadata for the model, such as normalization max/mins, expected training feature list, etc.
* `*_model.*`
* the model file saved to disk for reloading from crash. Can be `joblib` (typical boosting libs), `zip` (stable_baselines), `hd5` (keras type), etc.
* `*_pca_object.pkl`
* the PCA transform (if the user set `principal_component_analysis: true` in their config) which will be used to transform unseen prediction features.
* `*_svm_model.pkl`
* the Support Vector Machine model which is used to detect outliers in unseen prediction features.
* `*_trained_df.pkl`
* the dataframe containing all the training features used to train the particular model. This is used for computing the Dissimilarity Index and can be used for post-processing.
* `*_trained_dates.df.pkl`
* dates associated with the `trained_df.pkl`, useful for post-processing.
| Structure | Description |
|-----------|-------------|
| `config_*.json` | A copy of the model specific configuration file. |
| `historic_predictions.pkl` | A file containing all historic predictions generated during the lifetime of the `identifier` model during live deployment. `historic_predictions.pkl` is used to reload the model after a crash or a config change. A backup file is always held incase of corruption on the main file. **FreqAI automatically detects corruption and replaces the corrupted file with the backup**. |
| `pair_dictionary.json` | A file containing the training queue as well as the on disk location of the most recently trained model. |
| `sub-train-*_TIMESTAMP` | A folder containing all the files associated with a single model, such as: <br>
|| `*_metadata.json` - Metadata for the model, such as normalization max/mins, expected training feature list, etc. <br>
|| `*_model.*` - The model file saved to disk for reloading from a crash. Can be `joblib` (typical boosting libs), `zip` (stable_baselines), `hd5` (keras type), etc. <br>
|| `*_pca_object.pkl` - The [Principal component analysis (PCA)](freqai-feature-engineering.md#data-dimensionality-reduction-with-principal-component-analysis) transform (if the user set `principal_component_analysis: true` in their config) which will be used to transform unseen prediction features. <br>
|| `*_svm_model.pkl` - The [Support Vector Machine (SVM)](freqai-outlier-detection.md#identifying-outliers-using-a-support-vector-machine-svm) model which is used to detect outliers in unseen prediction features. <br>
|| `*_trained_df.pkl` - The dataframe containing all the training features used to train the `identifier` model. This is used for computing the [Dissimilarity Index (DI)](freqai-outlier-detection.md#identifying-outliers-with-the-dissimilarity-index-di) and can also be used for post-processing. <br>
|| `*_trained_dates.df.pkl` - The dates associated with the `trained_df.pkl`, which is useful for post-processing. |
The example file structure would look like this:
@@ -66,43 +58,19 @@ The example file structure would look like this:
│   └── cb_ada_1662821399_trained_df.pkl
```
## Backtesting
When users run a backtest, `FreqAI` will automatically save the predictions to be reused for future runs under the same `identifier`. This is a performance enhancement geared towards enabling high-level hyperopting of entry/exit criteria. That means the user will see an additional directory created in their `unique-id` folder called `predictions` which contains all the predictions stored in `hdf` format.
If users wish to change their features, they **must** use a new identifier which will signal to `FreqAI` to train new models. If users wish to save the models generated during a particular backtest so they can start a live deployment without an initial training, they must set `save_backtest_models` to `True` in their configuration file.
### Downloading data for backtesting
Live/dry instances will download the data automatically for the user, but users who wish to use backtesting functionality still need to download the necessary data using `download-data` (details [here](data-download.md#data-downloading)). FreqAI users need to pay careful attention to understanding how much *additional* data needs to be downloaded to ensure that they have a sufficient amount of training data *before* the start of their backtesting timerange. The amount of additional data can be roughly estimated by moving the start date of the timerange backwards by `train_period_days` and the `startup_candle_count` ([details](freqai-configuration.md#setting-the-startupcandlecount)) from the beginning of the desired backtesting timerange.
As an example, if we wish to backtest the `--timerange` above of `20210501-20210701`, and we use the example config which sets `train_period_days` to 15. The startup candle count is 40 on a maximum `include_timeframes` of 1h. We would need 20210501 - 15 days - 40 * 1h / 24 hours = 20210414 (16.7 days earlier than the start of the desired training timerange).
## Live deployments
### Auto data download
### Automatic data download
`FreqAI` automatically downloads and proper amount of data to ensure it can train a model using the user defined `train_period_days` and the strategy defined `startup_candle_count`.
FreqAI automatically downloads the proper amount of data needed to ensure training of a model through the user defined `train_period_days` and `startup_candle_count` (see the [parameter table](freqai-parameter-table.md) for detailed descriptions of these parameters).
### Historical predictions
### Saving prediction data
The historical predictions are collected for the life-time of a single `identifier` and stored in `historical_predictions.pkl`
### Defining model expirations
During dry/live mode, FreqAI trains each coin pair sequentially (on separate threads/GPU from the main Freqtrade bot). This means that there is always an age discrepancy between models. If a user is training on 50 pairs, and each pair requires 5 minutes to train, the oldest model will be over 4 hours old. This may be undesirable if the characteristic time scale (the trade duration target) for a strategy is less than 4 hours. The user can decide to only make trade entries if the model is less than
a certain number of hours old by setting the `expiration_hours` in the config file:
```json
"freqai": {
"expiration_hours": 0.5,
}
```
In the presented example config, the user will only allow predictions on models that are less than 1/2 hours old.
All predictions made during the lifetime of a specific `identifier` model are stored in `historical_predictions.pkl` to allow for reloading after a crash or changes made to the config.
### Purging old model data
FreqAI stores new model files each time it retrains. These files become obsolete as new models are trained and FreqAI adapts to new market conditions. Users planning to leave FreqAI running for extended periods of time with high frequency retraining should enable `purge_old_models` in their config:
FreqAI stores new model files after each successful training. These files become obsolete as new models are generated to adapt to new market conditions. The user who is planning to leave FreqAI running for extended periods of time with high frequency retraining should enable `purge_old_models` in their config:
```json
"freqai": {
@@ -110,15 +78,15 @@ FreqAI stores new model files each time it retrains. These files become obsolete
}
```
This will automatically purge all models older than the two most recently trained ones.
This will automatically purge all models older than the two most recently trained ones to save disk space.
### Returning additional info from training
The user may find that there are some important metrics that they'd like to return to the strategy at the end of each model training.
The user can include these metrics by assigning them to `dk.data['extra_returns_per_train']['my_new_value'] = XYZ` inside their custom prediction model class. FreqAI takes the `my_new_value` assigned in this dictionary and expands it to fit the return dataframe to the strategy.
The user can then use the value in the strategy with `dataframe['my_new_value']`. An example of how this is already used in FreqAI is
the `&*_mean` and `&*_std` values, which indicate the mean and standard deviation of the particular target (label) during the most recent training.
An example, where the user wants to use live metrics from the trade database, is shown below:
The user may find that there are important metrics that they would like to return to their strategy at the end of each model training. Such metrics are returned by assigning them to `dk.data['extra_returns_per_train']['my_new_value'] = XYZ` inside the user's custom prediction model class.
FreqAI takes the `my_new_value` assigned in this dictionary and expands it to fit the dataframe that is returned to the strategy. The user can then use the returned metrics in their strategy through `dataframe['my_new_value']`. An example of how return values can be used in FreqAI are the `&*_mean` and `&*_std` values that are used to [created a dynamic target threshold](freqai-configuration.md#creating-a-dynamic-target-threshold).
Another example, where the user wants to use live metrics from the trade database, is shown below:
```json
"freqai": {
@@ -126,4 +94,22 @@ An example, where the user wants to use live metrics from the trade database, is
}
```
The user needs to set the standard dictionary in the config so that FreqAI can return proper dataframe shapes. These values will likely be overridden by the prediction model, but in the case where the model has yet to set them, or needs a default initial value, this is the value that will be returned.
The user needs to set the standard dictionary in the config so that FreqAI can return proper dataframe shapes. These values will likely be overridden by the prediction model, but in the case where the model has yet to set them, or needs a default initial value, the preset values are what will be returned.
## Backtesting
### Saving prediction data
To allow the user to tweak their strategy (**not** the features!), FreqAI will automatically save the predictions during backtesting so that they can be reused for future backtests and live runs using the same `identifier` model. This provides a performance enhancement geared towards enabling **high-level hyperopting** of entry/exit criteria.
An additional directory called `predictions`, which contains all the predictions stored in `hdf` format, will be created in the `unique-id` folder.
If the user wishes to change their **features**, they **must** set a new `identifier` in the config to signal to `FreqAI` to train new models.
If the user wishes to save the models generated during a particular backtest so that they can start a live deployment from one of them instead of training a new model, they must set `save_backtest_models` to `True` in their configuration file.
### Downloading data to cover the full backtest period
For live/dry deployments, FreqAI will download the necessary data automatically. However, the user who wishes to use backtesting functionality needs to download the necessary data using `download-data` (details [here](data-download.md#data-downloading)). FreqAI users need to pay careful attention to understanding how much *additional* data needs to be downloaded to ensure that they have a sufficient amount of training data *before* the start of their backtesting timerange. The amount of additional data can be roughly estimated by moving the start date of the timerange backwards by `train_period_days` and the `startup_candle_count` (see the [parameter table](freqai-parameter-table.md) for detailed descriptions of these parameters) from the beginning of the desired backtesting timerange.
As an example, to backtest the `--timerange 20210501-20210701` using the [example config](freqai-configuration.md#setting-up-the-configuration-file) which sets `train_period_days` to 30, together with `startup_candle_count: 40` on a maximum `include_timeframes` of 1h, the start date for the downloaded data needs to be `20210501` - 30 days - 40 * 1h / 24 hours = 20210330 (31.7 days earlier than the start of the desired training timerange).