diff --git a/docs/developer.md b/docs/developer.md index 036109d5b..f09ae2c76 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -85,6 +85,35 @@ docker-compose exec freqtrade_develop /bin/bash ![image](https://user-images.githubusercontent.com/419355/65456522-ba671a80-de06-11e9-9598-df9ca0d8dcac.png) +## ErrorHandling + +Freqtrade Exceptions all inherit from `FreqtradeException`. +This general class of error should however not be used directly. Instead, multiple specialized sub-Exceptions exist. + +Below is an outline of exception inheritance hierarchy: + +``` ++ FreqtradeException +| ++---+ OperationalException +| ++---+ DependencyException +| | +| +---+ PricingError +| | +| +---+ ExchangeError +| | +| +---+ TemporaryError +| | +| +---+ DDosProtection +| | +| +---+ InvalidOrderException +| | +| +---+ RetryableOrderError +| ++---+ StrategyError +``` + ## Modules ### Dynamic Pairlist diff --git a/docs/edge.md b/docs/edge.md index c91e72a3a..dcb559f96 100644 --- a/docs/edge.md +++ b/docs/edge.md @@ -6,7 +6,8 @@ This page explains how to use Edge Positioning module in your bot in order to en Edge positioning is not compatible with dynamic (volume-based) whitelist. !!! Note - Edge does not consider anything else than buy/sell/stoploss signals. So trailing stoploss, ROI, and everything else are ignored in its calculation. + Edge does not consider anything other than *its own* buy/sell/stoploss signals. It ignores the stoploss, trailing stoploss, and ROI settings in the strategy configuration file. + Therefore, it is important to understand that Edge can improve the performance of some trading strategies but *decrease* the performance of others. ## Introduction @@ -89,7 +90,7 @@ You can also use this value to evaluate the effectiveness of modifications to th ## How does it work? -If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over *N* trades for each stoploss. Here is an example: +Edge combines dynamic stoploss, dynamic positions, and whitelist generation into one isolated module which is then applied to the trading strategy. If enabled in config, Edge will go through historical data with a range of stoplosses in order to find buy and sell/stoploss signals. It then calculates win rate and expectancy over *N* trades for each stoploss. Here is an example: | Pair | Stoploss | Win Rate | Risk Reward Ratio | Expectancy | |----------|:-------------:|-------------:|------------------:|-----------:| @@ -186,6 +187,12 @@ An example of its output: | APPC/BTC | -0.02 | 0.44 | 2.28 | 1.27 | 0.44 | 25 | 43 | | NEBL/BTC | -0.03 | 0.63 | 1.29 | 0.58 | 0.44 | 19 | 59 | +Edge produced the above table by comparing `calculate_since_number_of_days` to `minimum_expectancy` to find `min_trade_number` historical information based on the config file. The timerange Edge uses for its comparisons can be further limited by using the `--timerange` switch. + +In live and dry-run modes, after the `process_throttle_secs` has passed, Edge will again process `calculate_since_number_of_days` against `minimum_expectancy` to find `min_trade_number`. If no `min_trade_number` is found, the bot will return "whitelist empty". Depending on the trade strategy being deployed, "whitelist empty" may be return much of the time - or *all* of the time. The use of Edge may also cause trading to occur in bursts, though this is rare. + +If you encounter "whitelist empty" a lot, condsider tuning `calculate_since_number_of_days`, `minimum_expectancy` and `min_trade_number` to align to the trading frequency of your strategy. + ### Update cached pairs with the latest data Edge requires historic data the same way as backtesting does. diff --git a/docs/faq.md b/docs/faq.md index 151b2c054..48f52a566 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -1,5 +1,9 @@ # Freqtrade FAQ +## Beginner Tips & Tricks + +* When you work with your strategy & hyperopt file you should use a proper code editor like vscode or Pycharm. A good code editor will provide syntax highlighting as well as line numbers, making it easy to find syntax errors (most likely, pointed out by Freqtrade during startup). + ## Freqtrade common issues ### The bot does not start @@ -15,10 +19,12 @@ This could have the following reasons: ### I have waited 5 minutes, why hasn't the bot made any trades yet?! -Depending on the buy strategy, the amount of whitelisted coins, the +* Depending on the buy strategy, the amount of whitelisted coins, the situation of the market etc, it can take up to hours to find good entry position for a trade. Be patient! +* Or it may because of a configuration error? Best check the logs, it's usually telling you if the bot is simply not getting buy signals (only heartbeat messages), or if there is something wrong (errors / exceptions in the log). + ### I have made 12 trades already, why is my total profit negative?! I understand your disappointment but unfortunately 12 trades is just @@ -129,25 +135,27 @@ to find a great result (unless if you are very lucky), so you probably have to run it for 10.000 or more. But it will take an eternity to compute. -We recommend you to run it at least 10.000 epochs: +Since hyperopt uses Bayesian search, running for too many epochs may not produce greater results. + +It's therefore recommended to run between 500-1000 epochs over and over until you hit at least 10.000 epocs in total (or are satisfied with the result). You can best judge by looking at the results - if the bot keeps discovering better strategies, it's best to keep on going. ```bash -freqtrade hyperopt -e 10000 +freqtrade hyperopt -e 1000 ``` or if you want intermediate result to see ```bash -for i in {1..100}; do freqtrade hyperopt -e 100; done +for i in {1..100}; do freqtrade hyperopt -e 1000; done ``` -### Why it is so long to run hyperopt? +### Why does it take a long time to run hyperopt? -Finding a great Hyperopt results takes time. +* Discovering a great strategy with Hyperopt takes time. Study www.freqtrade.io, the Freqtrade Documentation page, join the Freqtrade [Slack community](https://join.slack.com/t/highfrequencybot/shared_invite/enQtNjU5ODcwNjI1MDU3LTU1MTgxMjkzNmYxNWE1MDEzYzQ3YmU4N2MwZjUyNjJjODRkMDVkNjg4YTAyZGYzYzlhOTZiMTE4ZjQ4YzM0OGE) - or the Freqtrade [discord community](https://discord.gg/X89cVG). While you patiently wait for the most advanced, free crypto bot in the world, to hand you a possible golden strategy specially designed just for you. -If you wonder why it takes a while to find great hyperopt results +* If you wonder why it can take from 20 minutes to days to do 1000 epocs here are some answers: -This answer was written during the under the release 0.15.1, when we had: +This answer was written during the release 0.15.1, when we had: - 8 triggers - 9 guards: let's say we evaluate even 10 values from each @@ -157,7 +165,14 @@ The following calculation is still very rough and not very precise but it will give the idea. With only these triggers and guards there is already 8\*10^9\*10 evaluations. A roughly total of 80 billion evals. Did you run 100 000 evals? Congrats, you've done roughly 1 / 100 000 th -of the search space. +of the search space, assuming that the bot never tests the same parameters more than once. + +* The time it takes to run 1000 hyperopt epocs depends on things like: The available cpu, harddisk, ram, timeframe, timerange, indicator settings, indicator count, amount of coins that hyperopt test strategies on and the resulting trade count - which can be 650 trades in a year or 10.0000 trades depending if the strategy aims for big profits by trading rarely or for many low profit trades. + +Example: 4% profit 650 times vs 0,3% profit a trade 10.000 times in a year. If we assume you set the --timerange to 365 days. + +Example: +`freqtrade --config config.json --strategy SampleStrategy --hyperopt SampleHyperopt -e 1000 --timerange 20190601-20200601` ## Edge module diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index 3a236ee87..4068e364b 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,2 +1,2 @@ -mkdocs-material==5.4.0 +mkdocs-material==5.5.3 mdx_truly_sane_lists==1.2 diff --git a/docs/rest-api.md b/docs/rest-api.md index a8d902b53..68754f79a 100644 --- a/docs/rest-api.md +++ b/docs/rest-api.md @@ -46,7 +46,7 @@ secrets.token_hex() ### Configuration with docker -If you run your bot using docker, you'll need to have the bot listen to incomming connections. The security is then handled by docker. +If you run your bot using docker, you'll need to have the bot listen to incoming connections. The security is then handled by docker. ``` json "api_server": { @@ -106,26 +106,29 @@ python3 scripts/rest_client.py --config rest_config.json [optional par ## Available commands -| Command | Default | Description | -|----------|---------|-------------| -| `start` | | Starts the trader -| `stop` | | Stops the trader -| `stopbuy` | | Stops the trader from opening new trades. Gracefully closes open trades according to their rules. -| `reload_config` | | Reloads the configuration file -| `show_config` | | Shows part of the current configuration with relevant settings to operation -| `status` | | Lists all open trades -| `count` | | Displays number of trades used and available -| `profit` | | Display a summary of your profit/loss from close trades and some stats about your performance -| `forcesell ` | | Instantly sells the given trade (Ignoring `minimum_roi`). -| `forcesell all` | | Instantly sells all open trades (Ignoring `minimum_roi`). -| `forcebuy [rate]` | | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True) -| `performance` | | Show performance of each finished trade grouped by pair -| `balance` | | Show account balance per currency -| `daily ` | 7 | Shows profit or loss per day, over the last n days -| `whitelist` | | Show the current whitelist -| `blacklist [pair]` | | Show the current blacklist, or adds a pair to the blacklist. -| `edge` | | Show validated pairs by Edge if it is enabled. -| `version` | | Show version +| Command | Description | +|----------|-------------| +| `ping` | Simple command testing the API Readiness - requires no authentication. +| `start` | Starts the trader +| `stop` | Stops the trader +| `stopbuy` | Stops the trader from opening new trades. Gracefully closes open trades according to their rules. +| `reload_config` | Reloads the configuration file +| `trades` | List last trades. +| `delete_trade ` | Remove trade from the database. Tries to close open orders. Requires manual handling of this trade on the exchange. +| `show_config` | Shows part of the current configuration with relevant settings to operation +| `status` | Lists all open trades +| `count` | Displays number of trades used and available +| `profit` | Display a summary of your profit/loss from close trades and some stats about your performance +| `forcesell ` | Instantly sells the given trade (Ignoring `minimum_roi`). +| `forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`). +| `forcebuy [rate]` | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True) +| `performance` | Show performance of each finished trade grouped by pair +| `balance` | Show account balance per currency +| `daily ` | Shows profit or loss per day, over the last n days (n defaults to 7) +| `whitelist` | Show the current whitelist +| `blacklist [pair]` | Show the current blacklist, or adds a pair to the blacklist. +| `edge` | Show validated pairs by Edge if it is enabled. +| `version` | Show version Possible commands can be listed from the rest-client script using the `help` command. diff --git a/docs/telegram-usage.md b/docs/telegram-usage.md index f423a9376..9776b26ba 100644 --- a/docs/telegram-usage.md +++ b/docs/telegram-usage.md @@ -9,7 +9,7 @@ Telegram user id. Start a chat with the [Telegram BotFather](https://telegram.me/BotFather) -Send the message `/newbot`. +Send the message `/newbot`. *BotFather response:* @@ -47,28 +47,30 @@ Per default, the Telegram bot shows predefined commands. Some commands are only available by sending them to the bot. The table below list the official commands. You can ask at any moment for help with `/help`. -| Command | Default | Description | -|----------|---------|-------------| -| `/start` | | Starts the trader -| `/stop` | | Stops the trader -| `/stopbuy` | | Stops the trader from opening new trades. Gracefully closes open trades according to their rules. -| `/reload_config` | | Reloads the configuration file -| `/show_config` | | Shows part of the current configuration with relevant settings to operation -| `/status` | | Lists all open trades -| `/status table` | | List all open trades in a table format. Pending buy orders are marked with an asterisk (*) Pending sell orders are marked with a double asterisk (**) -| `/count` | | Displays number of trades used and available -| `/profit` | | Display a summary of your profit/loss from close trades and some stats about your performance -| `/forcesell ` | | Instantly sells the given trade (Ignoring `minimum_roi`). -| `/forcesell all` | | Instantly sells all open trades (Ignoring `minimum_roi`). -| `/forcebuy [rate]` | | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True) -| `/performance` | | Show performance of each finished trade grouped by pair -| `/balance` | | Show account balance per currency -| `/daily ` | 7 | Shows profit or loss per day, over the last n days -| `/whitelist` | | Show the current whitelist -| `/blacklist [pair]` | | Show the current blacklist, or adds a pair to the blacklist. -| `/edge` | | Show validated pairs by Edge if it is enabled. -| `/help` | | Show help message -| `/version` | | Show version +| Command | Description | +|----------|-------------| +| `/start` | Starts the trader +| `/stop` | Stops the trader +| `/stopbuy` | Stops the trader from opening new trades. Gracefully closes open trades according to their rules. +| `/reload_config` | Reloads the configuration file +| `/show_config` | Shows part of the current configuration with relevant settings to operation +| `/status` | Lists all open trades +| `/status table` | List all open trades in a table format. Pending buy orders are marked with an asterisk (*) Pending sell orders are marked with a double asterisk (**) +| `/trades [limit]` | List all recently closed trades in a table format. +| `/delete ` | Delete a specific trade from the Database. Tries to close open orders. Requires manual handling of this trade on the exchange. +| `/count` | Displays number of trades used and available +| `/profit` | Display a summary of your profit/loss from close trades and some stats about your performance +| `/forcesell ` | Instantly sells the given trade (Ignoring `minimum_roi`). +| `/forcesell all` | Instantly sells all open trades (Ignoring `minimum_roi`). +| `/forcebuy [rate]` | Instantly buys the given pair. Rate is optional. (`forcebuy_enable` must be set to True) +| `/performance` | Show performance of each finished trade grouped by pair +| `/balance` | Show account balance per currency +| `/daily ` | Shows profit or loss per day, over the last n days (n defaults to 7) +| `/whitelist` | Show the current whitelist +| `/blacklist [pair]` | Show the current blacklist, or adds a pair to the blacklist. +| `/edge` | Show validated pairs by Edge if it is enabled. +| `/help` | Show help message +| `/version` | Show version ## Telegram commands in action @@ -113,6 +115,7 @@ For each open trade, the bot will send you the following message. ### /status table Return the status of all open trades in a table format. + ``` ID Pair Since Profit ---- -------- ------- -------- @@ -123,6 +126,7 @@ Return the status of all open trades in a table format. ### /count Return the number of trades used and available. + ``` current max --------- ----- @@ -208,7 +212,7 @@ Shows the current whitelist Shows the current blacklist. If Pair is set, then this pair will be added to the pairlist. -Also supports multiple pairs, seperated by a space. +Also supports multiple pairs, separated by a space. Use `/reload_config` to reset the blacklist. > Using blacklist `StaticPairList` with 2 pairs @@ -216,7 +220,7 @@ Use `/reload_config` to reset the blacklist. ### /edge -Shows pairs validated by Edge along with their corresponding winrate, expectancy and stoploss values. +Shows pairs validated by Edge along with their corresponding win-rate, expectancy and stoploss values. > **Edge only validated following pairs:** ``` diff --git a/docs/utils.md b/docs/utils.md index 793c84a93..8c7e381ff 100644 --- a/docs/utils.md +++ b/docs/utils.md @@ -432,9 +432,9 @@ usage: freqtrade hyperopt-list [-h] [-v] [--logfile FILE] [-V] [-c PATH] [--max-trades INT] [--min-avg-time FLOAT] [--max-avg-time FLOAT] [--min-avg-profit FLOAT] [--max-avg-profit FLOAT] - [--min-total-profit FLOAT] - [--max-total-profit FLOAT] [--no-color] - [--print-json] [--no-details] + [--min-total-profit FLOAT] [--max-total-profit FLOAT] + [--min-objective FLOAT] [--max-objective FLOAT] + [--no-color] [--print-json] [--no-details] [--export-csv FILE] optional arguments: @@ -453,6 +453,10 @@ optional arguments: Select epochs on above total profit. --max-total-profit FLOAT Select epochs on below total profit. + --min-objective FLOAT + Select epochs on above objective (- is added by default). + --max-objective FLOAT + Select epochs on below objective (- is added by default). --no-color Disable colorization of hyperopt results. May be useful if you are redirecting output to a file. --print-json Print best result detailization in JSON format. diff --git a/freqtrade/commands/arguments.py b/freqtrade/commands/arguments.py index e6f6f8167..4a87def88 100644 --- a/freqtrade/commands/arguments.py +++ b/freqtrade/commands/arguments.py @@ -73,6 +73,7 @@ ARGS_HYPEROPT_LIST = ["hyperopt_list_best", "hyperopt_list_profitable", "hyperopt_list_min_avg_time", "hyperopt_list_max_avg_time", "hyperopt_list_min_avg_profit", "hyperopt_list_max_avg_profit", "hyperopt_list_min_total_profit", "hyperopt_list_max_total_profit", + "hyperopt_list_min_objective", "hyperopt_list_max_objective", "print_colorized", "print_json", "hyperopt_list_no_details", "export_csv"] diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py index 3ed2f81d1..8eb5c3ce8 100644 --- a/freqtrade/commands/cli_options.py +++ b/freqtrade/commands/cli_options.py @@ -455,37 +455,49 @@ AVAILABLE_CLI_OPTIONS = { ), "hyperopt_list_min_avg_time": Arg( '--min-avg-time', - help='Select epochs on above average time.', + help='Select epochs above average time.', type=float, metavar='FLOAT', ), "hyperopt_list_max_avg_time": Arg( '--max-avg-time', - help='Select epochs on under average time.', + help='Select epochs below average time.', type=float, metavar='FLOAT', ), "hyperopt_list_min_avg_profit": Arg( '--min-avg-profit', - help='Select epochs on above average profit.', + help='Select epochs above average profit.', type=float, metavar='FLOAT', ), "hyperopt_list_max_avg_profit": Arg( '--max-avg-profit', - help='Select epochs on below average profit.', + help='Select epochs below average profit.', type=float, metavar='FLOAT', ), "hyperopt_list_min_total_profit": Arg( '--min-total-profit', - help='Select epochs on above total profit.', + help='Select epochs above total profit.', type=float, metavar='FLOAT', ), "hyperopt_list_max_total_profit": Arg( '--max-total-profit', - help='Select epochs on below total profit.', + help='Select epochs below total profit.', + type=float, + metavar='FLOAT', + ), + "hyperopt_list_min_objective": Arg( + '--min-objective', + help='Select epochs above objective.', + type=float, + metavar='FLOAT', + ), + "hyperopt_list_max_objective": Arg( + '--max-objective', + help='Select epochs below objective.', type=float, metavar='FLOAT', ), diff --git a/freqtrade/commands/hyperopt_commands.py b/freqtrade/commands/hyperopt_commands.py index 517f47d16..4fae51e28 100755 --- a/freqtrade/commands/hyperopt_commands.py +++ b/freqtrade/commands/hyperopt_commands.py @@ -35,7 +35,9 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None: 'filter_min_avg_profit': config.get('hyperopt_list_min_avg_profit', None), 'filter_max_avg_profit': config.get('hyperopt_list_max_avg_profit', None), 'filter_min_total_profit': config.get('hyperopt_list_min_total_profit', None), - 'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None) + 'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None), + 'filter_min_objective': config.get('hyperopt_list_min_objective', None), + 'filter_max_objective': config.get('hyperopt_list_max_objective', None), } results_file = (config['user_data_dir'] / @@ -45,7 +47,7 @@ def start_hyperopt_list(args: Dict[str, Any]) -> None: epochs = Hyperopt.load_previous_results(results_file) total_epochs = len(epochs) - epochs = _hyperopt_filter_epochs(epochs, filteroptions) + epochs = hyperopt_filter_epochs(epochs, filteroptions) if print_colorized: colorama_init(autoreset=True) @@ -92,14 +94,16 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None: 'filter_min_avg_profit': config.get('hyperopt_list_min_avg_profit', None), 'filter_max_avg_profit': config.get('hyperopt_list_max_avg_profit', None), 'filter_min_total_profit': config.get('hyperopt_list_min_total_profit', None), - 'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None) + 'filter_max_total_profit': config.get('hyperopt_list_max_total_profit', None), + 'filter_min_objective': config.get('hyperopt_list_min_objective', None), + 'filter_max_objective': config.get('hyperopt_list_max_objective', None) } # Previous evaluations epochs = Hyperopt.load_previous_results(results_file) total_epochs = len(epochs) - epochs = _hyperopt_filter_epochs(epochs, filteroptions) + epochs = hyperopt_filter_epochs(epochs, filteroptions) filtered_epochs = len(epochs) if n > filtered_epochs: @@ -119,7 +123,7 @@ def start_hyperopt_show(args: Dict[str, Any]) -> None: header_str="Epoch details") -def _hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: +def hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: """ Filter our items from the list of hyperopt results """ @@ -127,6 +131,24 @@ def _hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: epochs = [x for x in epochs if x['is_best']] if filteroptions['only_profitable']: epochs = [x for x in epochs if x['results_metrics']['profit'] > 0] + + epochs = _hyperopt_filter_epochs_trade_count(epochs, filteroptions) + + epochs = _hyperopt_filter_epochs_duration(epochs, filteroptions) + + epochs = _hyperopt_filter_epochs_profit(epochs, filteroptions) + + epochs = _hyperopt_filter_epochs_objective(epochs, filteroptions) + + logger.info(f"{len(epochs)} " + + ("best " if filteroptions['only_best'] else "") + + ("profitable " if filteroptions['only_profitable'] else "") + + "epochs found.") + return epochs + + +def _hyperopt_filter_epochs_trade_count(epochs: List, filteroptions: dict) -> List: + if filteroptions['filter_min_trades'] > 0: epochs = [ x for x in epochs @@ -137,6 +159,11 @@ def _hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: x for x in epochs if x['results_metrics']['trade_count'] < filteroptions['filter_max_trades'] ] + return epochs + + +def _hyperopt_filter_epochs_duration(epochs: List, filteroptions: dict) -> List: + if filteroptions['filter_min_avg_time'] is not None: epochs = [x for x in epochs if x['results_metrics']['trade_count'] > 0] epochs = [ @@ -149,6 +176,12 @@ def _hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: x for x in epochs if x['results_metrics']['duration'] < filteroptions['filter_max_avg_time'] ] + + return epochs + + +def _hyperopt_filter_epochs_profit(epochs: List, filteroptions: dict) -> List: + if filteroptions['filter_min_avg_profit'] is not None: epochs = [x for x in epochs if x['results_metrics']['trade_count'] > 0] epochs = [ @@ -173,10 +206,18 @@ def _hyperopt_filter_epochs(epochs: List, filteroptions: dict) -> List: x for x in epochs if x['results_metrics']['profit'] < filteroptions['filter_max_total_profit'] ] + return epochs - logger.info(f"{len(epochs)} " + - ("best " if filteroptions['only_best'] else "") + - ("profitable " if filteroptions['only_profitable'] else "") + - "epochs found.") + +def _hyperopt_filter_epochs_objective(epochs: List, filteroptions: dict) -> List: + + if filteroptions['filter_min_objective'] is not None: + epochs = [x for x in epochs if x['results_metrics']['trade_count'] > 0] + + epochs = [x for x in epochs if x['loss'] < filteroptions['filter_min_objective']] + if filteroptions['filter_max_objective'] is not None: + epochs = [x for x in epochs if x['results_metrics']['trade_count'] > 0] + + epochs = [x for x in epochs if x['loss'] > filteroptions['filter_max_objective']] return epochs diff --git a/freqtrade/commands/list_commands.py b/freqtrade/commands/list_commands.py index b29aabe25..c8c820c61 100644 --- a/freqtrade/commands/list_commands.py +++ b/freqtrade/commands/list_commands.py @@ -14,7 +14,7 @@ from freqtrade.configuration import setup_utils_configuration from freqtrade.constants import USERPATH_HYPEROPTS, USERPATH_STRATEGIES from freqtrade.exceptions import OperationalException from freqtrade.exchange import (available_exchanges, ccxt_exchanges, - market_is_active, symbol_is_pair) + market_is_active) from freqtrade.misc import plural from freqtrade.resolvers import ExchangeResolver, StrategyResolver from freqtrade.state import RunMode @@ -163,7 +163,7 @@ def start_list_markets(args: Dict[str, Any], pairs_only: bool = False) -> None: tabular_data.append({'Id': v['id'], 'Symbol': v['symbol'], 'Base': v['base'], 'Quote': v['quote'], 'Active': market_is_active(v), - **({'Is pair': symbol_is_pair(v['symbol'])} + **({'Is pair': exchange.market_is_tradable(v)} if not pairs_only else {})}) if (args.get('print_one_column', False) or diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py index 139e42084..08a600176 100644 --- a/freqtrade/configuration/configuration.py +++ b/freqtrade/configuration/configuration.py @@ -334,6 +334,12 @@ class Configuration: self._args_to_config(config, argname='hyperopt_list_max_total_profit', logstring='Parameter --max-total-profit detected: {}') + self._args_to_config(config, argname='hyperopt_list_min_objective', + logstring='Parameter --min-objective detected: {}') + + self._args_to_config(config, argname='hyperopt_list_max_objective', + logstring='Parameter --max-objective detected: {}') + self._args_to_config(config, argname='hyperopt_list_no_details', logstring='Parameter --no-details detected: {}') diff --git a/freqtrade/edge/edge_positioning.py b/freqtrade/edge/edge_positioning.py index 41252ee51..1993eded3 100644 --- a/freqtrade/edge/edge_positioning.py +++ b/freqtrade/edge/edge_positioning.py @@ -281,8 +281,8 @@ class Edge: # # Removing Pumps if self.edge_config.get('remove_pumps', False): - results = results.groupby(['pair', 'stoploss']).apply( - lambda x: x[x['profit_abs'] < 2 * x['profit_abs'].std() + x['profit_abs'].mean()]) + results = results[results['profit_abs'] < 2 * results['profit_abs'].std() + + results['profit_abs'].mean()] ########################################################################## # Removing trades having a duration more than X minutes (set in config) diff --git a/freqtrade/exceptions.py b/freqtrade/exceptions.py index c85fccc4b..e2bc969a9 100644 --- a/freqtrade/exceptions.py +++ b/freqtrade/exceptions.py @@ -29,7 +29,14 @@ class PricingError(DependencyException): """ -class InvalidOrderException(FreqtradeException): +class ExchangeError(DependencyException): + """ + Error raised out of the exchange. + Has multiple Errors to determine the appropriate error. + """ + + +class InvalidOrderException(ExchangeError): """ This is returned when the order is not valid. Example: If stoploss on exchange order is hit, then trying to cancel the order @@ -44,13 +51,6 @@ class RetryableOrderError(InvalidOrderException): """ -class ExchangeError(DependencyException): - """ - Error raised out of the exchange. - Has multiple Errors to determine the appropriate error. - """ - - class TemporaryError(ExchangeError): """ Temporary network or exchange related error. diff --git a/freqtrade/exchange/__init__.py b/freqtrade/exchange/__init__.py index a39f8f5df..bdf1f91ec 100644 --- a/freqtrade/exchange/__init__.py +++ b/freqtrade/exchange/__init__.py @@ -12,8 +12,7 @@ from freqtrade.exchange.exchange import (timeframe_to_seconds, timeframe_to_msecs, timeframe_to_next_date, timeframe_to_prev_date) -from freqtrade.exchange.exchange import (market_is_active, - symbol_is_pair) +from freqtrade.exchange.exchange import (market_is_active) from freqtrade.exchange.kraken import Kraken from freqtrade.exchange.binance import Binance from freqtrade.exchange.bibox import Bibox diff --git a/freqtrade/exchange/common.py b/freqtrade/exchange/common.py index 0610e8447..3bba9be72 100644 --- a/freqtrade/exchange/common.py +++ b/freqtrade/exchange/common.py @@ -107,12 +107,12 @@ def retrier_async(f): except TemporaryError as ex: logger.warning('%s() returned exception: "%s"', f.__name__, ex) if count > 0: + logger.warning('retrying %s() still for %s times', f.__name__, count) count -= 1 kwargs.update({'count': count}) - logger.warning('retrying %s() still for %s times', f.__name__, count) if isinstance(ex, DDosProtection): backoff_delay = calculate_backoff(count + 1, API_RETRY_COUNT) - logger.debug(f"Applying DDosProtection backoff delay: {backoff_delay}") + logger.info(f"Applying DDosProtection backoff delay: {backoff_delay}") await asyncio.sleep(backoff_delay) return await wrapper(*args, **kwargs) else: @@ -131,13 +131,13 @@ def retrier(_func=None, retries=API_RETRY_COUNT): except (TemporaryError, RetryableOrderError) as ex: logger.warning('%s() returned exception: "%s"', f.__name__, ex) if count > 0: + logger.warning('retrying %s() still for %s times', f.__name__, count) count -= 1 kwargs.update({'count': count}) - logger.warning('retrying %s() still for %s times', f.__name__, count) if isinstance(ex, DDosProtection) or isinstance(ex, RetryableOrderError): # increasing backoff backoff_delay = calculate_backoff(count + 1, retries) - logger.debug(f"Applying DDosProtection backoff delay: {backoff_delay}") + logger.info(f"Applying DDosProtection backoff delay: {backoff_delay}") time.sleep(backoff_delay) return wrapper(*args, **kwargs) else: diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py index 04ad10a68..c9c5a0027 100644 --- a/freqtrade/exchange/exchange.py +++ b/freqtrade/exchange/exchange.py @@ -24,7 +24,7 @@ from freqtrade.exceptions import (DDosProtection, ExchangeError, InvalidOrderException, OperationalException, RetryableOrderError, TemporaryError) from freqtrade.exchange.common import BAD_EXCHANGES, retrier, retrier_async -from freqtrade.misc import deep_merge_dicts, safe_value_fallback +from freqtrade.misc import deep_merge_dicts, safe_value_fallback2 CcxtModuleType = Any @@ -222,7 +222,7 @@ class Exchange: if quote_currencies: markets = {k: v for k, v in markets.items() if v['quote'] in quote_currencies} if pairs_only: - markets = {k: v for k, v in markets.items() if symbol_is_pair(v['symbol'])} + markets = {k: v for k, v in markets.items() if self.market_is_tradable(v)} if active_only: markets = {k: v for k, v in markets.items() if market_is_active(v)} return markets @@ -246,6 +246,19 @@ class Exchange: """ return self.markets.get(pair, {}).get('base', '') + def market_is_tradable(self, market: Dict[str, Any]) -> bool: + """ + Check if the market symbol is tradable by Freqtrade. + By default, checks if it's splittable by `/` and both sides correspond to base / quote + """ + symbol_parts = market['symbol'].split('/') + return (len(symbol_parts) == 2 and + len(symbol_parts[0]) > 0 and + len(symbol_parts[1]) > 0 and + symbol_parts[0] == market.get('base') and + symbol_parts[1] == market.get('quote') + ) + def klines(self, pair_interval: Tuple[str, str], copy: bool = True) -> DataFrame: if pair_interval in self._klines: return self._klines[pair_interval].copy() if copy else self._klines[pair_interval] @@ -258,8 +271,8 @@ class Exchange: api.urls['api'] = api.urls['test'] logger.info("Enabled Sandbox API on %s", name) else: - logger.warning(name, "No Sandbox URL in CCXT, exiting. " - "Please check your config.json") + logger.warning( + f"No Sandbox URL in CCXT for {name}, exiting. Please check your config.json") raise OperationalException(f'Exchange {name} does not provide a sandbox api') def _load_async_markets(self, reload: bool = False) -> None: @@ -480,6 +493,7 @@ class Exchange: "id": order_id, 'pair': pair, 'price': rate, + 'average': rate, 'amount': _amount, 'cost': _amount * rate, 'type': ordertype, @@ -974,7 +988,7 @@ class Exchange: except ccxt.BaseError as e: raise OperationalException(e) from e - # Assign method to fetch_stoploss_order to allow easy overriding in other classes + # Assign method to cancel_stoploss_order to allow easy overriding in other classes cancel_stoploss_order = cancel_order def is_cancel_order_result_suitable(self, corder) -> bool: @@ -999,7 +1013,7 @@ class Exchange: if self.is_cancel_order_result_suitable(corder): return corder except InvalidOrderException: - logger.warning(f"Could not cancel order {order_id}.") + logger.warning(f"Could not cancel order {order_id} for {pair}.") try: order = self.fetch_order(order_id, pair) except InvalidOrderException: @@ -1008,7 +1022,7 @@ class Exchange: return order - @retrier + @retrier(retries=5) def fetch_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: try: @@ -1022,10 +1036,10 @@ class Exchange: return self._api.fetch_order(order_id, pair) except ccxt.OrderNotFound as e: raise RetryableOrderError( - f'Order not found (id: {order_id}). Message: {e}') from e + f'Order not found (pair: {pair} id: {order_id}). Message: {e}') from e except ccxt.InvalidOrder as e: raise InvalidOrderException( - f'Tried to get an invalid order (id: {order_id}). Message: {e}') from e + f'Tried to get an invalid order (pair: {pair} id: {order_id}). Message: {e}') from e except ccxt.DDoSProtection as e: raise DDosProtection(e) from e except (ccxt.NetworkError, ccxt.ExchangeError) as e: @@ -1040,10 +1054,10 @@ class Exchange: @retrier def fetch_l2_order_book(self, pair: str, limit: int = 100) -> dict: """ - get order book level 2 from exchange - - Notes: - 20180619: bittrex doesnt support limits -.- + Get L2 order book from exchange. + Can be limited to a certain amount (if supported). + Returns a dict in the format + {'asks': [price, volume], 'bids': [price, volume]} """ try: @@ -1144,7 +1158,7 @@ class Exchange: if fee_curr in self.get_pair_base_currency(order['symbol']): # Base currency - divide by amount return round( - order['fee']['cost'] / safe_value_fallback(order, order, 'filled', 'amount'), 8) + order['fee']['cost'] / safe_value_fallback2(order, order, 'filled', 'amount'), 8) elif fee_curr in self.get_pair_quote_currency(order['symbol']): # Quote currency - divide by cost return round(order['fee']['cost'] / order['cost'], 8) if order['cost'] else None @@ -1157,7 +1171,7 @@ class Exchange: comb = self.get_valid_pair_combination(fee_curr, self._config['stake_currency']) tick = self.fetch_ticker(comb) - fee_to_quote_rate = safe_value_fallback(tick, tick, 'last', 'ask') + fee_to_quote_rate = safe_value_fallback2(tick, tick, 'last', 'ask') return round((order['fee']['cost'] * fee_to_quote_rate) / order['cost'], 8) except ExchangeError: return None @@ -1172,7 +1186,6 @@ class Exchange: return (order['fee']['cost'], order['fee']['currency'], self.calculate_fee_rate(order)) - # calculate rate ? (order['fee']['cost'] / (order['amount'] * order['price'])) def is_exchange_bad(exchange_name: str) -> bool: @@ -1258,20 +1271,6 @@ def timeframe_to_next_date(timeframe: str, date: datetime = None) -> datetime: return datetime.fromtimestamp(new_timestamp, tz=timezone.utc) -def symbol_is_pair(market_symbol: str, base_currency: str = None, - quote_currency: str = None) -> bool: - """ - Check if the market symbol is a pair, i.e. that its symbol consists of the base currency and the - quote currency separated by '/' character. If base_currency and/or quote_currency is passed, - it also checks that the symbol contains appropriate base and/or quote currency part before - and after the separating character correspondingly. - """ - symbol_parts = market_symbol.split('/') - return (len(symbol_parts) == 2 and - (symbol_parts[0] == base_currency if base_currency else len(symbol_parts[0]) > 0) and - (symbol_parts[1] == quote_currency if quote_currency else len(symbol_parts[1]) > 0)) - - def market_is_active(market: Dict) -> bool: """ Return True if the market is active. diff --git a/freqtrade/exchange/ftx.py b/freqtrade/exchange/ftx.py index b75f77ca4..441d97215 100644 --- a/freqtrade/exchange/ftx.py +++ b/freqtrade/exchange/ftx.py @@ -1,6 +1,6 @@ """ FTX exchange subclass """ import logging -from typing import Dict +from typing import Any, Dict import ccxt @@ -20,6 +20,16 @@ class Ftx(Exchange): "ohlcv_candle_limit": 1500, } + def market_is_tradable(self, market: Dict[str, Any]) -> bool: + """ + Check if the market symbol is tradable by Freqtrade. + Default checks + check if pair is spot pair (no futures trading yet). + """ + parent_check = super().market_is_tradable(market) + + return (parent_check and + market.get('spot', False) is True) + def stoploss_adjust(self, stop_loss: float, order: Dict) -> bool: """ Verify stop_loss against stoploss-order value (limit or price) @@ -78,7 +88,7 @@ class Ftx(Exchange): except ccxt.BaseError as e: raise OperationalException(e) from e - @retrier + @retrier(retries=5) def fetch_stoploss_order(self, order_id: str, pair: str) -> Dict: if self._config['dry_run']: try: diff --git a/freqtrade/exchange/kraken.py b/freqtrade/exchange/kraken.py index 7b9d0f09b..52b992dcc 100644 --- a/freqtrade/exchange/kraken.py +++ b/freqtrade/exchange/kraken.py @@ -1,6 +1,6 @@ """ Kraken exchange subclass """ import logging -from typing import Dict +from typing import Any, Dict import ccxt @@ -22,6 +22,16 @@ class Kraken(Exchange): "trades_pagination_arg": "since", } + def market_is_tradable(self, market: Dict[str, Any]) -> bool: + """ + Check if the market symbol is tradable by Freqtrade. + Default checks + check if pair is darkpool pair. + """ + parent_check = super().market_is_tradable(market) + + return (parent_check and + market.get('darkpool', False) is False) + @retrier def get_balances(self) -> dict: if self._config['dry_run']: diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py index a6d96ef77..2a95f58fc 100644 --- a/freqtrade/freqtradebot.py +++ b/freqtrade/freqtradebot.py @@ -20,7 +20,7 @@ from freqtrade.edge import Edge from freqtrade.exceptions import (DependencyException, ExchangeError, InvalidOrderException, PricingError) from freqtrade.exchange import timeframe_to_minutes, timeframe_to_next_date -from freqtrade.misc import safe_value_fallback +from freqtrade.misc import safe_value_fallback, safe_value_fallback2 from freqtrade.pairlist.pairlistmanager import PairListManager from freqtrade.persistence import Trade from freqtrade.resolvers import ExchangeResolver, StrategyResolver @@ -523,7 +523,7 @@ class FreqtradeBot: time_in_force=time_in_force): logger.info(f"User requested abortion of buying {pair}") return False - + amount = self.exchange.amount_to_precision(pair, amount) order = self.exchange.buy(pair=pair, ordertype=order_type, amount=amount, rate=buy_limit_requested, time_in_force=time_in_force) @@ -532,6 +532,7 @@ class FreqtradeBot: # we assume the order is executed at the price requested buy_limit_filled_price = buy_limit_requested + amount_requested = amount if order_status == 'expired' or order_status == 'rejected': order_tif = self.strategy.order_time_in_force['buy'] @@ -552,15 +553,15 @@ class FreqtradeBot: order['filled'], order['amount'], order['remaining'] ) stake_amount = order['cost'] - amount = order['amount'] - buy_limit_filled_price = order['price'] + amount = safe_value_fallback(order, 'filled', 'amount') + buy_limit_filled_price = safe_value_fallback(order, 'average', 'price') order_id = None # in case of FOK the order may be filled immediately and fully elif order_status == 'closed': stake_amount = order['cost'] - amount = order['amount'] - buy_limit_filled_price = order['price'] + amount = safe_value_fallback(order, 'filled', 'amount') + buy_limit_filled_price = safe_value_fallback(order, 'average', 'price') # Fee is applied twice because we make a LIMIT_BUY and LIMIT_SELL fee = self.exchange.get_fee(symbol=pair, taker_or_maker='maker') @@ -568,6 +569,7 @@ class FreqtradeBot: pair=pair, stake_amount=stake_amount, amount=amount, + amount_requested=amount_requested, fee_open=fee, fee_close=fee, open_rate=buy_limit_filled_price, @@ -660,7 +662,7 @@ class FreqtradeBot: trades_closed += 1 except DependencyException as exception: - logger.warning('Unable to sell trade: %s', exception) + logger.warning('Unable to sell trade %s: %s', trade.pair, exception) # Updating wallets if any trade occured if trades_closed: @@ -768,7 +770,7 @@ class FreqtradeBot: logger.debug('Found no sell signal for %s.', trade) return False - def create_stoploss_order(self, trade: Trade, stop_price: float, rate: float) -> bool: + def create_stoploss_order(self, trade: Trade, stop_price: float) -> bool: """ Abstracts creating stoploss orders from the logic. Handles errors and updates the trade database object. @@ -831,14 +833,13 @@ class FreqtradeBot: stoploss = self.edge.stoploss(pair=trade.pair) if self.edge else self.strategy.stoploss stop_price = trade.open_rate * (1 + stoploss) - if self.create_stoploss_order(trade=trade, stop_price=stop_price, rate=stop_price): + if self.create_stoploss_order(trade=trade, stop_price=stop_price): trade.stoploss_last_update = datetime.now() return False # If stoploss order is canceled for some reason we add it if stoploss_order and stoploss_order['status'] in ('canceled', 'cancelled'): - if self.create_stoploss_order(trade=trade, stop_price=trade.stop_loss, - rate=trade.stop_loss): + if self.create_stoploss_order(trade=trade, stop_price=trade.stop_loss): return False else: trade.stoploss_order_id = None @@ -875,8 +876,7 @@ class FreqtradeBot: f"for pair {trade.pair}") # Create new stoploss order - if not self.create_stoploss_order(trade=trade, stop_price=trade.stop_loss, - rate=trade.stop_loss): + if not self.create_stoploss_order(trade=trade, stop_price=trade.stop_loss): logger.warning(f"Could not create trailing stoploss order " f"for pair {trade.pair}.") @@ -921,7 +921,7 @@ class FreqtradeBot: if not trade.open_order_id: continue order = self.exchange.fetch_order(trade.open_order_id, trade.pair) - except (ExchangeError, InvalidOrderException): + except (ExchangeError): logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) continue @@ -954,7 +954,7 @@ class FreqtradeBot: for trade in Trade.get_open_order_trades(): try: order = self.exchange.fetch_order(trade.open_order_id, trade.pair) - except (DependencyException, InvalidOrderException): + except (ExchangeError): logger.info('Cannot query order for %s due to %s', trade, traceback.format_exc()) continue @@ -976,6 +976,12 @@ class FreqtradeBot: reason = constants.CANCEL_REASON['TIMEOUT'] corder = self.exchange.cancel_order_with_result(trade.open_order_id, trade.pair, trade.amount) + # Avoid race condition where the order could not be cancelled coz its already filled. + # Simply bailing here is the only safe way - as this order will then be + # handled in the next iteration. + if corder.get('status') not in ('canceled', 'closed'): + logger.warning(f"Order {trade.open_order_id} for {trade.pair} not cancelled.") + return False else: # Order was cancelled already, so we can reuse the existing dict corder = order @@ -984,7 +990,7 @@ class FreqtradeBot: logger.info('Buy order %s for %s.', reason, trade) # Using filled to determine the filled amount - filled_amount = safe_value_fallback(corder, order, 'filled', 'filled') + filled_amount = safe_value_fallback2(corder, order, 'filled', 'filled') if isclose(filled_amount, 0.0, abs_tol=constants.MATH_CLOSE_PREC): logger.info('Buy order fully cancelled. Removing %s from database.', trade) @@ -1249,7 +1255,8 @@ class FreqtradeBot: # Try update amount (binance-fix) try: new_amount = self.get_real_amount(trade, order, order_amount) - if not isclose(order['amount'], new_amount, abs_tol=constants.MATH_CLOSE_PREC): + if not isclose(safe_value_fallback(order, 'filled', 'amount'), new_amount, + abs_tol=constants.MATH_CLOSE_PREC): order['amount'] = new_amount order.pop('filled', None) trade.recalc_open_trade_price() @@ -1295,7 +1302,7 @@ class FreqtradeBot: """ # Init variables if order_amount is None: - order_amount = order['amount'] + order_amount = safe_value_fallback(order, 'filled', 'amount') # Only run for closed orders if trade.fee_updated(order.get('side', '')) or order['status'] == 'open': return order_amount diff --git a/freqtrade/misc.py b/freqtrade/misc.py index ac6084eb7..623f6cb8f 100644 --- a/freqtrade/misc.py +++ b/freqtrade/misc.py @@ -134,7 +134,21 @@ def round_dict(d, n): return {k: (round(v, n) if isinstance(v, float) else v) for k, v in d.items()} -def safe_value_fallback(dict1: dict, dict2: dict, key1: str, key2: str, default_value=None): +def safe_value_fallback(obj: dict, key1: str, key2: str, default_value=None): + """ + Search a value in obj, return this if it's not None. + Then search key2 in obj - return that if it's not none - then use default_value. + Else falls back to None. + """ + if key1 in obj and obj[key1] is not None: + return obj[key1] + else: + if key2 in obj and obj[key2] is not None: + return obj[key2] + return default_value + + +def safe_value_fallback2(dict1: dict, dict2: dict, key1: str, key2: str, default_value=None): """ Search a value in dict1, return this if it's not None. Fall back to dict2 - return key2 from dict2 if it's not None. diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py index 153ae3861..6d11e543b 100644 --- a/freqtrade/optimize/hyperopt.py +++ b/freqtrade/optimize/hyperopt.py @@ -312,11 +312,16 @@ class Hyperopt: trials = json_normalize(results, max_level=1) trials['Best'] = '' + if 'results_metrics.winsdrawslosses' not in trials.columns: + # Ensure compatibility with older versions of hyperopt results + trials['results_metrics.winsdrawslosses'] = 'N/A' + trials = trials[['Best', 'current_epoch', 'results_metrics.trade_count', + 'results_metrics.winsdrawslosses', 'results_metrics.avg_profit', 'results_metrics.total_profit', 'results_metrics.profit', 'results_metrics.duration', 'loss', 'is_initial_point', 'is_best']] - trials.columns = ['Best', 'Epoch', 'Trades', 'Avg profit', 'Total profit', + trials.columns = ['Best', 'Epoch', 'Trades', 'W/D/L', 'Avg profit', 'Total profit', 'Profit', 'Avg duration', 'Objective', 'is_initial_point', 'is_best'] trials['is_profit'] = False trials.loc[trials['is_initial_point'], 'Best'] = '* ' @@ -558,9 +563,17 @@ class Hyperopt: } def _calculate_results_metrics(self, backtesting_results: DataFrame) -> Dict: + wins = len(backtesting_results[backtesting_results.profit_percent > 0]) + draws = len(backtesting_results[backtesting_results.profit_percent == 0]) + losses = len(backtesting_results[backtesting_results.profit_percent < 0]) return { 'trade_count': len(backtesting_results.index), + 'wins': wins, + 'draws': draws, + 'losses': losses, + 'winsdrawslosses': f"{wins}/{draws}/{losses}", 'avg_profit': backtesting_results.profit_percent.mean() * 100.0, + 'median_profit': backtesting_results.profit_percent.median() * 100.0, 'total_profit': backtesting_results.profit_abs.sum(), 'profit': backtesting_results.profit_percent.sum() * 100.0, 'duration': backtesting_results.trade_duration.mean(), @@ -572,7 +585,10 @@ class Hyperopt: """ stake_cur = self.config['stake_currency'] return (f"{results_metrics['trade_count']:6d} trades. " + f"{results_metrics['wins']}/{results_metrics['draws']}" + f"/{results_metrics['losses']} Wins/Draws/Losses. " f"Avg profit {results_metrics['avg_profit']: 6.2f}%. " + f"Median profit {results_metrics['median_profit']: 6.2f}%. " f"Total profit {results_metrics['total_profit']: 11.8f} {stake_cur} " f"({results_metrics['profit']: 7.2f}\N{GREEK CAPITAL LETTER SIGMA}%). " f"Avg duration {results_metrics['duration']:5.1f} min." diff --git a/freqtrade/pairlist/IPairList.py b/freqtrade/pairlist/IPairList.py index 1cca00eba..67a96cc60 100644 --- a/freqtrade/pairlist/IPairList.py +++ b/freqtrade/pairlist/IPairList.py @@ -162,6 +162,11 @@ class IPairList(ABC): f"{self._exchange.name}. Removing it from whitelist..") continue + if not self._exchange.market_is_tradable(markets[pair]): + logger.warning(f"Pair {pair} is not tradable with Freqtrade." + "Removing it from whitelist..") + continue + if self._exchange.get_pair_quote_currency(pair) != self._config['stake_currency']: logger.warning(f"Pair {pair} is not compatible with your stake currency " f"{self._config['stake_currency']}. Removing it from whitelist..") diff --git a/freqtrade/persistence.py b/freqtrade/persistence.py index a6c1de402..28753ed48 100644 --- a/freqtrade/persistence.py +++ b/freqtrade/persistence.py @@ -17,6 +17,7 @@ from sqlalchemy.orm.session import sessionmaker from sqlalchemy.pool import StaticPool from freqtrade.exceptions import OperationalException +from freqtrade.misc import safe_value_fallback logger = logging.getLogger(__name__) @@ -86,7 +87,7 @@ def check_migrate(engine) -> None: logger.debug(f'trying {table_back_name}') # Check for latest column - if not has_column(cols, 'timeframe'): + if not has_column(cols, 'amount_requested'): logger.info(f'Running database migration - backup available as {table_back_name}') fee_open = get_column_def(cols, 'fee_open', 'fee') @@ -119,6 +120,7 @@ def check_migrate(engine) -> None: cols, 'close_profit_abs', f"(amount * close_rate * (1 - {fee_close})) - {open_trade_price}") sell_order_status = get_column_def(cols, 'sell_order_status', 'null') + amount_requested = get_column_def(cols, 'amount_requested', 'amount') # Schema migration necessary engine.execute(f"alter table trades rename to {table_back_name}") @@ -134,7 +136,7 @@ def check_migrate(engine) -> None: fee_open, fee_open_cost, fee_open_currency, fee_close, fee_close_cost, fee_open_currency, open_rate, open_rate_requested, close_rate, close_rate_requested, close_profit, - stake_amount, amount, open_date, close_date, open_order_id, + stake_amount, amount, amount_requested, open_date, close_date, open_order_id, stop_loss, stop_loss_pct, initial_stop_loss, initial_stop_loss_pct, stoploss_order_id, stoploss_last_update, max_rate, min_rate, sell_reason, sell_order_status, strategy, @@ -153,7 +155,7 @@ def check_migrate(engine) -> None: {fee_close_cost} fee_close_cost, {fee_close_currency} fee_close_currency, open_rate, {open_rate_requested} open_rate_requested, close_rate, {close_rate_requested} close_rate_requested, close_profit, - stake_amount, amount, open_date, close_date, open_order_id, + stake_amount, amount, {amount_requested}, open_date, close_date, open_order_id, {stop_loss} stop_loss, {stop_loss_pct} stop_loss_pct, {initial_stop_loss} initial_stop_loss, {initial_stop_loss_pct} initial_stop_loss_pct, @@ -215,6 +217,7 @@ class Trade(_DECL_BASE): close_profit_abs = Column(Float) stake_amount = Column(Float, nullable=False) amount = Column(Float) + amount_requested = Column(Float) open_date = Column(DateTime, nullable=False, default=datetime.utcnow) close_date = Column(DateTime) open_order_id = Column(String) @@ -256,6 +259,7 @@ class Trade(_DECL_BASE): 'is_open': self.is_open, 'exchange': self.exchange, 'amount': round(self.amount, 8), + 'amount_requested': round(self.amount_requested, 8) if self.amount_requested else None, 'stake_amount': round(self.stake_amount, 8), 'strategy': self.strategy, 'ticker_interval': self.timeframe, # DEPRECATED @@ -273,7 +277,7 @@ class Trade(_DECL_BASE): 'open_timestamp': int(self.open_date.timestamp() * 1000), 'open_rate': self.open_rate, 'open_rate_requested': self.open_rate_requested, - 'open_trade_price': self.open_trade_price, + 'open_trade_price': round(self.open_trade_price, 8), 'close_date_hum': (arrow.get(self.close_date).humanize() if self.close_date else None), @@ -365,20 +369,20 @@ class Trade(_DECL_BASE): """ order_type = order['type'] # Ignore open and cancelled orders - if order['status'] == 'open' or order['price'] is None: + if order['status'] == 'open' or safe_value_fallback(order, 'average', 'price') is None: return logger.info('Updating trade (id=%s) ...', self.id) if order_type in ('market', 'limit') and order['side'] == 'buy': # Update open rate and actual amount - self.open_rate = Decimal(order['price']) - self.amount = Decimal(order.get('filled', order['amount'])) + self.open_rate = Decimal(safe_value_fallback(order, 'average', 'price')) + self.amount = Decimal(safe_value_fallback(order, 'filled', 'amount')) self.recalc_open_trade_price() logger.info('%s_BUY has been fulfilled for %s.', order_type.upper(), self) self.open_order_id = None elif order_type in ('market', 'limit') and order['side'] == 'sell': - self.close(order['price']) + self.close(safe_value_fallback(order, 'average', 'price')) logger.info('%s_SELL has been fulfilled for %s.', order_type.upper(), self) elif order_type in ('stop_loss_limit', 'stop-loss', 'stop'): self.stoploss_order_id = None diff --git a/freqtrade/rpc/api_server.py b/freqtrade/rpc/api_server.py index 351842e10..06926ac35 100644 --- a/freqtrade/rpc/api_server.py +++ b/freqtrade/rpc/api_server.py @@ -56,7 +56,7 @@ def require_login(func: Callable[[Any, Any], Any]): # Type should really be Callable[[ApiServer], Any], but that will create a circular dependency -def rpc_catch_errors(func: Callable[[Any], Any]): +def rpc_catch_errors(func: Callable[..., Any]): def func_wrapper(obj, *args, **kwargs): @@ -200,6 +200,8 @@ class ApiServer(RPC): view_func=self._ping, methods=['GET']) self.app.add_url_rule(f'{BASE_URI}/trades', 'trades', view_func=self._trades, methods=['GET']) + self.app.add_url_rule(f'{BASE_URI}/trades/', 'trades_delete', + view_func=self._trades_delete, methods=['DELETE']) # Combined actions and infos self.app.add_url_rule(f'{BASE_URI}/blacklist', 'blacklist', view_func=self._blacklist, methods=['GET', 'POST']) @@ -424,6 +426,19 @@ class ApiServer(RPC): results = self._rpc_trade_history(limit) return self.rest_dump(results) + @require_login + @rpc_catch_errors + def _trades_delete(self, tradeid): + """ + Handler for DELETE /trades/ endpoint. + Removes the trade from the database (tries to cancel open orders first!) + get: + param: + tradeid: Numeric trade-id assigned to the trade. + """ + result = self._rpc_delete(tradeid) + return self.rest_dump(result) + @require_login @rpc_catch_errors def _whitelist(self): diff --git a/freqtrade/rpc/rpc.py b/freqtrade/rpc/rpc.py index 69faff533..f4e20c16f 100644 --- a/freqtrade/rpc/rpc.py +++ b/freqtrade/rpc/rpc.py @@ -6,14 +6,14 @@ from abc import abstractmethod from datetime import date, datetime, timedelta from enum import Enum from math import isnan -from typing import Any, Dict, List, Optional, Tuple +from typing import Any, Dict, List, Optional, Tuple, Union import arrow from numpy import NAN, mean -from freqtrade.exceptions import ExchangeError, PricingError - -from freqtrade.exchange import timeframe_to_msecs, timeframe_to_minutes +from freqtrade.exceptions import (ExchangeError, + PricingError) +from freqtrade.exchange import timeframe_to_minutes, timeframe_to_msecs from freqtrade.misc import shorten_date from freqtrade.persistence import Trade from freqtrade.rpc.fiat_convert import CryptoToFiatConverter @@ -252,9 +252,10 @@ class RPC: def _rpc_trade_history(self, limit: int) -> Dict: """ Returns the X last trades """ if limit > 0: - trades = Trade.get_trades().order_by(Trade.id.desc()).limit(limit) + trades = Trade.get_trades([Trade.is_open.is_(False)]).order_by( + Trade.id.desc()).limit(limit) else: - trades = Trade.get_trades().order_by(Trade.id.desc()).all() + trades = Trade.get_trades([Trade.is_open.is_(False)]).order_by(Trade.id.desc()).all() output = [trade.to_json() for trade in trades] @@ -537,6 +538,46 @@ class RPC: else: return None + def _rpc_delete(self, trade_id: str) -> Dict[str, Union[str, int]]: + """ + Handler for delete . + Delete the given trade and close eventually existing open orders. + """ + with self._freqtrade._sell_lock: + c_count = 0 + trade = Trade.get_trades(trade_filter=[Trade.id == trade_id]).first() + if not trade: + logger.warning('delete trade: Invalid argument received') + raise RPCException('invalid argument') + + # Try cancelling regular order if that exists + if trade.open_order_id: + try: + self._freqtrade.exchange.cancel_order(trade.open_order_id, trade.pair) + c_count += 1 + except (ExchangeError): + pass + + # cancel stoploss on exchange ... + if (self._freqtrade.strategy.order_types.get('stoploss_on_exchange') + and trade.stoploss_order_id): + try: + self._freqtrade.exchange.cancel_stoploss_order(trade.stoploss_order_id, + trade.pair) + c_count += 1 + except (ExchangeError): + pass + + Trade.session.delete(trade) + Trade.session.flush() + self._freqtrade.wallets.update() + return { + 'result': 'success', + 'trade_id': trade_id, + 'result_msg': f'Deleted trade {trade_id}. Closed {c_count} open orders.', + 'cancel_order_count': c_count, + } + def _rpc_performance(self) -> List[Dict[str, Any]]: """ Handler for performance. diff --git a/freqtrade/rpc/telegram.py b/freqtrade/rpc/telegram.py index 13cc1afaf..f1d3cde21 100644 --- a/freqtrade/rpc/telegram.py +++ b/freqtrade/rpc/telegram.py @@ -5,6 +5,7 @@ This module manage Telegram communication """ import json import logging +import arrow from typing import Any, Callable, Dict from tabulate import tabulate @@ -92,6 +93,8 @@ class Telegram(RPC): CommandHandler('stop', self._stop), CommandHandler('forcesell', self._forcesell), CommandHandler('forcebuy', self._forcebuy), + CommandHandler('trades', self._trades), + CommandHandler('delete', self._delete_trade), CommandHandler('performance', self._performance), CommandHandler('daily', self._daily), CommandHandler('count', self._count), @@ -496,6 +499,62 @@ class Telegram(RPC): except RPCException as e: self._send_msg(str(e)) + @authorized_only + def _trades(self, update: Update, context: CallbackContext) -> None: + """ + Handler for /trades + Returns last n recent trades. + :param bot: telegram bot + :param update: message update + :return: None + """ + stake_cur = self._config['stake_currency'] + try: + nrecent = int(context.args[0]) + except (TypeError, ValueError, IndexError): + nrecent = 10 + try: + trades = self._rpc_trade_history( + nrecent + ) + trades_tab = tabulate( + [[arrow.get(trade['open_date']).humanize(), + trade['pair'], + f"{(100 * trade['close_profit']):.2f}% ({trade['close_profit_abs']})"] + for trade in trades['trades']], + headers=[ + 'Open Date', + 'Pair', + f'Profit ({stake_cur})', + ], + tablefmt='simple') + message = (f"{min(trades['trades_count'], nrecent)} recent trades:\n" + + (f"
{trades_tab}
" if trades['trades_count'] > 0 else '')) + self._send_msg(message, parse_mode=ParseMode.HTML) + except RPCException as e: + self._send_msg(str(e)) + + @authorized_only + def _delete_trade(self, update: Update, context: CallbackContext) -> None: + """ + Handler for /delete . + Delete the given trade + :param bot: telegram bot + :param update: message update + :return: None + """ + + trade_id = context.args[0] if len(context.args) > 0 else None + try: + msg = self._rpc_delete(trade_id) + self._send_msg(( + '`{result_msg}`\n' + 'Please make sure to take care of this asset on the exchange manually.' + ).format(**msg)) + + except RPCException as e: + self._send_msg(str(e)) + @authorized_only def _performance(self, update: Update, context: CallbackContext) -> None: """ @@ -609,10 +668,12 @@ class Telegram(RPC): " *table :* `will display trades in a table`\n" " `pending buy orders are marked with an asterisk (*)`\n" " `pending sell orders are marked with a double asterisk (**)`\n" + "*/trades [limit]:* `Lists last closed trades (limited to 10 by default)`\n" "*/profit:* `Lists cumulative profit from all finished trades`\n" "*/forcesell |all:* `Instantly sells the given trade or all trades, " "regardless of profit`\n" f"{forcebuy_text if self._config.get('forcebuy_enable', False) else ''}" + "*/delete :* `Instantly delete the given trade in the database`\n" "*/performance:* `Show performance of each finished trade grouped by pair`\n" "*/daily :* `Shows profit or loss per day, over the last n days`\n" "*/count:* `Show number of trades running compared to allowed number of trades`" diff --git a/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 b/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 index c7ce41bb7..5ca6e6971 100644 --- a/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 +++ b/freqtrade/templates/subtemplates/strategy_methods_advanced.j2 @@ -34,7 +34,7 @@ def confirm_trade_entry(self, pair: str, order_type: str, amount: float, rate: f """ return True -def confirm_trade_exit(self, pair: str, trade: Trade, order_type: str, amount: float, +def confirm_trade_exit(self, pair: str, trade: 'Trade', order_type: str, amount: float, rate: float, time_in_force: str, sell_reason: str, **kwargs) -> bool: """ Called right before placing a regular sell order. diff --git a/requirements-common.txt b/requirements-common.txt index d5c5fd832..b7e71eada 100644 --- a/requirements-common.txt +++ b/requirements-common.txt @@ -1,12 +1,12 @@ # requirements without requirements installable via conda # mainly used for Raspberry pi installs -ccxt==1.31.37 +ccxt==1.32.88 SQLAlchemy==1.3.18 python-telegram-bot==12.8 -arrow==0.15.7 +arrow==0.15.8 cachetools==4.1.1 requests==2.24.0 -urllib3==1.25.9 +urllib3==1.25.10 wrapt==1.12.1 jsonschema==3.2.0 TA-Lib==0.4.18 diff --git a/requirements-dev.txt b/requirements-dev.txt index 9f9be638d..c02a439d3 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -8,7 +8,7 @@ flake8==3.8.3 flake8-type-annotations==0.1.0 flake8-tidy-imports==4.1.0 mypy==0.782 -pytest==5.4.3 +pytest==6.0.1 pytest-asyncio==0.14.0 pytest-cov==2.10.0 pytest-mock==3.2.0 diff --git a/requirements-hyperopt.txt b/requirements-hyperopt.txt index 4773d9877..ce08f08e0 100644 --- a/requirements-hyperopt.txt +++ b/requirements-hyperopt.txt @@ -2,7 +2,7 @@ -r requirements.txt # Required for hyperopt -scipy==1.5.1 +scipy==1.5.2 scikit-learn==0.23.1 scikit-optimize==0.7.4 filelock==3.0.12 diff --git a/requirements-plot.txt b/requirements-plot.txt index ec5af3dbf..51d14d636 100644 --- a/requirements-plot.txt +++ b/requirements-plot.txt @@ -1,5 +1,5 @@ # Include all requirements to run the bot. -r requirements.txt -plotly==4.8.2 +plotly==4.9.0 diff --git a/requirements.txt b/requirements.txt index 1e61d165f..d65f90325 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ # Load common requirements -r requirements-common.txt -numpy==1.19.0 -pandas==1.0.5 +numpy==1.19.1 +pandas==1.1.0 diff --git a/scripts/rest_client.py b/scripts/rest_client.py index 1f96bcb69..51ea596f6 100755 --- a/scripts/rest_client.py +++ b/scripts/rest_client.py @@ -62,6 +62,9 @@ class FtRestClient(): def _get(self, apipath, params: dict = None): return self._call("GET", apipath, params=params) + def _delete(self, apipath, params: dict = None): + return self._call("DELETE", apipath, params=params) + def _post(self, apipath, params: dict = None, data: dict = None): return self._call("POST", apipath, params=params, data=data) @@ -164,6 +167,15 @@ class FtRestClient(): """ return self._get("trades", params={"limit": limit} if limit else 0) + def delete_trade(self, trade_id): + """Delete trade from the database. + Tries to close open orders. Requires manual handling of this asset on the exchange. + + :param trade_id: Deletes the trade with this ID from the database. + :return: json object + """ + return self._delete("trades/{}".format(trade_id)) + def whitelist(self): """Show the current whitelist. diff --git a/tests/commands/test_commands.py b/tests/commands/test_commands.py index ffced956d..69d80d2cd 100644 --- a/tests/commands/test_commands.py +++ b/tests/commands/test_commands.py @@ -736,7 +736,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", - "--no-details" + "--no-details", ] pargs = get_args(args) pargs['config'] = None @@ -749,7 +749,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--best", - "--no-details" + "--no-details", ] pargs = get_args(args) pargs['config'] = None @@ -763,7 +763,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--profitable", - "--no-details" + "--no-details", ] pargs = get_args(args) pargs['config'] = None @@ -776,7 +776,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): " 11/12", " 12/12"]) args = [ "hyperopt-list", - "--profitable" + "--profitable", ] pargs = get_args(args) pargs['config'] = None @@ -792,7 +792,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): "hyperopt-list", "--no-details", "--no-color", - "--min-trades", "20" + "--min-trades", "20", ] pargs = get_args(args) pargs['config'] = None @@ -806,7 +806,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): "hyperopt-list", "--profitable", "--no-details", - "--max-trades", "20" + "--max-trades", "20", ] pargs = get_args(args) pargs['config'] = None @@ -821,7 +821,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): "hyperopt-list", "--profitable", "--no-details", - "--min-avg-profit", "0.11" + "--min-avg-profit", "0.11", ] pargs = get_args(args) pargs['config'] = None @@ -835,7 +835,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--no-details", - "--max-avg-profit", "0.10" + "--max-avg-profit", "0.10", ] pargs = get_args(args) pargs['config'] = None @@ -849,7 +849,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--no-details", - "--min-total-profit", "0.4" + "--min-total-profit", "0.4", ] pargs = get_args(args) pargs['config'] = None @@ -863,7 +863,35 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--no-details", - "--max-total-profit", "0.4" + "--max-total-profit", "0.4", + ] + pargs = get_args(args) + pargs['config'] = None + start_hyperopt_list(pargs) + captured = capsys.readouterr() + assert all(x in captured.out + for x in [" 1/12", " 2/12", " 3/12", " 5/12", " 6/12", " 7/12", " 8/12", + " 9/12", " 11/12"]) + assert all(x not in captured.out + for x in [" 4/12", " 10/12", " 12/12"]) + args = [ + "hyperopt-list", + "--no-details", + "--min-objective", "0.1", + ] + pargs = get_args(args) + pargs['config'] = None + start_hyperopt_list(pargs) + captured = capsys.readouterr() + assert all(x in captured.out + for x in [" 10/12"]) + assert all(x not in captured.out + for x in [" 1/12", " 2/12", " 3/12", " 4/12", " 5/12", " 6/12", " 7/12", " 8/12", + " 9/12", " 11/12", " 12/12"]) + args = [ + "hyperopt-list", + "--no-details", + "--max-objective", "0.1", ] pargs = get_args(args) pargs['config'] = None @@ -878,7 +906,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): "hyperopt-list", "--profitable", "--no-details", - "--min-avg-time", "2000" + "--min-avg-time", "2000", ] pargs = get_args(args) pargs['config'] = None @@ -892,7 +920,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--no-details", - "--max-avg-time", "1500" + "--max-avg-time", "1500", ] pargs = get_args(args) pargs['config'] = None @@ -906,7 +934,7 @@ def test_hyperopt_list(mocker, capsys, caplog, hyperopt_results): args = [ "hyperopt-list", "--no-details", - "--export-csv", "test_file.csv" + "--export-csv", "test_file.csv", ] pargs = get_args(args) pargs['config'] = None @@ -1089,7 +1117,7 @@ def test_show_trades(mocker, fee, capsys, caplog): pargs = get_args(args) pargs['config'] = None start_show_trades(pargs) - assert log_has("Printing 3 Trades: ", caplog) + assert log_has("Printing 4 Trades: ", caplog) captured = capsys.readouterr() assert "Trade(id=1" in captured.out assert "Trade(id=2" in captured.out diff --git a/tests/conftest.py b/tests/conftest.py index e2bdf7da5..1ac8256a8 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -176,6 +176,7 @@ def create_mock_trades(fee): pair='ETH/BTC', stake_amount=0.001, amount=123.0, + amount_requested=123.0, fee_open=fee.return_value, fee_close=fee.return_value, open_rate=0.123, @@ -188,6 +189,7 @@ def create_mock_trades(fee): pair='ETC/BTC', stake_amount=0.001, amount=123.0, + amount_requested=123.0, fee_open=fee.return_value, fee_close=fee.return_value, open_rate=0.123, @@ -199,11 +201,26 @@ def create_mock_trades(fee): ) Trade.session.add(trade) + trade = Trade( + pair='XRP/BTC', + stake_amount=0.001, + amount=123.0, + fee_open=fee.return_value, + fee_close=fee.return_value, + open_rate=0.05, + close_rate=0.06, + close_profit=0.01, + exchange='bittrex', + is_open=False, + ) + Trade.session.add(trade) + # Simulate prod entry trade = Trade( pair='ETC/BTC', stake_amount=0.001, amount=123.0, + amount_requested=124.0, fee_open=fee.return_value, fee_close=fee.return_value, open_rate=0.123, diff --git a/tests/data/test_btanalysis.py b/tests/data/test_btanalysis.py index b65db7fd8..718c02f05 100644 --- a/tests/data/test_btanalysis.py +++ b/tests/data/test_btanalysis.py @@ -43,7 +43,7 @@ def test_load_trades_from_db(default_conf, fee, mocker): trades = load_trades_from_db(db_url=default_conf['db_url']) assert init_mock.call_count == 1 - assert len(trades) == 3 + assert len(trades) == 4 assert isinstance(trades, DataFrame) assert "pair" in trades.columns assert "open_time" in trades.columns diff --git a/tests/edge/test_edge.py b/tests/edge/test_edge.py index cf9cb6fe1..7373778ad 100644 --- a/tests/edge/test_edge.py +++ b/tests/edge/test_edge.py @@ -409,3 +409,98 @@ def test_process_expectancy(mocker, edge_conf, fee, risk_reward_ratio, expectanc final = edge._process_expectancy(trades_df) assert len(final) == 0 assert isinstance(final, dict) + + +def test_process_expectancy_remove_pumps(mocker, edge_conf, fee,): + edge_conf['edge']['min_trade_number'] = 2 + edge_conf['edge']['remove_pumps'] = True + freqtrade = get_patched_freqtradebot(mocker, edge_conf) + + freqtrade.exchange.get_fee = fee + edge = Edge(edge_conf, freqtrade.exchange, freqtrade.strategy) + + trades = [ + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:05:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:10:00.000000000'), + 'open_index': 1, + 'close_index': 1, + 'trade_duration': '', + 'open_rate': 17, + 'close_rate': 15, + 'exit_type': 'sell_signal'}, + + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:20:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:25:00.000000000'), + 'open_index': 4, + 'close_index': 4, + 'trade_duration': '', + 'open_rate': 20, + 'close_rate': 10, + 'exit_type': 'sell_signal'}, + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:20:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:25:00.000000000'), + 'open_index': 4, + 'close_index': 4, + 'trade_duration': '', + 'open_rate': 20, + 'close_rate': 10, + 'exit_type': 'sell_signal'}, + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:20:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:25:00.000000000'), + 'open_index': 4, + 'close_index': 4, + 'trade_duration': '', + 'open_rate': 20, + 'close_rate': 10, + 'exit_type': 'sell_signal'}, + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:20:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:25:00.000000000'), + 'open_index': 4, + 'close_index': 4, + 'trade_duration': '', + 'open_rate': 20, + 'close_rate': 10, + 'exit_type': 'sell_signal'}, + + {'pair': 'TEST/BTC', + 'stoploss': -0.9, + 'profit_percent': '', + 'profit_abs': '', + 'open_time': np.datetime64('2018-10-03T00:30:00.000000000'), + 'close_time': np.datetime64('2018-10-03T00:40:00.000000000'), + 'open_index': 6, + 'close_index': 7, + 'trade_duration': '', + 'open_rate': 26, + 'close_rate': 134, + 'exit_type': 'sell_signal'} + ] + + trades_df = DataFrame(trades) + trades_df = edge._fill_calculable_fields(trades_df) + final = edge._process_expectancy(trades_df) + + assert 'TEST/BTC' in final + assert final['TEST/BTC'].stoploss == -0.9 + assert final['TEST/BTC'].nb_trades == len(trades_df) - 1 + assert round(final['TEST/BTC'].winrate, 10) == 0.0 diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py index 60c4847f6..571053b44 100644 --- a/tests/exchange/test_exchange.py +++ b/tests/exchange/test_exchange.py @@ -11,11 +11,12 @@ import ccxt import pytest from pandas import DataFrame -from freqtrade.exceptions import (DependencyException, InvalidOrderException, DDosProtection, - OperationalException, TemporaryError) +from freqtrade.exceptions import (DDosProtection, DependencyException, + InvalidOrderException, OperationalException, + TemporaryError) from freqtrade.exchange import Binance, Exchange, Kraken from freqtrade.exchange.common import API_RETRY_COUNT, calculate_backoff -from freqtrade.exchange.exchange import (market_is_active, symbol_is_pair, +from freqtrade.exchange.exchange import (market_is_active, timeframe_to_minutes, timeframe_to_msecs, timeframe_to_next_date, @@ -1818,7 +1819,7 @@ def test_cancel_order_with_result_error(default_conf, mocker, exchange_name, cap res = exchange.cancel_order_with_result('1234', 'ETH/BTC', 1541) assert isinstance(res, dict) - assert log_has("Could not cancel order 1234.", caplog) + assert log_has("Could not cancel order 1234 for ETH/BTC.", caplog) assert log_has("Could not fetch cancelled order 1234.", caplog) assert res['amount'] == 1541 @@ -1896,10 +1897,10 @@ def test_fetch_order(default_conf, mocker, exchange_name): assert tm.call_args_list[1][0][0] == 2 assert tm.call_args_list[2][0][0] == 5 assert tm.call_args_list[3][0][0] == 10 - assert api_mock.fetch_order.call_count == API_RETRY_COUNT + 1 + assert api_mock.fetch_order.call_count == 6 ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, - 'fetch_order', 'fetch_order', + 'fetch_order', 'fetch_order', retries=6, order_id='_', pair='TKN/BTC') @@ -1932,6 +1933,7 @@ def test_fetch_stoploss_order(default_conf, mocker, exchange_name): ccxt_exceptionhandlers(mocker, default_conf, api_mock, exchange_name, 'fetch_stoploss_order', 'fetch_order', + retries=6, order_id='_', pair='TKN/BTC') @@ -2217,25 +2219,42 @@ def test_timeframe_to_next_date(): assert timeframe_to_next_date("5m") > date -@pytest.mark.parametrize("market_symbol,base_currency,quote_currency,expected_result", [ - ("BTC/USDT", None, None, True), - ("USDT/BTC", None, None, True), - ("BTCUSDT", None, None, False), - ("BTC/USDT", None, "USDT", True), - ("USDT/BTC", None, "USDT", False), - ("BTCUSDT", None, "USDT", False), - ("BTC/USDT", "BTC", None, True), - ("USDT/BTC", "BTC", None, False), - ("BTCUSDT", "BTC", None, False), - ("BTC/USDT", "BTC", "USDT", True), - ("BTC/USDT", "USDT", "BTC", False), - ("BTC/USDT", "BTC", "USD", False), - ("BTCUSDT", "BTC", "USDT", False), - ("BTC/", None, None, False), - ("/USDT", None, None, False), +@pytest.mark.parametrize("market_symbol,base,quote,exchange,add_dict,expected_result", [ + ("BTC/USDT", 'BTC', 'USDT', "binance", {}, True), + ("USDT/BTC", 'USDT', 'BTC', "binance", {}, True), + ("USDT/BTC", 'BTC', 'USDT', "binance", {}, False), # Reversed currencies + ("BTCUSDT", 'BTC', 'USDT', "binance", {}, False), # No seperating / + ("BTCUSDT", None, "USDT", "binance", {}, False), # + ("USDT/BTC", "BTC", None, "binance", {}, False), + ("BTCUSDT", "BTC", None, "binance", {}, False), + ("BTC/USDT", "BTC", "USDT", "binance", {}, True), + ("BTC/USDT", "USDT", "BTC", "binance", {}, False), # reversed currencies + ("BTC/USDT", "BTC", "USD", "binance", {}, False), # Wrong quote currency + ("BTC/", "BTC", 'UNK', "binance", {}, False), + ("/USDT", 'UNK', 'USDT', "binance", {}, False), + ("BTC/EUR", 'BTC', 'EUR', "kraken", {"darkpool": False}, True), + ("EUR/BTC", 'EUR', 'BTC', "kraken", {"darkpool": False}, True), + ("EUR/BTC", 'BTC', 'EUR', "kraken", {"darkpool": False}, False), # Reversed currencies + ("BTC/EUR", 'BTC', 'USD', "kraken", {"darkpool": False}, False), # wrong quote currency + ("BTC/EUR", 'BTC', 'EUR', "kraken", {"darkpool": True}, False), # no darkpools + ("BTC/EUR.d", 'BTC', 'EUR', "kraken", {"darkpool": True}, False), # no darkpools + ("BTC/USD", 'BTC', 'USD', "ftx", {'spot': True}, True), + ("USD/BTC", 'USD', 'BTC', "ftx", {'spot': True}, True), + ("BTC/USD", 'BTC', 'USDT', "ftx", {'spot': True}, False), # Wrong quote currency + ("BTC/USD", 'USD', 'BTC', "ftx", {'spot': True}, False), # Reversed currencies + ("BTC/USD", 'BTC', 'USD', "ftx", {'spot': False}, False), # Can only trade spot markets + ("BTC-PERP", 'BTC', 'USD', "ftx", {'spot': False}, False), # Can only trade spot markets ]) -def test_symbol_is_pair(market_symbol, base_currency, quote_currency, expected_result) -> None: - assert symbol_is_pair(market_symbol, base_currency, quote_currency) == expected_result +def test_market_is_tradable(mocker, default_conf, market_symbol, base, + quote, add_dict, exchange, expected_result) -> None: + ex = get_patched_exchange(mocker, default_conf, id=exchange) + market = { + 'symbol': market_symbol, + 'base': base, + 'quote': quote, + **(add_dict), + } + assert ex.market_is_tradable(market) == expected_result @pytest.mark.parametrize("market,expected_result", [ @@ -2315,6 +2334,18 @@ def test_calculate_fee_rate(mocker, default_conf, order, expected) -> None: (3, 3, 1), (0, 1, 2), (1, 1, 1), + (0, 4, 17), + (1, 4, 10), + (2, 4, 5), + (3, 4, 2), + (4, 4, 1), + (0, 5, 26), + (1, 5, 17), + (2, 5, 10), + (3, 5, 5), + (4, 5, 2), + (5, 5, 1), + ]) def test_calculate_backoff(retrycount, max_retries, expected): assert calculate_backoff(retrycount, max_retries) == expected diff --git a/tests/exchange/test_ftx.py b/tests/exchange/test_ftx.py index eb7d83be3..bed92d276 100644 --- a/tests/exchange/test_ftx.py +++ b/tests/exchange/test_ftx.py @@ -154,4 +154,5 @@ def test_fetch_stoploss_order(default_conf, mocker): ccxt_exceptionhandlers(mocker, default_conf, api_mock, 'ftx', 'fetch_stoploss_order', 'fetch_orders', + retries=6, order_id='_', pair='TKN/BTC') diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py index 564725709..a6541f55b 100644 --- a/tests/optimize/test_hyperopt.py +++ b/tests/optimize/test_hyperopt.py @@ -744,8 +744,10 @@ def test_generate_optimizer(mocker, default_conf) -> None: } response_expected = { 'loss': 1.9840569076926293, - 'results_explanation': (' 1 trades. Avg profit 2.31%. Total profit 0.00023300 BTC ' - '( 2.31\N{GREEK CAPITAL LETTER SIGMA}%). Avg duration 100.0 min.' + 'results_explanation': (' 1 trades. 1/0/0 Wins/Draws/Losses. ' + 'Avg profit 2.31%. Median profit 2.31%. Total profit ' + '0.00023300 BTC ( 2.31\N{GREEK CAPITAL LETTER SIGMA}%). ' + 'Avg duration 100.0 min.' ).encode(locale.getpreferredencoding(), 'replace').decode('utf-8'), 'params_details': {'buy': {'adx-enabled': False, 'adx-value': 0, @@ -776,10 +778,15 @@ def test_generate_optimizer(mocker, default_conf) -> None: 'trailing_stop_positive_offset': 0.07}}, 'params_dict': optimizer_param, 'results_metrics': {'avg_profit': 2.3117, + 'draws': 0, 'duration': 100.0, + 'losses': 0, + 'winsdrawslosses': '1/0/0', + 'median_profit': 2.3117, 'profit': 2.3117, 'total_profit': 0.000233, - 'trade_count': 1}, + 'trade_count': 1, + 'wins': 1}, 'total_profit': 0.00023300 } diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py index c235367be..ad664abd2 100644 --- a/tests/pairlist/test_pairlist.py +++ b/tests/pairlist/test_pairlist.py @@ -468,7 +468,9 @@ def test_pairlist_class(mocker, whitelist_conf, markets, pairlist): # BCH/BTC not available (['ETH/BTC', 'TKN/BTC', 'BCH/BTC'], "is not compatible with exchange"), # BTT/BTC is inactive - (['ETH/BTC', 'TKN/BTC', 'BTT/BTC'], "Market is not active") + (['ETH/BTC', 'TKN/BTC', 'BTT/BTC'], "Market is not active"), + # XLTCUSDT is not a valid pair + (['ETH/BTC', 'TKN/BTC', 'XLTCUSDT'], "is not tradable with Freqtrade"), ]) def test__whitelist_for_active_markets(mocker, whitelist_conf, markets, pairlist, whitelist, caplog, log_message, tickers): diff --git a/tests/rpc/test_rpc.py b/tests/rpc/test_rpc.py index de9327ba9..9bbd34672 100644 --- a/tests/rpc/test_rpc.py +++ b/tests/rpc/test_rpc.py @@ -8,7 +8,7 @@ import pytest from numpy import isnan from freqtrade.edge import PairInfo -from freqtrade.exceptions import ExchangeError, TemporaryError +from freqtrade.exceptions import ExchangeError, InvalidOrderException, TemporaryError from freqtrade.persistence import Trade from freqtrade.rpc import RPC, RPCException from freqtrade.rpc.fiat_convert import CryptoToFiatConverter @@ -79,7 +79,8 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None: 'open_rate': 1.098e-05, 'close_rate': None, 'current_rate': 1.099e-05, - 'amount': 91.07468124, + 'amount': 91.07468123, + 'amount_requested': 91.07468123, 'stake_amount': 0.001, 'close_profit': None, 'close_profit_pct': None, @@ -142,7 +143,8 @@ def test_rpc_trade_status(default_conf, ticker, fee, mocker) -> None: 'open_rate': 1.098e-05, 'close_rate': None, 'current_rate': ANY, - 'amount': 91.07468124, + 'amount': 91.07468123, + 'amount_requested': 91.07468123, 'stake_amount': 0.001, 'close_profit': None, 'close_profit_pct': None, @@ -284,12 +286,66 @@ def test_rpc_trade_history(mocker, default_conf, markets, fee): assert isinstance(trades['trades'][1], dict) trades = rpc._rpc_trade_history(0) - assert len(trades['trades']) == 3 - assert trades['trades_count'] == 3 - # The first trade is for ETH ... sorting is descending - assert trades['trades'][-1]['pair'] == 'ETH/BTC' - assert trades['trades'][0]['pair'] == 'ETC/BTC' - assert trades['trades'][1]['pair'] == 'ETC/BTC' + assert len(trades['trades']) == 2 + assert trades['trades_count'] == 2 + # The first closed trade is for ETC ... sorting is descending + assert trades['trades'][-1]['pair'] == 'ETC/BTC' + assert trades['trades'][0]['pair'] == 'XRP/BTC' + + +def test_rpc_delete_trade(mocker, default_conf, fee, markets, caplog): + mocker.patch('freqtrade.rpc.telegram.Telegram', MagicMock()) + stoploss_mock = MagicMock() + cancel_mock = MagicMock() + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets), + cancel_order=cancel_mock, + cancel_stoploss_order=stoploss_mock, + ) + + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + freqtradebot.strategy.order_types['stoploss_on_exchange'] = True + create_mock_trades(fee) + rpc = RPC(freqtradebot) + with pytest.raises(RPCException, match='invalid argument'): + rpc._rpc_delete('200') + + create_mock_trades(fee) + trades = Trade.query.all() + trades[1].stoploss_order_id = '1234' + trades[2].stoploss_order_id = '1234' + assert len(trades) > 2 + + res = rpc._rpc_delete('1') + assert isinstance(res, dict) + assert res['result'] == 'success' + assert res['trade_id'] == '1' + assert res['cancel_order_count'] == 1 + assert cancel_mock.call_count == 1 + assert stoploss_mock.call_count == 0 + cancel_mock.reset_mock() + stoploss_mock.reset_mock() + + res = rpc._rpc_delete('2') + assert isinstance(res, dict) + assert cancel_mock.call_count == 1 + assert stoploss_mock.call_count == 1 + assert res['cancel_order_count'] == 2 + + stoploss_mock = mocker.patch('freqtrade.exchange.Exchange.cancel_stoploss_order', + side_effect=InvalidOrderException) + + res = rpc._rpc_delete('3') + assert stoploss_mock.call_count == 1 + stoploss_mock.reset_mock() + + cancel_mock = mocker.patch('freqtrade.exchange.Exchange.cancel_order', + side_effect=InvalidOrderException) + + res = rpc._rpc_delete('4') + assert cancel_mock.call_count == 1 + assert stoploss_mock.call_count == 0 def test_rpc_trade_statistics(default_conf, ticker, ticker_sell_up, fee, diff --git a/tests/rpc/test_rpc_apiserver.py b/tests/rpc/test_rpc_apiserver.py index 355b63f48..408f7e537 100644 --- a/tests/rpc/test_rpc_apiserver.py +++ b/tests/rpc/test_rpc_apiserver.py @@ -50,6 +50,12 @@ def client_get(client, url): 'Origin': 'http://example.com'}) +def client_delete(client, url): + # Add fake Origin to ensure CORS kicks in + return client.delete(url, headers={'Authorization': _basic_auth_str(_TEST_USER, _TEST_PASS), + 'Origin': 'http://example.com'}) + + def assert_response(response, expected_code=200, needs_cors=True): assert response.status_code == expected_code assert response.content_type == "application/json" @@ -352,7 +358,7 @@ def test_api_daily(botclient, mocker, ticker, fee, markets): assert rc.json['data'][0]['date'] == str(datetime.utcnow().date()) -def test_api_trades(botclient, mocker, ticker, fee, markets): +def test_api_trades(botclient, mocker, fee, markets): ftbot, client = botclient patch_get_signal(ftbot, (True, False)) mocker.patch.multiple( @@ -368,12 +374,53 @@ def test_api_trades(botclient, mocker, ticker, fee, markets): rc = client_get(client, f"{BASE_URI}/trades") assert_response(rc) - assert len(rc.json['trades']) == 3 - assert rc.json['trades_count'] == 3 - rc = client_get(client, f"{BASE_URI}/trades?limit=2") - assert_response(rc) assert len(rc.json['trades']) == 2 assert rc.json['trades_count'] == 2 + rc = client_get(client, f"{BASE_URI}/trades?limit=1") + assert_response(rc) + assert len(rc.json['trades']) == 1 + assert rc.json['trades_count'] == 1 + + +def test_api_delete_trade(botclient, mocker, fee, markets): + ftbot, client = botclient + patch_get_signal(ftbot, (True, False)) + stoploss_mock = MagicMock() + cancel_mock = MagicMock() + mocker.patch.multiple( + 'freqtrade.exchange.Exchange', + markets=PropertyMock(return_value=markets), + cancel_order=cancel_mock, + cancel_stoploss_order=stoploss_mock, + ) + rc = client_delete(client, f"{BASE_URI}/trades/1") + # Error - trade won't exist yet. + assert_response(rc, 502) + + create_mock_trades(fee) + ftbot.strategy.order_types['stoploss_on_exchange'] = True + trades = Trade.query.all() + trades[1].stoploss_order_id = '1234' + assert len(trades) > 2 + + rc = client_delete(client, f"{BASE_URI}/trades/1") + assert_response(rc) + assert rc.json['result_msg'] == 'Deleted trade 1. Closed 1 open orders.' + assert len(trades) - 1 == len(Trade.query.all()) + assert cancel_mock.call_count == 1 + + cancel_mock.reset_mock() + rc = client_delete(client, f"{BASE_URI}/trades/1") + # Trade is gone now. + assert_response(rc, 502) + assert cancel_mock.call_count == 0 + + assert len(trades) - 1 == len(Trade.query.all()) + rc = client_delete(client, f"{BASE_URI}/trades/2") + assert_response(rc) + assert rc.json['result_msg'] == 'Deleted trade 2. Closed 2 open orders.' + assert len(trades) - 2 == len(Trade.query.all()) + assert stoploss_mock.call_count == 1 def test_api_edge_disabled(botclient, mocker, ticker, fee, markets): @@ -519,7 +566,8 @@ def test_api_status(botclient, mocker, ticker, fee, markets): rc = client_get(client, f"{BASE_URI}/status") assert_response(rc) assert len(rc.json) == 1 - assert rc.json == [{'amount': 91.07468124, + assert rc.json == [{'amount': 91.07468123, + 'amount_requested': 91.07468123, 'base_currency': 'BTC', 'close_date': None, 'close_date_hum': None, @@ -641,6 +689,7 @@ def test_api_forcebuy(botclient, mocker, fee): fbuy_mock = MagicMock(return_value=Trade( pair='ETH/ETH', amount=1, + amount_requested=1, exchange='bittrex', stake_amount=1, open_rate=0.245441, @@ -657,6 +706,7 @@ def test_api_forcebuy(botclient, mocker, fee): data='{"pair": "ETH/BTC"}') assert_response(rc) assert rc.json == {'amount': 1, + 'amount_requested': 1, 'trade_id': None, 'close_date': None, 'close_date_hum': None, @@ -693,7 +743,7 @@ def test_api_forcebuy(botclient, mocker, fee): 'min_rate': None, 'open_order_id': '123456', 'open_rate_requested': None, - 'open_trade_price': 0.2460546025, + 'open_trade_price': 0.24605460, 'sell_reason': None, 'sell_order_status': None, 'strategy': None, diff --git a/tests/rpc/test_rpc_telegram.py b/tests/rpc/test_rpc_telegram.py index 631817624..bfa774856 100644 --- a/tests/rpc/test_rpc_telegram.py +++ b/tests/rpc/test_rpc_telegram.py @@ -21,8 +21,9 @@ from freqtrade.rpc import RPCMessageType from freqtrade.rpc.telegram import Telegram, authorized_only from freqtrade.state import State from freqtrade.strategy.interface import SellType -from tests.conftest import (get_patched_freqtradebot, log_has, patch_exchange, - patch_get_signal, patch_whitelist) +from tests.conftest import (create_mock_trades, get_patched_freqtradebot, + log_has, patch_exchange, patch_get_signal, + patch_whitelist) class DummyCls(Telegram): @@ -60,7 +61,7 @@ def test__init__(default_conf, mocker) -> None: assert telegram._config == default_conf -def test_init(default_conf, mocker, caplog) -> None: +def test_telegram_init(default_conf, mocker, caplog) -> None: start_polling = MagicMock() mocker.patch('freqtrade.rpc.telegram.Updater', MagicMock(return_value=start_polling)) @@ -72,10 +73,10 @@ def test_init(default_conf, mocker, caplog) -> None: assert start_polling.start_polling.call_count == 1 message_str = ("rpc.telegram is listening for following commands: [['status'], ['profit'], " - "['balance'], ['start'], ['stop'], ['forcesell'], ['forcebuy'], " - "['performance'], ['daily'], ['count'], ['reload_config', 'reload_conf'], " - "['show_config', 'show_conf'], ['stopbuy'], ['whitelist'], ['blacklist'], " - "['edge'], ['help'], ['version']]") + "['balance'], ['start'], ['stop'], ['forcesell'], ['forcebuy'], ['trades'], " + "['delete'], ['performance'], ['daily'], ['count'], ['reload_config', " + "'reload_conf'], ['show_config', 'show_conf'], ['stopbuy'], " + "['whitelist'], ['blacklist'], ['edge'], ['help'], ['version']]") assert log_has(message_str, caplog) @@ -690,8 +691,8 @@ def test_reload_config_handle(default_conf, update, mocker) -> None: assert 'reloading config' in msg_mock.call_args_list[0][0][0] -def test_forcesell_handle(default_conf, update, ticker, fee, - ticker_sell_up, mocker) -> None: +def test_telegram_forcesell_handle(default_conf, update, ticker, fee, + ticker_sell_up, mocker) -> None: mocker.patch('freqtrade.rpc.rpc.CryptoToFiatConverter._find_price', return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) mocker.patch('freqtrade.rpc.telegram.Telegram._init', MagicMock()) @@ -730,7 +731,7 @@ def test_forcesell_handle(default_conf, update, ticker, fee, 'pair': 'ETH/BTC', 'gain': 'profit', 'limit': 1.173e-05, - 'amount': 91.07468123861567, + 'amount': 91.07468123, 'order_type': 'limit', 'open_rate': 1.098e-05, 'current_rate': 1.173e-05, @@ -744,8 +745,8 @@ def test_forcesell_handle(default_conf, update, ticker, fee, } == last_msg -def test_forcesell_down_handle(default_conf, update, ticker, fee, - ticker_sell_down, mocker) -> None: +def test_telegram_forcesell_down_handle(default_conf, update, ticker, fee, + ticker_sell_down, mocker) -> None: mocker.patch('freqtrade.rpc.fiat_convert.CryptoToFiatConverter._find_price', return_value=15000.0) rpc_mock = mocker.patch('freqtrade.rpc.telegram.Telegram.send_msg', MagicMock()) @@ -790,7 +791,7 @@ def test_forcesell_down_handle(default_conf, update, ticker, fee, 'pair': 'ETH/BTC', 'gain': 'loss', 'limit': 1.043e-05, - 'amount': 91.07468123861567, + 'amount': 91.07468123, 'order_type': 'limit', 'open_rate': 1.098e-05, 'current_rate': 1.043e-05, @@ -839,7 +840,7 @@ def test_forcesell_all_handle(default_conf, update, ticker, fee, mocker) -> None 'pair': 'ETH/BTC', 'gain': 'loss', 'limit': 1.099e-05, - 'amount': 91.07468123861567, + 'amount': 91.07468123, 'order_type': 'limit', 'open_rate': 1.098e-05, 'current_rate': 1.099e-05, @@ -1146,6 +1147,63 @@ def test_edge_enabled(edge_conf, update, mocker) -> None: assert 'Pair Winrate Expectancy Stoploss' in msg_mock.call_args_list[0][0][0] +def test_telegram_trades(mocker, update, default_conf, fee): + msg_mock = MagicMock() + mocker.patch.multiple( + 'freqtrade.rpc.telegram.Telegram', + _init=MagicMock(), + _send_msg=msg_mock + ) + + freqtradebot = get_patched_freqtradebot(mocker, default_conf) + telegram = Telegram(freqtradebot) + context = MagicMock() + context.args = [] + + telegram._trades(update=update, context=context) + assert "0 recent trades:" in msg_mock.call_args_list[0][0][0] + assert "
" not in msg_mock.call_args_list[0][0][0]
+
+    msg_mock.reset_mock()
+    create_mock_trades(fee)
+
+    context = MagicMock()
+    context.args = [5]
+    telegram._trades(update=update, context=context)
+    msg_mock.call_count == 1
+    assert "2 recent trades:" in msg_mock.call_args_list[0][0][0]
+    assert "Profit (" in msg_mock.call_args_list[0][0][0]
+    assert "Open Date" in msg_mock.call_args_list[0][0][0]
+    assert "
" in msg_mock.call_args_list[0][0][0]
+
+
+def test_telegram_delete_trade(mocker, update, default_conf, fee):
+    msg_mock = MagicMock()
+    mocker.patch.multiple(
+        'freqtrade.rpc.telegram.Telegram',
+        _init=MagicMock(),
+        _send_msg=msg_mock
+    )
+
+    freqtradebot = get_patched_freqtradebot(mocker, default_conf)
+    telegram = Telegram(freqtradebot)
+    context = MagicMock()
+    context.args = []
+
+    telegram._delete_trade(update=update, context=context)
+    assert "invalid argument" in msg_mock.call_args_list[0][0][0]
+
+    msg_mock.reset_mock()
+    create_mock_trades(fee)
+
+    context = MagicMock()
+    context.args = [1]
+    telegram._delete_trade(update=update, context=context)
+    msg_mock.call_count == 1
+    assert "Deleted trade 1." in msg_mock.call_args_list[0][0][0]
+    assert "Please make sure to take care of this asset" in msg_mock.call_args_list[0][0][0]
+
+
 def test_help_handle(default_conf, update, mocker) -> None:
     msg_mock = MagicMock()
     mocker.patch.multiple(
diff --git a/tests/test_docs.sh b/tests/test_docs.sh
index 09e142b99..8a354daad 100755
--- a/tests/test_docs.sh
+++ b/tests/test_docs.sh
@@ -2,7 +2,8 @@
 # Test Documentation boxes -
 # !!! : is not allowed!
 # !!!  "title" - Title needs to be quoted!
-grep -Er '^!{3}\s\S+:|^!{3}\s\S+\s[^"]' docs/*
+# !!!  Spaces at the beginning are not allowed
+grep -Er '^!{3}\s\S+:|^!{3}\s\S+\s[^"]|^\s+!{3}\s\S+' docs/*
 
 if  [ $? -ne 0 ]; then
     echo "Docs test success."
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
index c7089abfe..3f42aa889 100644
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -595,7 +595,7 @@ def test_create_trade_minimal_amount(default_conf, ticker, limit_buy_order,
 
     freqtrade.create_trade('ETH/BTC')
     rate, amount = buy_mock.call_args[1]['rate'], buy_mock.call_args[1]['amount']
-    assert rate * amount >= default_conf['stake_amount']
+    assert rate * amount <= default_conf['stake_amount']
 
 
 def test_create_trade_too_small_stake_amount(default_conf, ticker, limit_buy_order,
@@ -782,7 +782,7 @@ def test_process_trade_creation(default_conf, ticker, limit_buy_order,
     assert trade.open_date is not None
     assert trade.exchange == 'bittrex'
     assert trade.open_rate == 0.00001098
-    assert trade.amount == 91.07468123861567
+    assert trade.amount == 91.07468123
 
     assert log_has(
         'Buy signal found: about create a new trade with stake_amount: 0.001 ...', caplog
@@ -1009,7 +1009,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
     call_args = buy_mm.call_args_list[0][1]
     assert call_args['pair'] == pair
     assert call_args['rate'] == bid
-    assert call_args['amount'] == stake_amount / bid
+    assert call_args['amount'] == round(stake_amount / bid, 8)
     buy_rate_mock.reset_mock()
 
     # Should create an open trade with an open order id
@@ -1029,7 +1029,7 @@ def test_execute_buy(mocker, default_conf, fee, limit_buy_order) -> None:
     call_args = buy_mm.call_args_list[1][1]
     assert call_args['pair'] == pair
     assert call_args['rate'] == fix_price
-    assert call_args['amount'] == stake_amount / fix_price
+    assert call_args['amount'] == round(stake_amount / fix_price, 8)
 
     # In case of closed order
     limit_buy_order['status'] = 'closed'
@@ -1301,7 +1301,7 @@ def test_create_stoploss_order_invalid_order(mocker, default_conf, caplog, fee,
     freqtrade.enter_positions()
     trade = Trade.query.first()
     caplog.clear()
-    freqtrade.create_stoploss_order(trade, 200, 199)
+    freqtrade.create_stoploss_order(trade, 200)
     assert trade.stoploss_order_id is None
     assert trade.sell_reason == SellType.EMERGENCY_SELL.value
     assert log_has("Unable to place a stoploss order on exchange. ", caplog)
@@ -1407,7 +1407,7 @@ def test_handle_stoploss_on_exchange_trailing(mocker, default_conf, fee, caplog,
     assert freqtrade.handle_stoploss_on_exchange(trade) is False
 
     cancel_order_mock.assert_called_once_with(100, 'ETH/BTC')
-    stoploss_order_mock.assert_called_once_with(amount=85.32423208191126,
+    stoploss_order_mock.assert_called_once_with(amount=85.32423208,
                                                 pair='ETH/BTC',
                                                 order_types=freqtrade.strategy.order_types,
                                                 stop_price=0.00002346 * 0.95)
@@ -1595,7 +1595,7 @@ def test_tsl_on_exchange_compatible_with_edge(mocker, edge_conf, fee, caplog,
     # stoploss should be set to 1% as trailing is on
     assert trade.stop_loss == 0.00002346 * 0.99
     cancel_order_mock.assert_called_once_with(100, 'NEO/BTC')
-    stoploss_order_mock.assert_called_once_with(amount=2132892.491467577,
+    stoploss_order_mock.assert_called_once_with(amount=2132892.49146757,
                                                 pair='NEO/BTC',
                                                 order_types=freqtrade.strategy.order_types,
                                                 stop_price=0.00002346 * 0.99)
@@ -1660,6 +1660,7 @@ def test_exit_positions_exception(mocker, default_conf, limit_buy_order, caplog)
     trade = MagicMock()
     trade.open_order_id = None
     trade.open_fee = 0.001
+    trade.pair = 'ETH/BTC'
     trades = [trade]
 
     # Test raise of DependencyException exception
@@ -1669,7 +1670,7 @@ def test_exit_positions_exception(mocker, default_conf, limit_buy_order, caplog)
     )
     n = freqtrade.exit_positions(trades)
     assert n == 0
-    assert log_has('Unable to sell trade: ', caplog)
+    assert log_has('Unable to sell trade ETH/BTC: ', caplog)
 
 
 def test_update_trade_state(mocker, default_conf, limit_buy_order, caplog) -> None:
@@ -1726,6 +1727,7 @@ def test_update_trade_state_withorderdict(default_conf, trades_for_order, limit_
         amount=amount,
         exchange='binance',
         open_rate=0.245441,
+        open_date=arrow.utcnow().datetime,
         fee_open=fee.return_value,
         fee_close=fee.return_value,
         open_order_id="123456",
@@ -1816,6 +1818,7 @@ def test_update_trade_state_sell(default_conf, trades_for_order, limit_sell_orde
         open_rate=0.245441,
         fee_open=0.0025,
         fee_close=0.0025,
+        open_date=arrow.utcnow().datetime,
         open_order_id="123456",
         is_open=True,
     )
@@ -2023,11 +2026,16 @@ def test_check_handle_timedout_buy_usercustom(default_conf, ticker, limit_buy_or
 
     rpc_mock = patch_RPCManager(mocker)
     cancel_order_mock = MagicMock(return_value=limit_buy_order_old)
+    cancel_buy_order = deepcopy(limit_buy_order_old)
+    cancel_buy_order['status'] = 'canceled'
+    cancel_order_wr_mock = MagicMock(return_value=cancel_buy_order)
+
     patch_exchange(mocker)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
         fetch_ticker=ticker,
         fetch_order=MagicMock(return_value=limit_buy_order_old),
+        cancel_order_with_result=cancel_order_wr_mock,
         cancel_order=cancel_order_mock,
         get_fee=fee
     )
@@ -2060,7 +2068,7 @@ def test_check_handle_timedout_buy_usercustom(default_conf, ticker, limit_buy_or
     freqtrade.strategy.check_buy_timeout = MagicMock(return_value=True)
     # Trade should be closed since the function returns true
     freqtrade.check_handle_timedout()
-    assert cancel_order_mock.call_count == 1
+    assert cancel_order_wr_mock.call_count == 1
     assert rpc_mock.call_count == 1
     trades = Trade.query.filter(Trade.open_order_id.is_(open_trade.open_order_id)).all()
     nb_trades = len(trades)
@@ -2071,7 +2079,9 @@ def test_check_handle_timedout_buy_usercustom(default_conf, ticker, limit_buy_or
 def test_check_handle_timedout_buy(default_conf, ticker, limit_buy_order_old, open_trade,
                                    fee, mocker) -> None:
     rpc_mock = patch_RPCManager(mocker)
-    cancel_order_mock = MagicMock(return_value=limit_buy_order_old)
+    limit_buy_cancel = deepcopy(limit_buy_order_old)
+    limit_buy_cancel['status'] = 'canceled'
+    cancel_order_mock = MagicMock(return_value=limit_buy_cancel)
     patch_exchange(mocker)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
@@ -2259,7 +2269,10 @@ def test_check_handle_cancelled_sell(default_conf, ticker, limit_sell_order_old,
 def test_check_handle_timedout_partial(default_conf, ticker, limit_buy_order_old_partial,
                                        open_trade, mocker) -> None:
     rpc_mock = patch_RPCManager(mocker)
-    cancel_order_mock = MagicMock(return_value=limit_buy_order_old_partial)
+    limit_buy_canceled = deepcopy(limit_buy_order_old_partial)
+    limit_buy_canceled['status'] = 'canceled'
+
+    cancel_order_mock = MagicMock(return_value=limit_buy_canceled)
     patch_exchange(mocker)
     mocker.patch.multiple(
         'freqtrade.exchange.Exchange',
@@ -2392,7 +2405,11 @@ def test_check_handle_timedout_exception(default_conf, ticker, open_trade, mocke
 def test_handle_cancel_buy(mocker, caplog, default_conf, limit_buy_order) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
-    cancel_order_mock = MagicMock(return_value=limit_buy_order)
+    cancel_buy_order = deepcopy(limit_buy_order)
+    cancel_buy_order['status'] = 'canceled'
+    del cancel_buy_order['filled']
+
+    cancel_order_mock = MagicMock(return_value=cancel_buy_order)
     mocker.patch('freqtrade.exchange.Exchange.cancel_order_with_result', cancel_order_mock)
 
     freqtrade = FreqtradeBot(default_conf)
@@ -2412,9 +2429,12 @@ def test_handle_cancel_buy(mocker, caplog, default_conf, limit_buy_order) -> Non
     assert not freqtrade.handle_cancel_buy(trade, limit_buy_order, reason)
     assert cancel_order_mock.call_count == 1
 
-    limit_buy_order['filled'] = 2
-    mocker.patch('freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException)
+    # Order remained open for some reason (cancel failed)
+    cancel_buy_order['status'] = 'open'
+    cancel_order_mock = MagicMock(return_value=cancel_buy_order)
+    mocker.patch('freqtrade.exchange.Exchange.cancel_order_with_result', cancel_order_mock)
     assert not freqtrade.handle_cancel_buy(trade, limit_buy_order, reason)
+    assert log_has_re(r"Order .* for .* not cancelled.", caplog)
 
 
 @pytest.mark.parametrize("limit_buy_order_canceled_empty", ['binance', 'ftx', 'kraken', 'bittrex'],
@@ -2578,7 +2598,7 @@ def test_execute_sell_up(default_conf, ticker, fee, ticker_sell_up, mocker) -> N
         'pair': 'ETH/BTC',
         'gain': 'profit',
         'limit': 1.172e-05,
-        'amount': 91.07468123861567,
+        'amount': 91.07468123,
         'order_type': 'limit',
         'open_rate': 1.098e-05,
         'current_rate': 1.173e-05,
@@ -2628,7 +2648,7 @@ def test_execute_sell_down(default_conf, ticker, fee, ticker_sell_down, mocker)
         'pair': 'ETH/BTC',
         'gain': 'loss',
         'limit': 1.044e-05,
-        'amount': 91.07468123861567,
+        'amount': 91.07468123,
         'order_type': 'limit',
         'open_rate': 1.098e-05,
         'current_rate': 1.043e-05,
@@ -2685,7 +2705,7 @@ def test_execute_sell_down_stoploss_on_exchange_dry_run(default_conf, ticker, fe
         'pair': 'ETH/BTC',
         'gain': 'loss',
         'limit': 1.08801e-05,
-        'amount': 91.07468123861567,
+        'amount': 91.07468123,
         'order_type': 'limit',
         'open_rate': 1.098e-05,
         'current_rate': 1.043e-05,
@@ -2891,7 +2911,7 @@ def test_execute_sell_market_order(default_conf, ticker, fee,
         'pair': 'ETH/BTC',
         'gain': 'profit',
         'limit': 1.172e-05,
-        'amount': 91.07468123861567,
+        'amount': 91.07468123,
         'order_type': 'market',
         'open_rate': 1.098e-05,
         'current_rate': 1.173e-05,
@@ -4087,14 +4107,14 @@ def test_sync_wallet_dry_run(mocker, default_conf, ticker, fee, limit_buy_order,
 def test_cancel_all_open_orders(mocker, default_conf, fee, limit_buy_order, limit_sell_order):
     default_conf['cancel_open_orders_on_exit'] = True
     mocker.patch('freqtrade.exchange.Exchange.fetch_order',
-                 side_effect=[DependencyException(), limit_sell_order, limit_buy_order])
+                 side_effect=[ExchangeError(), limit_sell_order, limit_buy_order])
     buy_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_cancel_buy')
     sell_mock = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.handle_cancel_sell')
 
     freqtrade = get_patched_freqtradebot(mocker, default_conf)
     create_mock_trades(fee)
     trades = Trade.query.all()
-    assert len(trades) == 3
+    assert len(trades) == 4
     freqtrade.cancel_all_open_orders()
     assert buy_mock.call_count == 1
     assert sell_mock.call_count == 1
diff --git a/tests/test_misc.py b/tests/test_misc.py
index 9fd6164d5..a185cbba4 100644
--- a/tests/test_misc.py
+++ b/tests/test_misc.py
@@ -11,7 +11,7 @@ from freqtrade.misc import (datesarray_to_datetimearray, file_dump_json,
                             file_load_json, format_ms_time, pair_to_filename,
                             plural, render_template,
                             render_template_with_fallback, safe_value_fallback,
-                            shorten_date)
+                            safe_value_fallback2, shorten_date)
 
 
 def test_shorten_date() -> None:
@@ -96,24 +96,40 @@ def test_format_ms_time() -> None:
 
 
 def test_safe_value_fallback():
+    dict1 = {'keya': None, 'keyb': 2, 'keyc': 5, 'keyd': None}
+    assert safe_value_fallback(dict1, 'keya', 'keyb') == 2
+    assert safe_value_fallback(dict1, 'keyb', 'keya') == 2
+
+    assert safe_value_fallback(dict1, 'keyb', 'keyc') == 2
+    assert safe_value_fallback(dict1, 'keya', 'keyc') == 5
+
+    assert safe_value_fallback(dict1, 'keyc', 'keyb') == 5
+
+    assert safe_value_fallback(dict1, 'keya', 'keyd') is None
+
+    assert safe_value_fallback(dict1, 'keyNo', 'keyNo') is None
+    assert safe_value_fallback(dict1, 'keyNo', 'keyNo', 55) == 55
+
+
+def test_safe_value_fallback2():
     dict1 = {'keya': None, 'keyb': 2, 'keyc': 5, 'keyd': None}
     dict2 = {'keya': 20, 'keyb': None, 'keyc': 6, 'keyd': None}
-    assert safe_value_fallback(dict1, dict2, 'keya', 'keya') == 20
-    assert safe_value_fallback(dict2, dict1, 'keya', 'keya') == 20
+    assert safe_value_fallback2(dict1, dict2, 'keya', 'keya') == 20
+    assert safe_value_fallback2(dict2, dict1, 'keya', 'keya') == 20
 
-    assert safe_value_fallback(dict1, dict2, 'keyb', 'keyb') == 2
-    assert safe_value_fallback(dict2, dict1, 'keyb', 'keyb') == 2
+    assert safe_value_fallback2(dict1, dict2, 'keyb', 'keyb') == 2
+    assert safe_value_fallback2(dict2, dict1, 'keyb', 'keyb') == 2
 
-    assert safe_value_fallback(dict1, dict2, 'keyc', 'keyc') == 5
-    assert safe_value_fallback(dict2, dict1, 'keyc', 'keyc') == 6
+    assert safe_value_fallback2(dict1, dict2, 'keyc', 'keyc') == 5
+    assert safe_value_fallback2(dict2, dict1, 'keyc', 'keyc') == 6
 
-    assert safe_value_fallback(dict1, dict2, 'keyd', 'keyd') is None
-    assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd') is None
-    assert safe_value_fallback(dict2, dict1, 'keyd', 'keyd', 1234) == 1234
+    assert safe_value_fallback2(dict1, dict2, 'keyd', 'keyd') is None
+    assert safe_value_fallback2(dict2, dict1, 'keyd', 'keyd') is None
+    assert safe_value_fallback2(dict2, dict1, 'keyd', 'keyd', 1234) == 1234
 
-    assert safe_value_fallback(dict1, dict2, 'keyNo', 'keyNo') is None
-    assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo') is None
-    assert safe_value_fallback(dict2, dict1, 'keyNo', 'keyNo', 1234) == 1234
+    assert safe_value_fallback2(dict1, dict2, 'keyNo', 'keyNo') is None
+    assert safe_value_fallback2(dict2, dict1, 'keyNo', 'keyNo') is None
+    assert safe_value_fallback2(dict2, dict1, 'keyNo', 'keyNo', 1234) == 1234
 
 
 def test_plural() -> None:
diff --git a/tests/test_persistence.py b/tests/test_persistence.py
index 8dd27e53a..65c83e05b 100644
--- a/tests/test_persistence.py
+++ b/tests/test_persistence.py
@@ -457,6 +457,7 @@ def test_migrate_old(mocker, default_conf, fee):
     assert trade.close_rate_requested is None
     assert trade.is_open == 1
     assert trade.amount == amount
+    assert trade.amount_requested == amount
     assert trade.stake_amount == default_conf.get("stake_amount")
     assert trade.pair == "ETC/BTC"
     assert trade.exchange == "bittrex"
@@ -546,6 +547,7 @@ def test_migrate_new(mocker, default_conf, fee, caplog):
     assert trade.close_rate_requested is None
     assert trade.is_open == 1
     assert trade.amount == amount
+    assert trade.amount_requested == amount
     assert trade.stake_amount == default_conf.get("stake_amount")
     assert trade.pair == "ETC/BTC"
     assert trade.exchange == "binance"
@@ -725,6 +727,7 @@ def test_to_json(default_conf, fee):
         pair='ETH/BTC',
         stake_amount=0.001,
         amount=123.0,
+        amount_requested=123.0,
         fee_open=fee.return_value,
         fee_close=fee.return_value,
         open_date=arrow.utcnow().shift(hours=-2).datetime,
@@ -757,6 +760,7 @@ def test_to_json(default_conf, fee):
                       'close_rate': None,
                       'close_rate_requested': None,
                       'amount': 123.0,
+                      'amount_requested': 123.0,
                       'stake_amount': 0.001,
                       'close_profit': None,
                       'close_profit_abs': None,
@@ -786,6 +790,7 @@ def test_to_json(default_conf, fee):
         pair='XRP/BTC',
         stake_amount=0.001,
         amount=100.0,
+        amount_requested=101.0,
         fee_open=fee.return_value,
         fee_close=fee.return_value,
         open_date=arrow.utcnow().shift(hours=-2).datetime,
@@ -808,6 +813,7 @@ def test_to_json(default_conf, fee):
                       'open_rate': 0.123,
                       'close_rate': 0.125,
                       'amount': 100.0,
+                      'amount_requested': 101.0,
                       'stake_amount': 0.001,
                       'stop_loss': None,
                       'stop_loss_abs': None,
@@ -989,7 +995,7 @@ def test_get_overall_performance(fee):
     create_mock_trades(fee)
     res = Trade.get_overall_performance()
 
-    assert len(res) == 1
+    assert len(res) == 2
     assert 'pair' in res[0]
     assert 'profit' in res[0]
     assert 'count' in res[0]
@@ -1004,5 +1010,5 @@ def test_get_best_pair(fee):
     create_mock_trades(fee)
     res = Trade.get_best_pair()
     assert len(res) == 2
-    assert res[0] == 'ETC/BTC'
-    assert res[1] == 0.005
+    assert res[0] == 'XRP/BTC'
+    assert res[1] == 0.01