How to tune hyperparameters
Introduction
After defining a reinforcement learning agent, its environment, and making sure it’s functional, the natural next step is hyperparameter tuning.
Hyperparameter tuning is a fancy way to talk about finding the best parameters of your agent that maximize your utility function.
This process is fairly computationally intensive and requires testing a large number of parameters, usually by a procedure called Grid Search.
This is where all_combinations_list
and search_similar
comes to aid in this process and automate the grid search process.
Definitions
Here is the definition of the all_combinations_list
function.
def all_combinations_list(arguments: Dict[str, List[Any]]) -> List[Dict[str, Any]]:
"""Given a dictionary of type Dict[str, List[Any]],
outputs a list of all the combinatorial combinations
(cartesian product) of the elements in the list.
This is useful in the case of trying many different
combinations of parameter for a reinforcement learning agent.
Example:
Given arguments: {"a":[1,2], "b":[3,4]}
Outputs: [{"a":1, "b":3}, {"a":1, "b":4}, {"a":2, "b":3}, {"a":2, "b":4}]
Args:
arguments (Dict[str, List[Any]]): Dictionary containing key-value pairs where the key is a string
and the value is a list of parameters.
Returns:
List[Dict[str,Any]]: Cartesian product of the elements.
"""
Here is the definition of search_similar
function.
def search_similar(searching_directory: str, subset_log_dict: Dict[str, Any]) -> bool:
"""Utilities function to be used when hyperparameter tuning to avoid training already trained agent specifications.
Given a searching directory and dictionary of parameters, will search the directory to see
if any logged parameter dictionary contains the specified parameters.
For example, one can specify subset_log_dict as the parameters of an agent to be trained and
the searching directory the directory of logs of this agent. Then, one may use this function
to avoid training already trained agent specifications by first checking if it already exists
in the logs.
Args:
searching_directory (str): Absolute path to the directory where the logs should be searched
subset_log_dict (Dict[str, Any]): Dictionary of parameters.
Returns:
bool: If True, there exists a log containing those parameters.
"""
Usage
The main idea is to leverage the from_dict
method from the Agent class to easily give the dictionary of arguments (that we obtain from all_combinations_list
) to an Agent.
search_similar
allows us to avoid recomputation of already trained agents.
An example is given below:
from Performance import all_combinations_list
agent_arguments = {
"inside_dim": [128,256,512],
"num_hidden_layers": [1,2,3],
"seed": [778]
}
logging_path = r"C:\ICCP\logs\simple_simulation"
searching_directory = r"C:\ICCP\logs\simple_simulation\DQN_Agent\results"
for curr_agent_arguments in all_combinations_list(agent_arguments):
agent = DQNAgent(env).from_dict(dict_arguments=curr_agent_arguments)
## creating the dictionary of parameters against which to check
log_dict = {**agent.log_dict(), **env.log_dict()}
num_episodes = 10
log_dict["num_episodes"] = num_episodes
## checking if logs already exist
if(not(search_similar(searching_directory, log_dict))):
results_path, summary_df = agent.train(logging_path= logging_path,
num_episodes=num_episodes,
num_iterations=None,
log=True)