Quick Start

Installation

  1. Clone the project
git clone https://github.com/GAIR-NLP/InnovatorBench.git
cd InnovatorBench
  1. Install Conda for Agents and Evaluations
# Install Conda
conda create -n ai_innovator python=3.10

pip install -r requirements.txt

# Editable install with no version control (alpaca_eval==0.6.2)
pip install -e alpaca_eval-0.6.2

Important: This environemnt also contains the package for evaluation. If you want to use your own agent, you should also set this conda environemnt to start the InnovatorBench evaluation environemnt.

  1. Prepare InnovatorBench Dataset
  • Download InnovatorBench Dataset from huggingface
  • Unzip the Dataset and save it to a path that can be accessed by all computers in the cluster. (Suggest: ./datasets)
  • Download datasets & checkpoints based on README.md in the dataset.
  • Copy & Paste corpus folder in task_10 and task_16
  • Move evaluations in InnovatorBench folder. (i.e. ./evaluations)
  • Update workspace_dataset_path in the research_gym/configs/tasks/task_i.yaml into the real dataset path like ./datasets
  1. Prepare ResearchGym
  • Set api in alpaca_eval-0.6.2/client_configs/openai_configs.yaml (Reference: alpaca_eval)
  • Prepare docker, web browse, etc. in backend
  • Deploy dockers and web server, update the relative key (computer_ip,web_server_host)in research_gym/configs/tasks/task_i.yaml, each task needs unique computer ips in the same time, but it can use the same web server. (You many need to deploy several dockers at the same time.)
  • If all your computers needs proxy to connect each other, set cmd_proxy_url in the research_gym/configs/tasks/task_i.yaml, otherwise, set it to null
  • Update other keys in the research_gym/configs/tasks/task_i.yaml
  • Set api in evaluations/base/data_classes.py

Configuration

  1. Config agents/config/agent_config.yaml or agents/config/agent_config_browse.yaml
  2. Change the config in research_gym/configs/tasks/task_i.yaml to your own path/keys.
  3. If you want to create your own task, copy research_gym/configs/tasks/base.yamland change the parameter inside it.

Run

Run a single task

conda activate ai_innovator
cd path_to_InnovatorBench
python main.py -t path_to_env_config -a path_to_agent_config