diff --git a/dataset/Evaluation.md b/dataset/Evaluation.md new file mode 100644 index 0000000..34118f3 --- /dev/null +++ b/dataset/Evaluation.md @@ -0,0 +1,26 @@ + +### OKVQA + +### GQA +Images and question-answer pairs will be loaded during the evaluation. + +``` python run_eval.py xxxx ``` + +### VSR +Images and question-answer pairs will be loaded during the evaluation. + +``` python run_eval.py xxxx ``` + +### IconVQA + +### VizWiz +1. Download [`test.json`](https://vizwiz.cs.colorado.edu/VizWiz_final/vqa_data/Annotations.zip) and extract [`test.zip`](https://vizwiz.cs.colorado.edu/VizWiz_final/images/test.zip) to `test`. Put them under `your_path/vizwiz`. +2. Single-GPU inference. +``` python run_eval.py xxxx ``` + +### HM + + + + + diff --git a/dataset/README_MINIGPTv2_FINETUNE.md b/dataset/README_MINIGPTv2_FINETUNE.md index 2d5c825..f7af60d 100644 --- a/dataset/README_MINIGPTv2_FINETUNE.md +++ b/dataset/README_MINIGPTv2_FINETUNE.md @@ -1,133 +1,94 @@ ## Download the COCO captions, RefCOCO, RefCOCO+. RefCOCOg, visual genome, textcaps, LLaVA, gqa, AOK-VQA, OK-VQA, OCR-VQA, filtered Flickr-30k, multi-task conversation, and Unnatural instruction datasets + ### COCO captions +- [train2017](http://images.cocodataset.org/zips/train2017.zip) -### RefCOCO, RefCOCO+, RefCOCOg - ### Visual genome +- [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip) -### textcaps +### TextCaps + +- [TextCaps_0.1_train](https://dl.fbaipublicfiles.com/textvqa/data/textcaps/TextCaps_0.1_train.json) +- [Images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip) + +### RefCOCO, RefCOCO+, RefCOCOg +Make sure you have the COCO 2014 images first. + +Then, +download RefCOCO, RefCOCO+, and RefCOCOg annotation files in the following links. + +- https://bvisionweb1.cs.unc.edu/licheng/referit/data/refcoco.zip +- https://bvisionweb1.cs.unc.edu/licheng/referit/data/refcoco+.zip +- https://bvisionweb1.cs.unc.edu/licheng/referit/data/refcocog.zip + +Unzip these files to the location you like. It should have the structure like the following + +``` +Location_you_like +├── refcoco +│ ├── instances.json +│ ├── refs(google).p +│ └── refs(unc).p +├── refcoco+ +│ ├── instances.json +│ └── refs(unc).p +└── refcocog + ├── instances.json + ├── refs(google).p + └── refs(umd).p +``` + +Set **image_path** in all the following dataset configuration files to the COCO 2014 image folder. +Similarly, set **ann_path** in all the following configs to the above folder (Location_you_like) that contains refcoco, refcoco+, and refcocog. + +- [minigpt4/configs/datasets/coco_bbox/refcoco.yaml](../minigpt4/configs/datasets/coco_bbox/refcoco.yaml) +- [minigpt4/configs/datasets/coco_bbox/refcocog.yaml](../minigpt4/configs/datasets/coco_bbox/refcocog.yaml) +- [minigpt4/configs/datasets/coco_bbox/refcocop.yaml](../minigpt4/configs/datasets/coco_bbox/refcocop.yaml) +- [minigpt4/configs/datasets/coco_bbox/invrefcoco.yaml](../minigpt4/configs/datasets/coco_bbox/invrefcoco.yaml) +- [minigpt4/configs/datasets/coco_bbox/invrefcocog.yaml](../minigpt4/configs/datasets/coco_bbox/invrefcocog.yaml) +- [minigpt4/configs/datasets/coco_bbox/invrefcocop.yaml](../minigpt4/configs/datasets/coco_bbox/invrefcocop.yaml) ### LLaVA +Makesure you have the COCO 2014 images first. + +Download Llava annotation files in the following link to the place you like. + +- https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/conversation_58k.json +- https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/detail_23k.json +- https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/complex_reasoning_77k.json + +Set **image_path** in all the following dataset configuration files to the COCO 2014 image folder. +Similarly, set **ann_path** to the location of the previous downloaded conversation_58k.json, +detail_23k.json, and complex_reasoning_77k.json in conversation.yaml, detail.yaml, and reason.yaml, respectively. + + +- [minigpt4/configs/datasets/llava/conversation.yaml](../minigpt4/configs/datasets/llava/conversation.yaml) +- [minigpt4/configs/datasets/llava/detail.yaml](../minigpt4/configs/datasets/llava/detail.yaml) +- [minigpt4/configs/datasets/llava/reason.yaml](../minigpt4/configs/datasets/llava/reason.yaml) -### gqa ### OKVQA +- [OK-VQA Input Questions](https://okvqa.allenai.org/static/data/OpenEnded_mscoco_train2014_questions.json.zip) +- [OK-VQA Annotations](https://okvqa.allenai.org/static/data/mscoco_train2014_annotations.json.zip) +- [okvqa_train](https://storage.googleapis.com/sfr-vision-language-research/LAVIS/datasets/okvqa/okvqa_train.json) +- Images are from COCO + ### AOK-VQA +``` +export AOKVQA_DIR=YOUR_DATASET_PATH +mkdir -p ${AOKVQA_DIR} +curl -fsSL https://prior-datasets.s3.us-east-2.amazonaws.com/aokvqa/aokvqa_v1p0.tar.gz | tar xvz -C ${AOKVQA_DIR} +``` + ### OCR-VQA +- [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing), **we save all files as `.jpg`** ### filtered Flickr-30k ### Multi-task conversation ### Unnatural instruction - - - - - - - - - - - - - -### Pre-training datasets download: -We use the filtered synthetic captions prepared by BLIP. For more details about the dataset, please refer to [BLIP](https://github.com/salesforce/BLIP). - -It requires ~2.3T to store LAION and CC3M+CC12M+SBU datasets - -Image source | Filtered synthetic caption by ViT-L ---- | :---: -CC3M+CC12M+SBU | Download -LAION115M | Download - -This will download two json files -``` -ccs_synthetic_filtered_large.json -laion_synthetic_filtered_large.json -``` - -## prepare the data step-by-step - - -### setup the dataset folder and move the annotation file to the data storage folder -``` -export MINIGPT4_DATASET=/YOUR/PATH/FOR/LARGE/DATASET/ -mkdir ${MINIGPT4_DATASET}/cc_sbu -mkdir ${MINIGPT4_DATASET}/laion -mv ccs_synthetic_filtered_large.json ${MINIGPT4_DATASET}/cc_sbu -mv laion_synthetic_filtered_large.json ${MINIGPT4_DATASET}/laion -``` - -### Convert the scripts to data storate folder -``` -cp convert_cc_sbu.py ${MINIGPT4_DATASET}/cc_sbu -cp download_cc_sbu.sh ${MINIGPT4_DATASET}/cc_sbu -cp convert_laion.py ${MINIGPT4_DATASET}/laion -cp download_laion.sh ${MINIGPT4_DATASET}/laion -``` - - -### Convert the laion and cc_sbu annotation file format to be img2dataset format -``` -cd ${MINIGPT4_DATASET}/cc_sbu -python convert_cc_sbu.py - -cd ${MINIGPT4_DATASET}/laion -python convert_laion.py -``` - -### Download the datasets with img2dataset -``` -cd ${MINIGPT4_DATASET}/cc_sbu -sh download_cc_sbu.sh -cd ${MINIGPT4_DATASET}/laion -sh download_laion.sh -``` - - -The final dataset structure - -``` -. -├── ${MINIGPT4_DATASET} -│ ├── cc_sbu -│ ├── convert_cc_sbu.py -│ ├── download_cc_sbu.sh -│ ├── ccs_synthetic_filtered_large.json -│ ├── ccs_synthetic_filtered_large.tsv -│ └── cc_sbu_dataset -│ ├── 00000.tar -│ ├── 00000.parquet -│ ... -│ ├── laion -│ ├── convert_laion.py -│ ├── download_laion.sh -│ ├── laion_synthetic_filtered_large.json -│ ├── laion_synthetic_filtered_large.tsv -│ └── laion_dataset -│ ├── 00000.tar -│ ├── 00000.parquet -│ ... -... -``` - - -## Set up the dataset configuration files - -Then, set up the LAION dataset loading path in -[here](../minigpt4/configs/datasets/laion/defaults.yaml#L5) at Line 5 as -${MINIGPT4_DATASET}/laion/laion_dataset/{00000..10488}.tar - -and the Conceptual Captoin and SBU datasets loading path in -[here](../minigpt4/configs/datasets/cc_sbu/defaults.yaml#L5) at Line 5 as -${MINIGPT4_DATASET}/cc_sbu/cc_sbu_dataset/{00000..01255}.tar - - - diff --git a/minigpt4/models/minigpt_base.py b/minigpt4/models/minigpt_base.py index b5dd9d4..cd051ec 100644 --- a/minigpt4/models/minigpt_base.py +++ b/minigpt4/models/minigpt_base.py @@ -178,7 +178,6 @@ class MiniGPTBase(BaseModel): answers = [self.llama_tokenizer(a + self.end_sym, return_tensors="pt", add_special_tokens=False).to(self.device) for a in answers] - cur_id = [] cur_target = [] for i in range(len(questions)): @@ -226,8 +225,6 @@ class MiniGPTBase(BaseModel): conv_q = [[self.prompt_template.format(item) for item in items] for items in conv_q] - - cond_embeds, cond_atts = self.prompt_wrap(img_embeds, img_atts, [q[0] for q in conv_q]) regress_token_ids, regress_atts, part_targets = self.tokenize_conversation(conv_q, conv_a) diff --git a/minigpt4/models/modeling_llama.py b/minigpt4/models/modeling_llama.py index 6d28020..5d59a53 100644 --- a/minigpt4/models/modeling_llama.py +++ b/minigpt4/models/modeling_llama.py @@ -75,7 +75,7 @@ class LlamaForCausalLM(LlamaForCausalLMOrig): ) hidden_states = outputs[0] - if self.config.pretraining_tp > 1: + if hasattr(self.config, 'pretraining_tp') and self.config.pretraining_tp > 1: lm_head_slices = self.lm_head.weight.split(self.vocab_size // self.config.pretraining_tp, dim=0) logits = [F.linear(hidden_states, lm_head_slices[i]) for i in range(self.config.pretraining_tp)] logits = torch.cat(logits, dim=-1) diff --git a/train.py b/train.py index 072a078..0013079 100644 --- a/train.py +++ b/train.py @@ -12,6 +12,7 @@ import random import numpy as np import torch import torch.backends.cudnn as cudnn +import wandb import minigpt4.tasks as tasks from minigpt4.common.config import Config @@ -30,7 +31,6 @@ from minigpt4.models import * from minigpt4.processors import * from minigpt4.runners import * from minigpt4.tasks import * -import wandb def parse_args(): @@ -44,12 +44,10 @@ def parse_args(): "in xxx=yyy format will be merged into config file (deprecate), " "change to --cfg-options instead.", ) - parser.add_argument("--wandb_log", default=False) - parser.add_argument("--job_name",default="minigpt_v2",type=str) + parser.add_argument("--job_name", default="minigpt_v2",type=str) args = parser.parse_args() - return args @@ -80,16 +78,13 @@ def main(): # set before init_distributed_mode() to ensure the same job_id shared across all ranks. job_id = now() args = parse_args() - - cfg = Config(parse_args()) + cfg = Config(args) init_distributed_mode(cfg.run_cfg) - setup_seeds(cfg) # set after init_distributed_mode() to only log on master. setup_logger() - cfg.pretty_print() task = tasks.setup_task(cfg) @@ -98,10 +93,9 @@ def main(): if cfg.run_cfg.wandb_log: wandb.login() - wandb.init(project="minigptv2",name=args.job_name) + wandb.init(project="minigptv", name=cfg.run_cfg.job_name) wandb.watch(model) - runner = get_runner_class(cfg)( cfg=cfg, job_id=job_id, task=task, model=model, datasets=datasets ) diff --git a/train_configs/minigpt4_llama2_stage1_pretrain.yaml b/train_configs/minigpt4_llama2_stage1_pretrain.yaml index c13d31f..bcc458e 100644 --- a/train_configs/minigpt4_llama2_stage1_pretrain.yaml +++ b/train_configs/minigpt4_llama2_stage1_pretrain.yaml @@ -52,4 +52,7 @@ run: device: "cuda" world_size: 1 dist_url: "env://" - distributed: True \ No newline at end of file + distributed: True + + wandb_log: True + job_name: minigpt4_llama2_pretrain \ No newline at end of file diff --git a/train_configs/minigpt4_llama2_stage2_finetune.yaml b/train_configs/minigpt4_llama2_stage2_finetune.yaml index 8c138ae..29b5358 100644 --- a/train_configs/minigpt4_llama2_stage2_finetune.yaml +++ b/train_configs/minigpt4_llama2_stage2_finetune.yaml @@ -46,4 +46,7 @@ run: device: "cuda" world_size: 1 dist_url: "env://" - distributed: True \ No newline at end of file + distributed: True + + wandb_log: True + job_name: minigpt4_llama2_finetune \ No newline at end of file diff --git a/train_configs/minigpt4_stage1_pretrain.yaml b/train_configs/minigpt4_stage1_pretrain.yaml index ce8bc87..bd9a451 100644 --- a/train_configs/minigpt4_stage1_pretrain.yaml +++ b/train_configs/minigpt4_stage1_pretrain.yaml @@ -52,4 +52,7 @@ run: device: "cuda" world_size: 1 dist_url: "env://" - distributed: True \ No newline at end of file + distributed: True + + wandb_log: True + job_name: minigpt4_pretrain \ No newline at end of file diff --git a/train_configs/minigpt4_stage2_finetune.yaml b/train_configs/minigpt4_stage2_finetune.yaml index 531a3a0..89d1100 100644 --- a/train_configs/minigpt4_stage2_finetune.yaml +++ b/train_configs/minigpt4_stage2_finetune.yaml @@ -46,4 +46,7 @@ run: device: "cuda" world_size: 1 dist_url: "env://" - distributed: True \ No newline at end of file + distributed: True + + wandb_log: True + job_name: minigpt4_finetune \ No newline at end of file diff --git a/train_configs/minigpt_v2_finetune.yaml b/train_configs/minigpt_v2_finetune.yaml index 7bf6bbf..4039ea6 100644 --- a/train_configs/minigpt_v2_finetune.yaml +++ b/train_configs/minigpt_v2_finetune.yaml @@ -276,7 +276,6 @@ run: init_lr: 1e-5 min_lr: 8e-5 warmup_lr: 1e-6 - wandb_log: True weight_decay: 0.05 max_epoch: 50 @@ -296,4 +295,7 @@ run: device: "cuda" world_size: 1 dist_url: "env://" - distributed: True \ No newline at end of file + distributed: True + + wandb_log: True + job_name: minigptv2_finetune \ No newline at end of file