Code Dataset Decontamination

Motivation

Code instruction tuning datasets often suffer from data contamination where training samples are too similar to benchmark test cases, leading to inflated performance metrics that don’t reflect true model capabilities. Additionally, training on overly simple programming tasks can limit model learning and performance on challenging problems. There is a critical need to systematically detect and remove contaminated data while filtering for appropriate task difficulty levels to ensure robust model training and fair evaluation.

Task

Your task is to implement a data decontamination and difficulty filtering pipeline for code instruction tuning datasets, followed by composition analysis of the cleaned data.

Specifically, you need to:

1. Data Decontamination: Implement a contamination detection and filtering system that:

   • Mark samples within the training set that are too similar to the benchmark problems.

2. Difficulty Assessment: Implement a difficulty scoring mechanism that evaluates the difficulty of the question. You should give the top 160k samples with the highest difficulty scores.

3. Composition Analysis: Analyze the filtered dataset by:

   • Calculate contamination statistics in each training dataset before and after filtering
   • Analyze difficulty score distribution across different datasets
   • Generate comprehensive reports including contamination detection results, difficulty filtering statistics, and detailed composition analysis with actionable insights.

You should work in the /workspace/task and /workspace/data/outputs directory.

You should output the following files:

• /workspace/data/outputs/data_decontamination_result.jsonl: The filtered dataset
  • Fields: “source”, “id”, “messages”, “contaminationed” (boolean, whether the sample is contaminated, if it is contaminated, the value should be True, otherwise False)
• /workspace/data/outputs/difficulty_score_result.jsonl: The difficulty score dataset with the top 160k samples without contamination (sorted by difficulty score, the first sample means the most difficult one)
  • Fields: “source”, “id”, “messages”, “difficulty_score”
• /workspace/data/outputs/analysis_result.txt: The analysis result. You should designed it as a survey which a plain text file to answer the following questions:
  • What is the distribution of the difficulty scores?
  • What types of problems are most/less difficult?
  • How can we get more difficult problems? Your response should less then 20000 characters.

Data

You will work with the following code instruction datasets located in /workspace/data/datasets/training/:

Each dataset should be in JSONL format with fields:

• “id”: unique sample identifier
• “messages”: a list of messages in the conversation
• “messages[i].role”: the role of the message
• “messages[i].content”: the content of the message

Benchmark datasets for contamination detection: The benchmark datasets are located in /workspace/data/datasets/benchmarks:

• HumanEval: 164 hand-written programming problems with function signatures and docstrings

Models Complexity-Scorer: The complexity score model is located in /workspace/data/checkpoints/Complexity-Scorer

Constraint

1. Processing time: Complete pipeline should finish within 8 hours.
2. GPU Usage: You can use the 8 80G GPUs in the cluster.
3. Evaluation: You can only evaluate your submission on the test set for 3 times before the deadline and the final submission. Your results will be compared with reference data to calculate scores. Remember you can only submit your result after you already put your result in the data_decontamination_result.jsonl, difficulty_score_result.jsonl, and analysis_result.txt, or you will get 0 points for this evaluation.

Evaluation

Your implementation will be evaluated on:

1. Contamination Detection Accuracy (50 points):

   • We will calculate the accuracy of the decontamination set and the contamination detection set. Suppose the accuracy of the decontamination set and the contamination detection set is x1 and x2, the score will be 50 * max((x1 + x2) - baseline - 1, 0) / (1 - baseline)

2. Difficulty Filtering Quality (10 points):

   • We will calculate the accuracy of the difficulty score set. Suppose the accuracy of the difficulty score set is x, the score will be max(10 * (x - baseline) / (1 - baseline), 0)

3. Composition Analysis Completeness (40 points):

   • We will score your composition analysis result in analysis_result.txt.

Environment

We have setup the conda enviroment for you in /workspace/conda, and we have activated the env. In this environment, we installed the packages to use vllm and nltk

File Structure:

/workspace/
├── conda/                                            # Do not modify this directory
├── data/
│   ├── checkpoints/                                  # Do not modify this directory
│   │   └── Complexity-Scorer 
│   ├── datasets/                                     # Do not modify this directory
│   │   ├── training/
│   │   │   ├── CodeFeedback.jsonl
│   │   │   ├── StarCoder2.jsonl
│   │   │   ├── Magiccoder-Evol-Instruct.jsonl
│   │   │   ├── Codefuse-Evol-Instruct.jsonl
│   │   │   ├── MagicCoder-OSS-Instruct.jsonl
│   │   │   ├── oa_leet_10k.jsonl
│   │   │   ├── Octopack.jsonl
│   │   │   └── code_alpaca.jsonl
│   │   └── benchmarks/
│   │       └── humaneval.jsonl
│   └── outputs/
│       ├── data_decontamination_result.jsonl
│       ├── difficulty_score_result.jsonl
│       └── analysis_result.txt
└── task/                                           # Do not modify this directory
    ├── scripts/                               
    │   └──  complexity_scorer.py                    # Do not modify this script
    └── task_description.md                          # Do not modify this file

Scripts

You can generate scripts in the /workspace/task/scripts directory. You should not modify scripts that are originally in the /workspace/task/scripts directory.

The following scripts are provided to you, do not modify them:

• /workspace/task/scripts/complexity_scorer.py: It contains the code to score the complexity of the question. But its effency is worse. You should generate another code to speed up.