Skip to content

create-study.agent.md

Latest commit

 

History

History
877 lines (694 loc) · 41.2 KB

create-study.agent.md

File metadata and controls

877 lines (694 loc) · 41.2 KB
Raw
name create-study
description Creates subjective speech quality tests using the P.808 toolkit — handles study setup, gold/trapping clip generation, storage upload, project building for crowdsourcing platforms, prompts for study-specific screen-out completion codes (never reused across studies), internal-vs-general master-script assets, required_playback_device, and P.804 allow_cannot_rate/show_loudness_alert settings, and writes a re-runnable regenerate_study.bat that re-runs master_script.py. Quantity and quality bonuses apply to AMT only, not Prolific.

Create subjective test instructions

Use this runbook when asked to create a new subjective speech quality test with the P.808 toolkit.

Trigger phrases: "create a study", "run a [method] test", "set up a [method] study", "prepare a [method] test for these files".

Platform and shell adaptation

Code examples use PowerShell on Windows (\ paths). Adapt for other OS/shells: replace PowerShell cmdlets with equivalents, use python3 if needed, convert paths. Replace REPO_ROOT with the actual absolute path of this repository.

Best-practice variables

These are best-practice defaults. Confirm or override them with the requester before first use. After confirmation, save a .cfg file next to the input files so future runs can reuse it. When asked to re-run a test or "go yolo", look for an existing config file first.

For Prolific, ignore the quantity and quality bonus defaults: these bonuses are not applied. Do not ask the requester to confirm bonus amounts or thresholds, include them in payment estimates, or flag bonus-threshold warnings as publishing issues. Base payment still applies.

BEST_PRACTICE_PLATFORM             = Prolific
BEST_PRACTICE_VALID_VOTE_BUFFER    = 20%
BEST_PRACTICE_CLIPS_PER_SESSION    = 10
BEST_PRACTICE_GOLD_PER_SESSION     = 1   (use 2 for P.804 — see method-specific notes)
BEST_PRACTICE_TRAPPING_PER_SESSION = 1
BEST_PRACTICE_TRAINING_CLIPS       = 5
BEST_PRACTICE_GOLD_SOURCE_COUNT    = max(3, ceil(0.05 * number_of_rating_clips))
BEST_PRACTICE_TRAPPING_SOURCE_COUNT= max(3, ceil(0.05 * number_of_rating_clips))
BEST_PRACTICE_MAX_GOLD_SOURCE_CLIPS    = 15
BEST_PRACTICE_MAX_TRAPPING_SOURCE_CLIPS= 15
BEST_PRACTICE_ALLOWED_MAX_HITS     = min(int(number_of_rating_clips / 10), 50)
BEST_PRACTICE_BASE_PAYMENT         = 0.50
BEST_PRACTICE_QUANTITY_BONUS       = 0.10
BEST_PRACTICE_QUALITY_BONUS        = 0.15
BEST_PRACTICE_BW_MIN               = FB

Scope

This instruction covers preparing inputs, generating gold/trapping clips, uploading to storage, running master_script.py, and handing off for publishing. Setting up the HIT in a HITAPP server and publishing is done by the requester.

Mandatory pre-check

Before editing or running anything in this repository:

  1. Read AGENTS.md and .github\copilot-instructions.md.
  2. Confirm this is a creation task, not analysis. For analysis, use .github\evaluate.instruction.md instead.

Environment prerequisites

Verify once at the start:

  1. az CLI logged in: az account show. If expired, prompt az login.
  2. Python deps: pip install -r requirements.txt --quiet in src\.
  3. PowerShell: use Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass. Prefer running Python scripts directly over .ps1 wrappers.

Once prerequisites pass, proceed without pausing except at [ASK] decision points.

Reusing information from prior studies

When creating a new study that reuses settings from a previous study or session knowledge (e.g. same storage account, contact email, platform, source clips, gold/trapping clips, training clips), do not silently reuse them. Instead:

  1. Collect all reused or assumed values into a single summary list.
  2. [ASK] Present the list and ask: "I plan to reuse the following from the previous study. Are all of these correct?"
    • If the user confirms all → proceed.
    • If the user says they want to modify → ask about each item one by one, then proceed with the updated values.

Never reuse the platform screen-out completion code (screenout_code) from a prior study or session. The crowd platform issues it per study, so a copied code is invalid. Always [ASK] for the new study's code, or leave it unset to emit the HIT App Server ${...} placeholder — never silently carry it over.

Supported test methods

Method --method flag Gold clip generation Trapping config Template
ACR acr --method acr trapping.cfg or trapping_p835.cfg ACR_template.html
DCR dcr N/A (manual) N/A — see note below DCR_template.html
CCR ccr N/A (manual) N/A — see note below CCR_template.html
P.835 p835 --method acr (not p835) trapping.cfg or trapping_p835.cfg P835_template.html
P.804 p804 --method p804 trapping_p804.cfg via pp835_p804 path
Echo impairment echo_impairment_test --method acr trapping.cfg echo_impairment_test_template.html
Personalized P.835 pp835 special (per-dimension) trapping.cfg P835_personalized_template3.html

Critical: For plain p835, use --method acr when generating gold clips with create_gold_clips.py. The p835 method in the gold generator produces per-dimension columns (gold_sig_ans, gold_bak_ans, gold_ovrl_ans) but master_script.py expects gold_clips_ans for plain P.835.

DCR/CCR trapping note (legacy): DCR and CCR do not use traditional trapping questions (i.e. clips with overlaid spoken scores generated by create_trapping_stimuli.py). Instead, the TP field in the publish batch contains a reference clip that functions as a gold/control question — the worker compares a clip to itself, so the expected answer is "about the same" (0). The trapping_clips.csv for DCR/CCR should contain reference clip URLs only (no trapping_ans column). Do not run create_trapping_stimuli.py for these methods. This is a legacy naming issue in the code and may be updated in the future.

Inputs the agent must confirm

Do not guess these values if they are missing:

  1. Test method: one of acr, dcr, ccr, p835, p804, echo_impairment_test, pp835.
  2. Crowd platform: Prolific (recommended), AMT, or another panel.
  3. Project name: for generated output folder and files.
  4. Input resources:
    • rating_clips.csv — required.
    • training_clips.csv — required for most methods (can be auto-generated from rating clips, but manual selection is recommended — see section 4). Not needed for P.804 or pp835 when training_gold_clips.csv is provided or generated — the two are mutually exclusive (see section 4).
    • training_gold_clips.csv — P.804 and pp835 only. Contains training clips with per-dimension answers, variance, and feedback messages. When available (provided or auto-generated), this takes priority over training_clips.csv — do not ask for or generate plain training clips. If not provided the agent can generate one from gold clips (see section 4b).
    • gold_clips.csv — optional (can be generated from source clips).
    • trapping_clips.csv — optional (can be generated from source clips).
  5. Source clips for gold/trapping generation:
    • Gold clips: if gold_clips.csv is not provided, the agent must not blindly download random rating clips. Gold clips require high-quality, clean reference audio. Ask the user one of:
      • Do they have a local directory of clean reference WAV files from the same dataset?
      • Can they identify clean clips by a URL pattern (e.g. */clean/*, */reference/*)?
      • Would they like the agent to download a small subset of rating clips for the user to listen to and manually remove any clips with distortion before gold generation? Mandatory: however the candidate clean clips are obtained, they must go through the user review in section 5a (keep only clips the user would rate 5 on all scales) before gold generation. This step is required and must not be skipped. Important: never use the sample clips bundled in this repository (src\test_inputs\). Source clips must come from the same dataset as the rating clips.
    • Trapping clips: if trapping_clips.csv is not provided, the quality of source clips does not matter for trapping questions. The agent can download a random sample of rating clips and use them directly — no manual review needed.
  6. Storage: the Azure storage account name and container for uploading generated clips. The container must be publicly accessible — crowd workers need unauthenticated access. See "Storage and public accessibility" below for the full check procedure.
  7. Contact email: the email address to show in the HIT app for worker inquiries. Do not use a hardcoded default — always ask.
  8. Screen-out completion code (screenout_code): a study-specific code issued by the crowd platform for paying screened-out participants. A single code covers both the qualification and setup screen-out (platforms such as Prolific support only one code per study). Never copy it from a prior study — always ask for the new study's code, or leave it unset to emit the HIT App Server ${...} placeholder. See the config template below.
  9. Max assignments per worker (for Prolific) or worker requirements and payment (for AMT).
  10. Target valid votes per clip: suggest publishing target + BEST_PRACTICE_VALID_VOTE_BUFFER.
  11. Master-script general assets (--general_assets): the shared reference material for the setup section — the math questions, comparison (CMP) pairs, hearing test and bandwidth-check clips with their answers/hashes. [ASK] whether the requester will provide/point to the internal assets (src/assets_master_script/general_assets_internal.csv, the larger internal set) or use the default general set (src/assets_master_script/general.csv, applied automatically when --general_assets is omitted). Pass --general_assets only when using the internal set.
  12. Required playback device (required_playback_device): [ASK] during study-details collection whether to require headset, loudspeaker, or any. Explain that headset is the toolkit default and any skips the playback-device check. Save the confirmed value explicitly in [hit_app_html]; do not silently assume the default.
  13. P.804 "Cannot rate it" option (allow_cannot_rate): [ASK] for P.804 studies whether to enable this option on the Coloration, Discontinuity, Reverb, and Signal-quality scales. Suggest false as the default and save the confirmed boolean in [hit_app_html]. Do not ask for this P.804-only setting for other methods.
  14. P.804 loudness warning (show_loudness_alert): [ASK] for P.804 studies whether to display the rating-section warning that clips may be very loud and recommend loudspeaker playback. Suggest false as the default and save the confirmed boolean in [hit_app_html]. If enabled, recommend required_playback_device: any so workers can use headsets for qualification/setup, which can be difficult without them, and loudspeakers for ratings. Explain the possible conflict with headset and ask the requester to confirm the device choice; never silently change it. Do not ask for this P.804-only setting for other methods.

Storage and public accessibility

All clip URLs must be publicly accessible for crowd workers.

Check accessibility: pick one URL from the clip list and test with HTTP HEAD:

$testUrl = "<first_url_from_rating_clips>"
try {
    $r = Invoke-WebRequest -Uri $testUrl -Method Head -UseBasicParsing -ErrorAction Stop
    Write-Host "PUBLIC — HTTP $($r.StatusCode)"
} catch { Write-Host "NOT PUBLIC — $($_.Exception.Message)" }
  • If public (HTTP 200): upload gold/trapping clips to the same account/container.
  • If not public (HTTP 403/409): ask the user which public container to copy all clips to. Use a random opaque subdirectory (e.g. PROJECT_NAME/stim_x7k2m9).

Preserving directory structure when copying

Rating clips often share filenames across subdirectories (each representing a different condition). Never copy to a flat directory — preserve the parent directory name:

Source:  .../condition_A/clip1.wav  →  Dest: .../<random_subdir>/condition_A/clip1.wav
Source:  .../condition_B/clip1.wav  →  Dest: .../<random_subdir>/condition_B/clip1.wav

Copy method: azcopy with SAS tokens (preferred)

For private-to-public transfers, use azcopy with user-delegation SAS tokens on both source and destination URLs. Never use azcopy login — its token cache is separate from az login, expires after 90 days of inactivity (AADSTS700082), and cannot be refreshed via az login. Always generate SAS tokens with az instead.

Generate SAS tokens for both containers and copy:

$expiry = (Get-Date).AddHours(2).ToUniversalTime().ToString("yyyy-MM-ddTHH:mmZ")

$srcSas = az storage container generate-sas `
    --account-name SOURCE_ACCOUNT --name SOURCE_CONTAINER `
    --permissions rl --expiry $expiry `
    --auth-mode login --as-user -o tsv

$destSas = az storage container generate-sas `
    --account-name DEST_ACCOUNT --name DEST_CONTAINER `
    --permissions rwl --expiry $expiry `
    --auth-mode login --as-user -o tsv

azcopy copy `
    "https://SOURCE_ACCOUNT.blob.core.windows.net/SOURCE_CONTAINER/path?$srcSas" `
    "https://DEST_ACCOUNT.blob.core.windows.net/DEST_CONTAINER/dest_path?$destSas" `
    --recursive

This uses your az login session to generate the tokens — no azcopy login needed. Requires the Storage Blob Delegator role (included in Storage Blob Data Contributor) on both storage accounts.

Important: azcopy with --recursive preserves the source directory tree. The copied paths will include intermediate directories from the source prefix. After copying, update the rating clips CSV to reflect the actual destination paths (verify with az storage blob list).

Common failures:

Symptom Fix
AADSTS700082: refresh token expired You used azcopy login — switch to the SAS token approach above
AuthorizationPermissionMismatch Assign Storage Blob Data Contributor role on both accounts
azcopy: command not found Install from https://aka.ms/azcopy or use the az fallback below

Fallback: if azcopy is not installed, use az storage blob copy start with SAS-authenticated source URIs:

$clips | ForEach-Object -Parallel {
    $url = $_.rating_clips
    $sasToken = $using:srcSas
    if ($url -match 'https?:/+([^.]+)\.blob\.core\.windows\.net/([^/]+)/(.+)/([^/]+)$') {
        $parentDir = ($Matches[3] -split '/')[-1]; $fileName = $Matches[4]
        az storage blob copy start `
            --account-name DEST_ACCOUNT --destination-container DEST_CONTAINER `
            --destination-blob "DEST_PREFIX/$parentDir/$fileName" `
            --source-uri "$url`?$sasToken" `
            --auth-mode login 2>&1 | Out-Null
    }
} -ThrottleLimit 20

CSV column names by method

These are the actual column names expected by the code in src\create_input.py and src\master_script.py.

Single-stimulus methods (ACR, P.835, echo_impairment_test)

CSV file Columns
rating_clips.csv rating_clips
training_clips.csv training_clips
gold_clips.csv gold_clips, gold_clips_ans
trapping_clips.csv trapping_clips, trapping_ans

P.804

P.804 gold clips use per-dimension answer columns and a ver column to assign clips to gold slots. The master_script.py internally renames columns via update_gold_clips_for_p804().

CSV file Columns
rating_clips.csv rating_clips
training_clips.csv training_clips (not needed if training_gold_clips.csv is used)
training_gold_clips.csv training_clips, noise_ans, noise_var, noise_msg, disc_ans, disc_var, disc_msg, col_ans, col_var, col_msg, loud_ans, loud_var, loud_msg, reverb_ans, reverb_var, reverb_msg, sig_ans, sig_var, sig_msg, ovrl_ans, ovrl_var, ovrl_msg
gold_clips.csv gold_url, col_ans, disc_ans, loud_ans, noise_ans, reverb_ans, sig_ans, ovrl_ans, ver
trapping_clips.csv trapping_clips, trapping_ans

Column mapping note: create_gold_clips.py --method p804 outputs a column named gold_clips. You must rename it to gold_url before passing it to master_script.py. The answer columns (col_ans, disc_ans, etc.) are output without the gold_ prefix and should be kept as-is — the master script adds the prefix internally.

The ver column is required and must contain an integer (1 or 2) indicating which gold slot the clip belongs to. See section 5 for how to generate two sets.

Double-stimulus methods (DCR, CCR)

CSV file Columns
rating_clips.csv rating_clips, references
training_clips.csv training_clips, training_references
trapping_clips.csv trapping_clips (uses references as trapping)

Personalized P.835 (pp835)

CSV file Columns
gold_clips.csv gold_url, gold_sig_ans, gold_bak_ans, gold_ovrl_ans
training_gold_clips.csv training_clips, sig_ans, sig_var, sig_msg, bak_ans, bak_var, bak_msg, ovrl_ans, ovrl_var, ovrl_msg

See src\test_inputs\ for example CSV files.

Execution workflow

1. Prepare the environment

Environment setup is covered in "Environment prerequisites" above. Verify az login and install dependencies before entering the workflow.

2. Validate the input clip list

Before proceeding, validate every URL to catch typos, broken links, or inaccessible files early.

  1. Parse the CSV and fix common URL formatting issues (e.g. https:/host → https://host).
  2. Validate URLs using one of the approaches below.
  3. If any URLs are invalid, stop and report the full list to the user.
  4. Check for duplicate URLs — report them and ask the user to clarify before continuing.

Validation approaches

For public storage: use check_urls_in_files_exist from master_script.py for fast multicore validation:

import sys, os
sys.path.insert(0, os.path.join("REPO_ROOT", "src"))
from master_script import check_urls_in_files_exist
check_urls_in_files_exist("CLIP_LIST_PATH", ["COLUMN_NAME"])

For private storage: check_urls_in_files_exist uses plain HTTP HEAD and will fail with HTTP 409. Generate a SAS token using az, append it to every URL temporarily, then validate:

$expiry = (Get-Date).AddHours(2).ToUniversalTime().ToString("yyyy-MM-ddTHH:mmZ")
$sasToken = az storage container generate-sas `
    --account-name SOURCE_ACCOUNT --name SOURCE_CONTAINER `
    --permissions rl --expiry $expiry `
    --auth-mode login --as-user -o tsv

Then temporarily append ?$sasToken to every URL in the CSV, call check_urls_in_files_exist, and strip the token afterwards. Store the SAS token for reuse in later steps (downloading clips, azcopy transfers).

[ASK] If clips are on private storage: "Your clips are on private storage. I can generate a SAS token using your az login session to validate and download clips. Should I go ahead?"

3. Check for existing project config

Look for a .cfg file next to the rating_clips.csv in the requester's data directory. If one exists, offer to reuse it. If this is a re-run or "go yolo" request, use it directly.

4. Prepare training clips

Training clips anchor participants' perception and should represent the quality distribution within the dataset — from worst to best.

P.804 and pp835 — training gold clips take priority:

For P.804 and pp835, training_gold_clips.csv and training_clips.csv are mutually exclusive — the master script accepts one or the other, not both. Because training gold clips provide richer per-dimension feedback, they are always preferred:

  1. If the user provides training_gold_clips.csv → use it, skip plain training clips.
  2. If neither is provided → generate training_gold_clips.csv from gold clips (see section 4b), skip plain training clips.
  3. Only generate or ask for training_clips.csv when the method is not P.804/pp835, or when the user explicitly opts out of training gold clips.

For all other methods (ACR, DCR, CCR, P.835, echo impairment):

[ASK] Ask the user: "Can you provide a training_clips.csv file with manually selected clips that represent the quality distribution in your dataset? For multi-scale tests (P.804, P.835), training clips should also show variations across all dimensions. If not, I can randomly select some samples, but manual selection is recommended."

If the user provides a file, use it directly. Otherwise, auto-generate:

Set-Location REPO_ROOT\src
python utils\select_training_clips.py `
	--input RATING_CLIPS_PATH\rating_clips.csv `
	--output RATING_CLIPS_PATH\training_clips.csv `
	--count 5

Note: select_training_clips.py selects clips purely by list position without knowledge of actual quality. Manual selection is always preferred.

4b. Prepare training gold clips (P.804 and pp835 only)

For P.804 and personalized P.835, you can provide training_gold_clips.csv which adds per-dimension answers, accepted variance, and feedback messages to training clips. This enables the HIT app to show participants feedback if their training answers deviate too far from the expected score.

[ASK] Ask the user: "For P.804/pp835, do you have a training_gold_clips.csv with per-dimension answers and feedback messages? If not, I can generate one from the gold clips by selecting those with the highest deviation across dimensions (up to 5 clips)."

CSV format for P.804 training_gold_clips.csv:

Column Description
training_clips URL of the training clip
noise_ans Expected noise score (1–5)
noise_var Accepted deviation (e.g. 1); use 0 to skip feedback for this dimension
noise_msg Feedback message shown if the answer deviates
disc_ans, disc_var, disc_msg Same for discontinuity
col_ans, col_var, col_msg Same for coloration
loud_ans, loud_var, loud_msg Same for loudness
reverb_ans, reverb_var, reverb_msg Same for reverberation
sig_ans, sig_var, sig_msg Same for signal distortion
ovrl_ans, ovrl_var, ovrl_msg Same for overall quality

For pp835, use columns: sig_ans/var/msg, bak_ans/var/msg, ovrl_ans/var/msg.

See src\test_inputs\training_gold_clips_p804.csv for an example.

Rules: _var = 1 accepts ±1 deviation; _var = 0 skips feedback. _msg is a short feedback message. Empty _ans cells accept any answer.

Auto-generating from gold clips:

If the user does not provide training gold clips, generate them from the gold clips:

  1. Select up to 5 gold clips with the most distinctive quality characteristics (prefer clips with extreme or opposite dimension values).
  2. Assign _var = 1 for all dimensions that have an answer.
  3. Write brief feedback messages for each dimension describing the expected quality.
  4. Upload these clips to public storage (they may already be uploaded as gold clips).

5. Generate gold clips (if not provided)

Gold clips require high-quality, clean reference WAV files — not arbitrary rating clips.

[ASK] Source clips: ask the user how to obtain clean source audio (see "Inputs the agent must confirm", item 5). Options:

  • The user provides a directory of clean WAV files from the same dataset.
  • The user identifies clean clips by a URL pattern (e.g. */clean/*).
  • Download a subset and let the user review them to remove any with distortion.

If downloading clips from Azure private storage, either:

  • Use download_clips.py with --sas_token to authenticate, or
  • Use az storage blob download with --auth-mode login for each clip individually.

How many source clips? Use BEST_PRACTICE_GOLD_SOURCE_COUNT capped at BEST_PRACTICE_MAX_GOLD_SOURCE_CLIPS.

5a. MANDATORY — user review of clean reference clips before generation

The clean reference clips are the foundation of every gold question: the generator assumes each source clip is perfect and would be rated 5 on all scales. If a reference clip has any audible flaw, every gold clip derived from it will have a wrong expected answer and will wrongly fail workers. Therefore this review step is required and must not be skipped, regardless of how the candidate clips were obtained.

You must:

  1. Copy the candidate clean/reference clips into a temporary review directory, e.g. RATING_CLIPS_PATH\gold_source_review.
  2. [ASK] Instruct the user, in these exact terms, to review them:
    • "Please listen to every clip in gold_source_review."
    • "Keep only the clips you would personally rate 5 on ALL scales (coloration, discontinuity, loudness, noise, reverb, signal, and overall)."
    • "If a clip is not a perfect 5 on every scale, delete that file from the folder."
    • "Do not edit the clips — only keep or delete them."
  3. Wait for the user to explicitly confirm they have finished reviewing and deleting. Do not proceed on assumption.
  4. After confirmation, use only the clips that remain in the review directory as the gold source (copy the survivors into RATING_CLIPS_PATH\gold_source). If the user deleted everything, or too few remain, ask for more candidate clips and repeat — do not fall back to unreviewed clips.

Generate gold clips (filenames are anonymized by default — do not use --no_anonymize):

python create_gold_clips.py `
	--input_dir RATING_CLIPS_PATH\gold_source `
	--output_dir RATING_CLIPS_PATH\gold_output `
	--method GOLD_METHOD

Method mapping for create_gold_clips.py:

Study method Use --method Output columns
acr acr gold_clips, gold_clips_ans
p835 acr gold_clips, gold_clips_ans
echo_impairment_test acr gold_clips, gold_clips_ans
p804 p804 gold_clips, col_ans, disc_ans, loud_ans, noise_ans, reverb_ans, sig_ans, ovrl_ans
pp835 p835 gold_clips, gold_sig_ans, gold_bak_ans, gold_ovrl_ans

Note: Each source clip produces multiple gold clips (clean, noisy, distorted, etc.). With 3 source clips you get approximately 12 gold clips for ACR, more for P.804 (~11 variants per source clip).

P.804-specific: assigning ver column from a single gold set

For P.804, always use number_of_gold_clips_per_session = 2. You do not need two independent sets of source clips. Instead, generate one set and assign ver based on the ovrl_ans value:

  • Clips with ovrl_ans = 5 (clean/high-quality) → ver = 1
  • Clips with ovrl_ans = 1 (degraded) → ver = 2
  1. Run create_gold_clips.py --method p804 once with all source clips.
  2. Rename gold_clips → gold_url in the output CSV.
  3. Add a ver column: ver=1 when ovrl_ans=5 (clean), ver=2 when ovrl_ans=1 (degraded).
  4. Export as gold_clips.csv.

After generation, upload to public storage:

python utils\copy_to_pub_storage.py upload `
	--input RATING_CLIPS_PATH\gold_output\gold_clips_report.csv `
	--columns gold_clips --local-dir RATING_CLIPS_PATH\gold_output `
	--account-name STORAGE_ACCOUNT_NAME `
	--target-container TARGET_CONTAINER `
	--dest-path PROJECT_NAME/RANDOM_SUBDIR

Use a random subdirectory name (not gold or trapping). This uploads via az login credentials and produces gold_clips_report_public.csv with public URLs. If az CLI is unavailable, fall back to upload-local mode.

For P.804, apply column renaming (gold_clips → gold_url) and add ver after URLs have been updated to public paths.

6. Generate trapping clips (if not provided)

Trapping clips can be generated from any rating clips — they do not need to be high-quality references (unlike gold clips). Download a sample of rating clips:

python utils\download_clips.py `
	--input RATING_CLIPS_PATH\rating_clips.csv `
	--column rating_clips `
	--output_dir RATING_CLIPS_PATH\trapping_source `
	--sample BEST_PRACTICE_TRAPPING_SOURCE_COUNT `
	--strategy random --seed 99 `
	--sas_token "SAS_TOKEN_VALUE"

Omit --sas_token for public storage. If no SAS token is available for private storage, fall back to az storage blob download --auth-mode login for each clip.

Use a different seed or strategy than gold to avoid overlap with gold source clips.

Clear the toolkit's trapping source directory and copy source clips there:

$trapSrc = "REPO_ROOT\src\trapping_clips_assets\source"
$trapOut = "REPO_ROOT\src\trapping_clips_assets\output"
Get-ChildItem $trapSrc -File | Remove-Item -Force
if (Test-Path $trapOut) { Get-ChildItem $trapOut -File | Remove-Item -Force }
Copy-Item "RATING_CLIPS_PATH\trapping_source\*.wav" $trapSrc -Force

Select the correct trapping config:

Study method Config file
acr configurations\trapping.cfg or configurations\trapping_p835.cfg
p835 configurations\trapping.cfg or configurations\trapping_p835.cfg
echo_impairment_test configurations\trapping.cfg
p804 configurations\trapping_p804.cfg

DCR and CCR do not use generated trapping clips — see the legacy note in "Supported test methods". For these methods, skip this section entirely and use reference clips as the trapping_clips.csv (column: trapping_clips only).

Run the trapping clip generator:

Set-Location REPO_ROOT\src
python create_trapping_stimuli.py `
	--cfg configurations\TRAPPING_CONFIG

Output goes to trapping_clips_assets\output\. The report is at trapping_clips_assets\output\output_report.csv with columns trapping_ans, trapping_clips.

Prepare for upload — use a random subdirectory name (not trapping):

python utils\copy_to_pub_storage.py upload `
	--input "trapping_clips_assets\output\output_report.csv" `
	--columns trapping_clips `
	--local-dir "trapping_clips_assets\output" `
	--account-name STORAGE_ACCOUNT_NAME `
	--target-container TARGET_CONTAINER `
	--dest-path PROJECT_NAME/RANDOM_SUBDIR

Copy the public CSV as trapping_clips.csv next to the rating clips.

6b. Review generated clips

[ASK] After generating and uploading gold, trapping, and training clips, ask: "Gold, trapping, and training clips are generated and uploaded. Would you like to review them before I run the master script, or should I continue?"

7. Create the project config

Create a .cfg file next to the input CSVs with the project name.

Template (values unquoted):

[create_input]
number_of_clips_per_session:10
number_of_trapping_per_session:1
number_of_gold_clips_per_session:GOLD_PER_SESSION
clip_packing_strategy: random
platform: PLATFORM

[hit_app_html]
allowed_max_hit_in_project:COMPUTED_VALUE
bw_min: FB
bw_max: FB
hit_base_payment:0.5
quantity_hits_more_than: COMPUTED_VALUE
quantity_bonus: 0.1
quality_top_percentage: 20
quality_bonus: 0.15
contact_email:USER_PROVIDED_EMAIL
# Screen-out completion code, paid on the platform for the screening effort. A single
# code covers both the qualification and setup screen-out (platforms such as Prolific
# support only one code per study). It is STUDY-SPECIFIC: the crowd platform issues it
# per study, so never copy it from another study's cfg. Ask the requester for this study's
# code, or leave unset to keep the ${screenout_code} placeholder for the HIT App Server to
# fill; with no code the participant returns the task.
#screenout_code: XXXXXXXX
# Online (in-HIT) grading of the qualification / setup sections (both default true).
# Set to false to grade that section only offline in result_parser (answers not embedded,
# check button hidden).
#run_online_eval_qualification: true
#run_online_eval_setup: true
# Objective listening-device check (acoustic echo test right after qualification).
# required_playback_device: headset | loudspeaker | any (default headset; "any" skips the check).
# device_check_threshold_db: net-coupling dB above which a loudspeaker is inferred (default 20).
# device_check_probe_gain: probe-tone level 0..1, kept low for hearing safety (default 0.1).
#required_playback_device: headset
#device_check_threshold_db: 20
#device_check_probe_gain: 0.1

Key rules:

  • number_of_gold_clips_per_session = 2 for P.804, 1 for others.
  • bw_min defaults to FB. Valid: NB-WB, SWB, FB.
  • screenout_code (optional) = completion code a screened-out participant enters on the platform to be paid for the screening effort (one code for both qualification and setup screen-out). It is study-specific — never copy it between studies; ask for each new study's code. If unset, the master script prints a warning and leaves a ${...} placeholder for the HIT App Server to fill; when no code is provided at all, the participant is asked to return the task.
  • run_online_eval_qualification / run_online_eval_setup (optional, default true) = grade the qualification / setup section in the browser. Set to false to hide that section's answers and "Check answers" button and grade it only offline in result_parser.
  • required_playback_device (optional, default headset) = the device the task requires: headset, loudspeaker, or any. An objective acoustic echo test verifies it right after qualification (and confirms audibility via a beep count to catch a muted device). With any the check is skipped. Tune device_check_threshold_db (default 20) and device_check_probe_gain (default 0.1) only if browser testing shows detection issues or the tone is too loud.
  • contact_email = user-provided. Never hardcode.
  • allow_cannot_rate / show_loudness_alert = requester-confirmed booleans for P.804, both defaulting to false in the toolkit. Include the confirmed values explicitly in [hit_app_html], together with required_playback_device.
  • allowed_max_hit_in_project = BEST_PRACTICE_ALLOWED_MAX_HITS.
  • For AMT only, quantity_hits_more_than ≈ floor(total_sessions / 2), at least 2; use the quantity and quality bonus defaults in the template.
  • For Prolific, set quantity_bonus and quality_bonus to 0 when creating a new config. Retain any bonus-threshold fields required by the toolkit for compatibility, but ignore them; do not adjust or warn about their relationship to the worker cap.
  • platform = the crowd platform (prolific or mturk), from the requester's answer (defaults to prolific). It is written into the generated *_result_parser.cfg so the result parser knows the platform; for prolific no bonus report is generated and the toolkit's quantity and quality bonuses do not apply.

8. Run the master script

Always include --check_urls and --create_local_test flags. URL checking validates that all clip URLs are accessible and catches broken links before publishing. The local test generates a preview HTML file for visual inspection.

--check_urls may be skipped only if this is a re-run and the URLs were already validated in a previous run (e.g. when re-running due to a config change).

Set-Location RATING_CLIPS_PATH
python REPO_ROOT\src\master_script.py `
	--project PROJECT_NAME `
	--method METHOD `
	--cfg PROJECT_CONFIG.cfg `
	--clips rating_clips.csv `
	--training_clips training_clips.csv `
	--gold_clips gold_clips.csv `
	--trapping_clips trapping_clips.csv `
	--check_urls `
	--create_local_test

For P.804 and pp835, also pass --training_gold_clips if a training gold clips CSV was provided or generated in step 4b:

python REPO_ROOT\src\master_script.py `
	--project PROJECT_NAME `
	--method p804 `
	--cfg PROJECT_CONFIG.cfg `
	--clips rating_clips.csv `
	--training_gold_clips training_gold_clips.csv `
	--gold_clips gold_clips.csv `
	--trapping_clips trapping_clips.csv `
	--check_urls `
	--create_local_test

Note: when --training_gold_clips is used, the --training_clips flag is not needed — training clips are embedded in the training gold CSV.

Add --general_assets REPO_ROOT\src\assets_master_script\general_assets_internal.csv when the requester chose the internal assets (input 11). Omit the flag to use the default general set (assets_master_script/general.csv).

Notes:

  • Use full absolute paths for all arguments to avoid path resolution issues.
  • The working directory should be the folder containing the input CSVs so that the project output directory is created there.
  • Supported --method values: acr, dcr, ccr, p835, echo_impairment_test, pp835, p804.
  • For AMT only, if quantity_hits_more_than triggers a warning, update the config file with the suggested value and re-run. Ignore this bonus warning for Prolific.

8b. Write a re-run batch file (regenerate_study.bat)

Always write a regenerate_study.bat next to the input CSVs so the requester can rebuild the study later (e.g. after a config tweak) without re-deriving the command. It must reproduce the exact master_script.py invocation from step 8, using %BASE% (the folder the .bat lives in) for all input paths and the absolute repo path for the script.

Include only the flags that were actually used: --training_gold_clips (P.804/pp835) or --training_clips (other methods), and --general_assets if an internal assets CSV was passed. Omit --check_urls (URLs were already validated on first run); keep --create_local_test so a preview is regenerated.

@echo off
REM Regenerate the METHOD study PROJECT_NAME from existing input CSVs.
REM Clips (rating, gold, trapping) are already uploaded to public storage; this
REM only rebuilds the HIT app, publish batch, result-parser cfg, and preview.

setlocal
set "BASE=%~dp0"
set "REPO=REPO_ROOT"
set "PROJECT=PROJECT_NAME"

cd /d "%BASE%"

python "%REPO%\src\master_script.py" ^
	--project %PROJECT% ^
	--method METHOD ^
	--cfg "%BASE%PROJECT_CONFIG.cfg" ^
	--clips "%BASE%rating_clips.csv" ^
	--gold_clips "%BASE%gold_clips.csv" ^
	--trapping_clips "%BASE%trapping_clips.csv" ^
	--create_local_test

endlocal

Save as regenerate_study.bat next to the input CSVs and confirm it runs.

9. Verify the generated project artifacts

The output project directory (PROJECT_NAME\) should contain:

File Purpose
PROJECT_NAME_METHOD.html HIT app (HTML) for the crowd platform
PROJECT_NAME_publish_batch.csv Session data with clip URLs for publishing
PROJECT_NAME_METHOD_result_parser.cfg Config for result_parser.py when analyzing results
url_mapping.csv Mapping of original (private/local) URLs to final public URLs

Verify:

  1. All three files exist.
  2. The publish batch CSV has the expected number of rows (sessions).
  3. The HTML file is non-empty.

9b. Generate URL mapping CSV

If clips were copied from private to public storage, generate url_mapping.csv in the project output directory mapping every original URL to its public URL.

Columns: original_url, public_url, clip_type (one of: rating, gold, trapping, training, training_gold).

Include all clip types that were uploaded or copied. For clips already public (e.g. training gold clips), set original_url = public_url.

Save as PROJECT_NAME\url_mapping.csv.

10. Clean up temporary files

Always remove:

  • tmp_gold.csv (debug artifact from master_script.py).
  • Downloaded source clips directories (gold_source\, trapping_source\).
  • Toolkit trapping directories (src\trapping_clips_assets\source\*.wav, src\trapping_clips_assets\output\*).

[ASK] Ask whether to also remove local gold_output\ (clips are already uploaded).

11. Handoff

Upload status: If the upload mode was used, gold and trapping clips are already uploaded and publicly accessible. If upload-local was used as a fallback (no az CLI), remind the requester to run the azcopy commands before publishing the study.

Handoff checklist:

  1. The project directory with all three artifacts.
  2. The config file used (saved next to input CSVs for future re-runs).
  3. The regenerate_study.bat (next to the input CSVs) for one-command re-runs.
  4. The azcopy commands for uploading generated clips (if applicable).
  5. The method and scale used.
  6. Any warnings or deviations from the documented flow.
  7. Instructions for the requester to publish on their chosen platform:
    • Prolific: follow the team's Prolific workflow or docs\running_test_prolific.md.
    • AMT: follow docs\running_test_mturk.md.

Utility scripts reference

Script Purpose
src\utils\download_clips.py Download clips from URLs in a CSV to local directory
src\utils\select_training_clips.py Select N evenly-spaced training clips from rating clips
src\utils\copy_to_pub_storage.py Upload clips to Azure Blob Storage (direct via az login) or prepare azcopy commands
src\utils\preview_html.py Generate local preview HTML from master script output
src\create_gold_clips.py Generate gold standard clips from clean source WAVs
src\create_trapping_stimuli.py Generate trapping stimuli by overlaying messages on source clips