Assessment 3 — Model Customization

Brings a custom-imported model into the same chat app as the four native Bedrock foundation models, and fine-tunes it on the Bitext customer-support dataset. Everything below is live behind the same CloudFront + Cognito pair as the rest of the app.

1. Base Model

Qwen 2.5 1.5B Instruct, imported via Amazon Bedrock Custom Model Import (CMI). Chosen because:

Well inside the 3B parameter cap the assessment specifies
Ungated on Hugging Face — no license-acceptance workflow
Qwen 2.5 architecture is officially supported by Bedrock CMI
Chat template is documented and stable

2. Fine-Tuning Approach (L250)

Supervised LoRA fine-tune on the Bitext customer-support chatbot training dataset (26,872 rows). The task is intent extraction and action-plan generation — for each customer message the model should emit a single JSON object: {"intent", "category", "response"}.

Setting	Value
Technique	LoRA (r=16, α=32) on q_proj, k_proj, v_proj, o_proj
Epochs	1
Effective batch size	16 (batch 4 × grad accum 4)
Learning rate	2e-4 with cosine schedule, 3% warmup
Precision	bf16 with gradient checkpointing
Split	95 / 5 train / val (25,528 / 1,344)
Training platform	SageMaker HuggingFace DLC, 1× `ml.g5.xlarge` (A10G 24 GB)
Training time	57 minutes billable, ~$1.34

Adapters are merged back into the base weights at the end of training so CMI can import a standard HF-format model — CMI does not accept LoRA adapters directly.

3. Latency and Cost — vs Native Bedrock Models

3 prompts × 3 runs = 9 measurements per model, medians reported. Measured end-to-end through the production chat Lambda, same code path the UI uses.

Model	TTFT	Total	Out tokens	Cost/req
Claude Sonnet 4.6 (native)	1.06s	5.17s	207	$0.003147
GPT-OSS 20B (native)	0.39s	1.42s	242	$0.000060
Qwen 2.5 1.5B (CMI base)	0.18s	1.23s	216	$0.000048
Qwen 2.5 1.5B (CMI Bitext FT)	0.22s	0.75s	112	$0.000027

The fine-tuned Qwen model is the fastest and cheapest of the four: 5× lower TTFT than Claude Sonnet, ~7× lower total time, ~116× lower cost per request. The speed-up over the base Qwen is entirely from shorter replies — the FT model learned the Bitext dataset's concise response style.

Cost note: CMI is actually billed per Custom Model Unit per 5-minute window, not per token. The figures above use the same per-token estimate we show in the chat UI so columns are directly comparable; at our traffic volume the two numbers are within ~10% of each other.

4. Five Example Inputs — Fine-Tuned Model

Produced by invoke_ft_model.py against the imported FT model.

[1] (1.13s) I want to cancel my subscription please

{"intent": "cancel_account", "category": "ACCOUNT", "response": "Thank you for reaching out about canceling your subscription. I understand that you may have changed your mind or need assistance in unsubscribing from our services. To proceed with the cancellation process, could you please provide me with some additional details? Specifically, I would like to know the email address associated with your subscription so that I can locate your account information accurately. Once I have this information, I will guide you through the steps necessary to cancel your subscription successfully."}

[2] (0.74s) Where is my refund? It's been 3 weeks.

{"intent": "track_refund", "category": "REFUND", "response": "I understand your frustration about the delay in receiving your refund. I apologize for any inconvenience caused. To provide you with an accurate update on the status of your refund, could you please provide me with your {{Order Number}} or {{Refund Reference}}? With this information, I will be able to assist you further and ensure that your refund is processed promptly."}

[3] (0.56s) Can you help me change the shipping address on my order?

{"intent": "change_shipping_address", "category": "SHIPPING", "response": "Sure! I can assist you in changing the shipping address for your order. Please provide me with the current shipping address and the new address you would like to update it to."}

[4] (0.68s) I got charged twice for the same item.

{"intent": "track_invoice", "category": "INVOICE", "response": "Thank you for bringing this to our attention. I apologize for the inconvenience caused by the duplicate charges on your account. To assist you further, could you please provide me with some additional details such as the {{Invoice Number}} or any other relevant information? This will help us investigate the issue promptly and ensure that it is resolved in a timely manner."}

[5] (1.66s) How do I reset my password?

{"intent": "recover_password", "category": "ACCOUNT", "response": "I'll get right on it! I understand that you're looking for guidance on how to reset your password. Let me walk you through the process step by step:\n\n1. Start by visiting our platform's \"{{Login Page URL}}\".\n...\nIf you encounter any difficulties during the process or if you have any further questions, feel free to reach out to us."}

Every reply is well-formed JSON with canonical Bitext intent labels (track_refund, recover_password, change_shipping_address, etc.) and the matching uppercase category (REFUND, ACCOUNT, SHIPPING). The dataset's {{placeholder}} convention for customer-specific details comes through in replies 2 and 5 — that's verbatim from the training set.

5. Trade-offs Observed

Base vs fine-tuned

The base Qwen model happily free-forms an answer in prose. The fine-tuned model always emits the JSON schema, uses the Bitext intent vocabulary, and picks up the dataset's stylistic quirks (explicit empathy statements, {{placeholders}} for unknown customer data, numbered step-by-step walkthroughs). Useful for a structured pipeline; less good for open-ended chat.

CMI vs native

Latency: CMI wins — our 1.5B Qwen is 5× faster TTFT than Claude Sonnet because it's small and runs on less-contended hardware.
Quality: On general reasoning the native models win. For the narrow customer-support task we trained on, the FT model is more reliable because the native models don't follow the JSON schema as strictly.
Platform maturity: CMI has rough edges. The Converse API docs claim Qwen 2.5 support but our imported model rejects Converse — we had to use InvokeModel with the Qwen chat template. Streaming responses report inputTokenCount: 0, so we estimate it ourselves (see footnote). Idle models need 30-60s to restore after eviction, which is why we run a 5-minute keep-warm Lambda against both CMI models.
Cost dynamics: At low traffic the per-token estimate for CMI is cheapest. At scale CMI's flat-rate CMU billing flips the comparison — high-throughput workloads favour CMI, low-traffic favours on-demand native.

6. Pipeline Summary

Base import (L200)
  HF snapshot (Qwen 2.5 1.5B)
    → upload to S3 (2.9 GB)
      → CreateModelImportJob (~2-10 min)
        → ARN in cmi.auto.tfvars
          → terraform apply (Lambda env + frontend dropdown)

Fine-tune (L250)
  Bitext CSV (27K rows)
    → JSONL with {intent, category, response} targets
      → SageMaker HuggingFace DLC, ml.g5.xlarge (~60 min, ~$1.50)
        → LoRA merged into base weights in /opt/ml/model
          → tar.gz → unpack_and_stage.py → flat HF in S3
            → CreateModelImportJob (~13 min)
              → ARN in cmi.auto.tfvars → terraform apply

7. Automated Evaluation (L250)

Account 577638392815 is granted s3:GetObject and prefix-scoped s3:ListBucket on the CMI bucket's two model prefixes (qwen2.5-1.5b-base/, qwen2.5-1.5b-bitext/) for the SUP SA daily leaderboard. No other actions, no access to training data.

See also: Eval Dashboard to run the LLM-as-judge test suite across all models including the two CMI variants.