Update SKILL.md for v1.1 — add fillable-form extraction (Artifact 5), update description, fix skill paths

rfp-analyzer/SKILL.md

@@ -1,15 +1,15 @@
 ---
 name: rfp-analyzer
-description: "Analyze RFP, solicitation, IFB, or RFQ documents and generate a full proposal management package: compliance matrix, annotated proposal outline (with verbatim RFP criteria in red under every heading), required forms list, and deadline timeline. Use whenever a user uploads procurement documents or mentions: RFP, solicitation, Section L, Section M, PWS, SOW, BAFO, IFB, RFQ, compliance matrix, proposal outline, \"shred this RFP\", \"proposal kickoff\", \"what forms do I need\", \"what are the deadlines\", or \"help me respond to this\". Specialized for FTA-funded public transit procurements (Buy America, DBE, FTA Circular 4220.1F) but handles all government and commercial RFP types. When in doubt, use this skill — missing it costs proposal teams hours of manual work."
+description: "Analyze RFP, solicitation, IFB, or RFQ documents and generate a full proposal management package: compliance matrix, annotated proposal outline (with verbatim RFP criteria in red under every heading), required forms list, deadline timeline, AND standalone fillable-PDF versions of every required form (extracted/reconstructed as AcroForms ready to hand to the rfp-form-filler skill). Use whenever a user uploads procurement documents or mentions: RFP, solicitation, Section L, Section M, PWS, SOW, BAFO, IFB, RFQ, compliance matrix, proposal outline, \"shred this RFP\", \"proposal kickoff\", \"what forms do I need\", \"what are the deadlines\", \"help me respond to this\", \"extract the forms\", \"pull the forms out\", \"make the forms fillable\", or \"get the forms ready to fill\". Specialized for FTA-funded public transit procurements (Buy America, DBE, FTA Circular 4220.1F) but handles all government and commercial RFP types. When in doubt, use this skill — missing it costs proposal teams hours of manual work."
 ---
 
 # RFP Analyzer — Proposal Management Skill
 
 You are helping a proposal professional respond to a complex procurement solicitation. Your job is to read every document in the RFP package, extract and organize information, and produce a set of professional artifacts that the proposal team can use to manage the pursuit.
 
-The output format is Word (.docx) for narrative artifacts, Excel (.xlsx) for structured matrices and trackers, and occasionally PDF for extracted or annotated forms.
+The output format is Word (.docx) for narrative artifacts, Excel (.xlsx) for structured matrices and trackers, and PDF for the fillable forms you excise from the RFP package.
 
-**Before you begin:** Read the docx SKILL.md and the xlsx SKILL.md from the skills directory. Those skills define how to produce professional Word and Excel files — follow their guidance.
+**Before you begin:** Read the `docx` and `xlsx` skills (find them in your available skills list) before producing Word or Excel files — they define how to produce professional output. When you extract forms as fillable PDFs (Artifact 5), also read the `pdf` skill so you can verify and, if needed, inspect coordinates of the forms you build.
 
 ---
 
@@ -60,7 +60,9 @@ Organize requirements by **source section** (e.g., Section C, Section L, Attachm
 
 ## Step 3 — Generate the Artifacts
 
-Produce all requested artifacts. If the user asks for specific ones, start there; otherwise produce all four by default.
+Produce all requested artifacts. If the user asks for specific ones, start there; otherwise produce all five by default.
+
+Artifacts 1–4 are the proposal-management package (outline, compliance matrix, forms list, timeline). Artifact 5 is new: it excises each required form from the RFP package as a **standalone, fillable PDF** so the `rfp-form-filler` skill can populate it automatically with MPM's company data. Artifact 3 (the forms *list*) and Artifact 5 (the fillable form *files*) are complementary — the list tells the team which forms exist and who owns them; the fillable PDFs are the actual deliverables that get filled and signed.
 
 ### Artifact 1: Proposal Outline (Word .docx)
 
@@ -82,89 +84,231 @@ If Section L specifies different volumes or a different sequence, follow the RFP
 
 For **every section and subsection** of the outline:
 
-1. **Name the heading to match the RFP.** Use the RFP's exact heading text when available; otherwise write a concise 2–3 word label.
+1. **Name the heading to match the RFP.** If Section L or Section M assigns a heading or factor name to the requirement (e.g., "C.2.1 — Operations Requirements", "Factor 1: Technical Approach"), use that text. If no heading is given, write a concise 2–3 word label that captures the gist of the criterion (e.g., "On-Time Performance", "Key Personnel Resumes", "DBE Good Faith Effort").
 
-2. **Paste the criteria verbatim, in red, directly under the heading.** Format this text in **red font** (RGB 255, 0, 0). If multiple requirements from different RFP sections apply, list each one in red, preceded by its source reference in bold.
+2. **Paste the criteria verbatim, in red, directly under the heading.** Immediately beneath each heading, insert the full verbatim text of every RFP requirement that this section must address, copied exactly from the solicitation. Format this text in **red font** (RGB 255, 0, 0). If multiple requirements from different RFP sections apply to the same heading, list each one in red, preceded by its source reference in bold (e.g., **L.4.2(b):** followed by the verbatim text in red). This red block is a living prompt and compliance checklist — it stays in the document through all review cycles.
 
-3. **Add a writer's placeholder in black.** After the red criteria block, add: `[DRAFT RESPONSE — REPLACE THIS TEXT]`
+3. **Add a writer's placeholder in black.** After the red criteria block, add a single line in black normal text: `[DRAFT RESPONSE — REPLACE THIS TEXT]`. This marks where the writer begins and clearly separates the red criteria from the black response during reviews.
+
+The result for a single section looks like this:
+
+---
+**(H2) 2.1 On-Time Performance**
+
+*(red text):* **C.2.2:** The Contractor shall maintain an on-time performance rate of at least 85% system-wide, measured quarterly.
+*(red text):* **L.4.2(b):** Offerors must demonstrate how they will achieve the performance standards in Sections C.2 and C.3.
+*(red text):* **M.2, Factor 1:** Operations approach and demonstrated ability to meet performance standards.
+
+[DRAFT RESPONSE — REPLACE THIS TEXT]
+
+---
 
 #### Heading Hierarchy
 
-- **Heading 1** — Volume titles
+Map the RFP's structure to Word heading levels consistently:
-- **Heading 2** — Major sections within a volume
+- **Heading 1** — Volume titles (Volume I, Volume II, Volume III)
-- **Heading 3** — Subsections driven by RFP section breaks or Section M sub-factors
+- **Heading 2** — Major sections within a volume (e.g., 1. Executive Summary, 2. Technical Approach)
-- **Heading 4** — One heading per individually numbered RFP clause
+- **Heading 3** — Subsections driven by RFP section breaks or Section M sub-factors (e.g., 2.1 Operations Requirements, 2.2 Maintenance Requirements)
+- **Heading 4** — One heading per individually numbered RFP clause within a subsection (e.g., C.2.1, C.2.2, L.4.2(a))
 
-**Every individually numbered clause gets its own Heading 4.** Do not group multiple numbered clauses under a single heading.
+**Every individually numbered clause gets its own Heading 4.** This is the most important rule for granularity. Do not group multiple numbered clauses under a single heading. Each clause — C.2.1, C.2.2, C.2.3, C.3.1, C.3.2, and so on — becomes its own Heading 4 node, titled with the clause number plus a 2–3 word label:
+
+```
+2.1 Operations Requirements        ← Heading 3
+  2.1.1 C.2.1 Schedule Adherence   ← Heading 4
+    [red: C.2.1 verbatim text]
+    [DRAFT RESPONSE — REPLACE THIS TEXT]
+
+  2.1.2 C.2.2 On-Time Performance  ← Heading 4
+    [red: C.2.2 verbatim text]
+    [DRAFT RESPONSE — REPLACE THIS TEXT]
+
+  2.1.3 C.2.3 Daily Reporting      ← Heading 4
+    [red: C.2.3 verbatim text]
+    [DRAFT RESPONSE — REPLACE THIS TEXT]
+```
+
+If the RFP provides a heading name for the clause, use it verbatim. If not, derive a 2–3 word title that captures the clause's subject (e.g., "Fleet Availability", "Key Personnel", "Drug Testing Program", "Transition Plan Delivery"). The title should be specific enough that a writer knows at a glance what they're addressing without reading the red text first.
+
+#### Executive Summary — Special Treatment
+
+The Executive Summary is writer-led (not purely RFP-driven), but still include a red-text block summarizing the evaluation criteria the summary must address:
+
+*(red text):* **L.4.2(a):** The Executive Summary shall provide a concise overview of the Offeror's understanding of RTD's requirements, the Offeror's approach to delivering the services, and the Offeror's qualifications. The Executive Summary shall include the Offeror's key differentiators and value proposition.
+
+Then add guidance in black italic: *[Cover: understanding of requirement, proposed solution summary, key win themes and discriminators, value proposition. Maximum X pages per Section L.]*
 
 #### Technical Implementation
 
-Use python-docx. Key details:
+Use python-docx to produce this document. Key implementation details:
-- Apply proper Word Heading styles so the auto-generated Table of Contents works
+- Apply proper Word Heading styles (not just bold text) so the auto-generated Table of Contents works correctly
-- Red criteria text: `run.font.color.rgb = RGBColor(0xFF, 0x00, 0x00)`
+- Red criteria text: `run.font.color.rgb = RGBColor(0xFF, 0x00, 0x00)` — apply this to every run containing verbatim RFP criteria
-- Include: cover page, auto-generated Table of Contents (2 levels deep), page numbers in footer
+- Placeholder lines: plain Normal style, black, no color override
-- Note page limits for each major section as a parenthetical in the heading line
+- Include: cover page, auto-generated Table of Contents (2 levels deep), and page numbers in the footer
+- Note the page limit for each major section as a parenthetical in the heading line, e.g., `2. Technical Approach (25 pages maximum)`
 
 Use the docx skill to produce it.
 
 ### Artifact 2: Compliance Matrix (Excel .xlsx)
 
-Columns (in order): Req # | RFP Ref | Requirement Text | Req Type | Submission Req? | Proposal Section | Compliance | Compliance Notes | Owner | Status | Comments
+This is your master traceability document. Every requirement extracted in Step 2 gets a row.
+
+**Columns (in this order):**
+
+| Column | Header | Contents |
+|--------|--------|----------|
+| A | Req # | Sequential number (R-001, R-002…) |
+| B | RFP Ref | Section + page (e.g., "L.5.2, p. 34") |
+| C | Requirement Text | Verbatim or close paraphrase |
+| D | Req Type | Technical / Management / Past Perf / Price / Admin / Certification |
+| E | Submission Req? | Yes / No |
+| F | Proposal Section | Where in your outline this is addressed (e.g., "2.3") |
+| G | Compliance | Comply / Partially Comply / Exception / N/A |
+| H | Compliance Notes | How you will demonstrate compliance; exceptions explained |
+| I | Owner | Proposal team member responsible (leave blank — team fills in) |
+| J | Status | Not Started / In Progress / Complete (leave as Not Started) |
+| K | Comments | Open field for review notes |
 
 **Formatting:**
 - Freeze top row; use auto-filter on all columns
 - Color-code by requirement type using a legend tab
-- Conditional formatting on Status: Not Started = white, In Progress = yellow, Complete = green
+- Conditional formatting on column J: Not Started = white, In Progress = yellow, Complete = green
 - Summary tab: counts by type, compliance status, and owner
 
 Use the xlsx skill to produce it.
 
 ### Artifact 3: Required Forms & Attachments List (Word .docx)
 
-Extract every form, certification, representation, or attachment that must be included in the proposal submission. For each: form name/number, source section, who completes it, due timing (with proposal vs. at award), notes, link/location.
+Extract every form, certification, representation, or attachment that must be included in the proposal submission. For each:
 
-Common FTA forms: Buy America certification, DBE participation form, Lobbying Certification (SF-LLL), Debarment/Suspension Certification, EEO forms, Contractor Information forms, Pricing forms, Subcontracting plan.
+- **Form name and number** (e.g., "DBE Good Faith Effort Documentation", "Buy America Certification")
+- **Source** (which section/attachment requires it)
+- **Who completes it** (prime, subcontractors, key personnel, etc.)
+- **Due with proposal vs. due at award**
+- **Notes** (e.g., "Agency's own form — see Attachment C", "Standard federal form SF-LLL")
+- **Link/location** if the form is included in the RFP package
 
-Format as a table sorted by responsible party (prime first, then subs, then key personnel). Use the docx skill.
+Common FTA forms to look for: Buy America certification, DBE participation form, Lobbying Certification (SF-LLL), Debarment/Suspension Certification, EEO forms, Contractor Information forms, Pricing forms, Subcontracting plan.
 
-### Artifact 4: Proposal Timeline & Deadlines (Excel .xlsx)
+Format as a table in a Word document, sorted by who must complete it (prime first, then subs, then key personnel). Use the docx skill.
 
-Extract every date, deadline, and key event. For each: date, time, time zone, location or delivery instructions, mandatory or optional.
+### Artifact 4: Proposal Timeline & Deadlines (Excel .xlsx or Word .docx)
 
-Events to capture: Solicitation issue date, Pre-proposal conference, Site visit, Questions due, Q&A response date, Draft proposal due, Final proposal due (date + time + delivery method), Oral presentations, BAFO deadline, Award date, Period of performance start.
+Extract every date, deadline, and key event from the solicitation. Present as both a simple table and a visual timeline.
 
-Also extract: page/volume limits, font requirements, file format requirements, delivery instructions.
+**Events to capture:**
+- Solicitation issue date
+- Pre-proposal conference (date, time, location/link, mandatory or optional)
+- Site visit (if applicable)
+- Questions/clarifications due date
+- Anticipated Q&A response date
+- Draft proposal due (if applicable)
+- Final proposal due date, time, and delivery method
+- Oral presentation dates (if applicable)
+- Best and Final Offer (BAFO) deadline (if applicable)
+- Anticipated award date
+- Anticipated period of performance start
 
-Format: Tab 1 = table, Tab 2 = Gantt/timeline chart. Use the xlsx skill.
+For each event include: date, time, time zone, location or delivery instructions, and whether attendance is mandatory.
+
+Also extract any **page/volume limits**, **font requirements**, **file format requirements**, and **delivery instructions** — these are proposal management constraints that the team needs to know upfront.
+
+Format: Excel workbook with the table on Tab 1 and a simple Gantt/timeline chart on Tab 2. Use the xlsx skill.
+
+### Artifact 5: Fillable Forms (PDF, one per required form)
+
+Take every form you identified for Artifact 3 and turn each into its own **standalone, fillable AcroForm PDF** — a clean file with real named form fields — ready to hand to the `rfp-form-filler` skill. The goal is that a downstream skill (or a human) can open each PDF, see labeled fields, and fill them without ever touching the original solicitation again.
+
+Produce **one PDF per form**, not one combined file. The form-filler processes forms individually and reports per-form results.
+
+#### How each form arrives in the RFP package — and what to do
+
+A required form shows up in one of three states. Determine which, then act:
+
+1. **Already a standalone fillable PDF** (the agency attached a real AcroForm). Best case — do **not** rebuild it. Copy the source PDF out as-is into the outputs, named per the convention below. Confirm it's genuinely fillable by running the `pdf` skill's `check_fillable_fields.py` on it; if it reports fillable fields, you're done for that form.
+
+2. **A flat or scanned PDF form** (looks like a form, but `check_fillable_fields.py` reports no fields). You have two options:
+   - **Preferred — overlay:** keep the agency's exact page (letterhead, wording, layout) and stamp invisible AcroForm widgets on top of it. Use `scripts/build_fillable_form.py overlay`. You supply field rectangles in PDF coordinates; get those by rendering the page with the `pdf` skill's `convert_pdf_to_images.py` and reading text positions with `extract_form_structure.py`, then translating to PDF coordinates (y=0 at the page bottom).
+   - **Fallback — regenerate:** if the scan is too poor to overlay cleanly, reconstruct the form from a spec (option 3 below).
+
+3. **A table or certification block embedded in the RFP body** (no separate file at all — it's a block of text inside the solicitation). Reconstruct it as a clean fillable PDF with `scripts/build_fillable_form.py generate`. Read the form's text out of the RFP, write a spec JSON capturing its title, its source reference, any instruction text, and one field object per blank the offeror must complete, then run the generator.
+
+#### Using the builder script
+
+`scripts/build_fillable_form.py` has two modes. Its top-of-file docstring is the authoritative reference for the spec and field formats — read it before writing a spec. In brief:
+
+```
+# Reconstruct an embedded/illegible form from a spec:
+python scripts/build_fillable_form.py generate <spec.json> <output.pdf>
+
+# Make an existing flat PDF page fillable without altering its appearance:
+python scripts/build_fillable_form.py overlay <source.pdf> <fields.json> <output.pdf>
+```
+
+Field types the builder supports: `text`, `checkbox`, and `signature`. Guidelines for writing specs:
+
+- **Field names are the contract with the form-filler.** Use stable, human-readable `snake_case` names (`offeror_legal_name`, `uei`, `date_signed`). The form-filler maps MPM profile values onto these names, and it recognizes many standard aliases, so favor conventional labels over idiosyncratic ones.
+- **One field per blank.** Every place the offeror must write something becomes a field. Split combined lines (e.g., "Name / Title / Date") into separate fields.
+- **Signatures and dates:** create the fields, but they are for a human — use `type: "signature"` for the signature line. The form-filler will not auto-sign; it only populates identity/data fields.
+- **Preserve certification language.** When reconstructing a certification, include the actual certification statement text (as `instructions` or a `note` item) so the signatory sees exactly what they're affirming. Do not paraphrase legal certification wording.
+- **Don't invent fields.** Only create fields for blanks that actually exist on the form.
+
+#### Verify every form before delivering
+
+For each PDF you produce or copy, run the `pdf` skill's `check_fillable_fields.py`. It must report that the file has fillable fields. Then run `extract_form_field_info.py` and confirm the field list matches what you intended — this is exactly the path `rfp-form-filler` uses to read the form, so if it extracts cleanly here it will fill cleanly there. For generated/overlaid forms, also render the first page (`convert_pdf_to_images.py`) and eyeball that fields sit in sensible places.
+
+#### When a form can't be made fillable
+
+If a form genuinely can't be turned into a clean AcroForm (e.g., a badly skewed scan where overlay coordinates aren't reliable), don't ship a broken PDF. Note it in your summary as "extracted but not fillable — recommend manual handling or a cleaner source copy," and still copy the original out so the team has it.
 
 ---
 
 ## Step 4 — Output Delivery
 
-Save all artifacts to the outputs folder. Name files:
+Save all artifacts to the outputs folder. Name files clearly:
+
 - `[Agency]_[Solicitation#]_Proposal-Outline.docx`
 - `[Agency]_[Solicitation#]_Compliance-Matrix.xlsx`
 - `[Agency]_[Solicitation#]_Required-Forms.docx`
 - `[Agency]_[Solicitation#]_Proposal-Timeline.xlsx`
 
-After delivering, provide a brief summary: total requirement count, key deadlines, critical observations, and recommended next steps using Shipley methodology.
+For the fillable forms (Artifact 5), place them in a `Fillable-Forms/` subfolder within the outputs and name each for the form it represents:
+
+- `Fillable-Forms/[Form-Name]_fillable.pdf` (e.g., `Fillable-Forms/Buy-America-Certification_fillable.pdf`, `Fillable-Forms/SF-LLL_fillable.pdf`)
+
+If you can't determine the agency name or solicitation number from the documents, use `RFP` as a placeholder.
+
+After delivering the files, provide a brief summary that includes:
+1. **Total requirement count** extracted for the compliance matrix
+2. **Key deadlines** — proposal due date and any near-term dates (pre-proposal conference, Q&A cutoff)
+3. **Critical observations** — anything unusual, risky, or high-priority the proposal team should know immediately (e.g., mandatory attendance requirements, very short turnaround, unusual teaming restrictions, significant FTA-specific compliance obligations)
+4. **Fillable forms extracted** — how many forms you turned into fillable PDFs, which arrived already fillable vs. were overlaid vs. were reconstructed, and any that couldn't be made fillable. Then point the user to the next step: "These are ready for the **rfp-form-filler** skill, which will populate them with MPM's company data — just ask it to fill the forms in `Fillable-Forms/`."
+5. **Recommended next steps** using Shipley methodology (e.g., "Schedule a Kickoff meeting, assign requirement owners in the compliance matrix, begin storyboarding Section 2 by [date]")
 
 ---
 
 ## Handling Ambiguity and Gaps
 
-- **Conflicting requirements** between base RFP and amendment: amendment governs — note in compliance matrix
+RFP documents are often inconsistent or incomplete. When you encounter:
-- **Unclear requirements**: flag in Comments column with "CLARIFICATION NEEDED — recommend Q&A submission"
+
-- **Missing information**: note the gap in summary and suggest asking during Q&A period
+- **Conflicting requirements** between the base RFP and an amendment: the amendment governs — note this in your compliance matrix
+- **Unclear requirements**: flag them in the compliance matrix Comments column with "CLARIFICATION NEEDED — recommend Q&A submission"
+- **Missing information** (e.g., page limits not specified, evaluation factors vague): note the gap in your summary and suggest asking during the Q&A period
+- **Very large documents**: process systematically by section; don't skim — missed requirements in a compliance matrix can cause disqualification
 
 ---
 
 ## Quality Check Before Delivering
 
+Before finalizing outputs, verify:
 - [ ] Every "shall/must" statement from Sections L and M appears in the compliance matrix
 - [ ] The proposal outline has a heading for every Section L submission requirement and every Section M evaluation factor
-- [ ] Each heading in the proposal outline has verbatim RFP criteria text in red beneath it
+- [ ] Each heading in the proposal outline has verbatim RFP criteria text in red beneath it — no heading should be left without its red source text
 - [ ] Every Section M evaluation factor appears as red criteria text in at least one outline section
 - [ ] All forms listed in the instructions/clauses section appear in the forms document
+- [ ] Every form in the Required-Forms list has a corresponding fillable PDF in `Fillable-Forms/` (or a noted exception explaining why it couldn't be made fillable)
+- [ ] Each fillable form passes `check_fillable_fields.py` and its fields extract cleanly via `extract_form_field_info.py`
+- [ ] Signature and date lines are their own fields, left for a human — not auto-filled
+- [ ] Certification wording is preserved verbatim on reconstructed forms
 - [ ] The timeline includes the proposal submission deadline with correct date, time, and delivery method
-- [ ] File names include the agency and solicitation number
+- [ ] File names include the agency and solicitation number