Skip to content

Spec-Driven Development: Let's Get Situated ​

Software development has gone through a major renovation in the last few years with the innovation and integration of LLMs into our standard processes. IBM Bob, Cursor, Claude Code, and other coding assistants have, at least from an outside perspective, reduced the barrier to entry and sped up coding. However, after many months of dedicated research and development with these new tools, our teams at IBM have come to a central conclusion that the broader development community has also settled on: these tools, while powerful, have gaps. In this article, I will dive into these gaps, and our proposed solutions for each, with the caveat that we are continuing to improve upon our solutions. Just as the velocity of AI research continues to accellerate, so too do our methods for using these powerful tools.

Gaps and solutions ​

Intro to SDD ​

Working with AI to accelerate and inform our development has generated the need for optimizing and standardizing how we work with these agents. LLMs can be incredibly useful but are unnecessarily verbose, often get sidetracked in longer conversations, and lack the critical thinking needed for many development projects. The widely-adopted approach to curtailing some of AI's more negative qualities has so far been spec-driven-development (SDD), a classic solution for a modern problem. In short, SDD is a "software methodology in which a detailed specification of implementation details is authored and agreed upon before development begin"; to learn more about SDD, take a look at Anna Gutowska's What is spec-driven development?.

The Current SDD Landscape ​

For years, the software industry has relied on documentation-based methods to keep development in line with intention of the designers while preserving knowledge for future projects. However, SDD has taken on a new meaning with the advent of generative AI, and has been leaned on more than ever before; standards like GitHub's speckit or Fission-AI's OpenSpec have sought to add structure to the non-deterministic output that comes with integrated coding assistants. Building on the classic idea of SDD, these solutions (what we call SDD Frameworks) install spec files and other supporting documents alongside the code itself, adding precious contextual knowledge that would otherwise be missing when developing with AI.

These frameworks are great at reducing knowledge drift, the first of our principle gaps. To varying degrees, every existing framework captures information that would otherwise be lost, or at best, assumed (from reading through code) by the LLM, into markdown documents that live beside the code itself. These documents serve a few purposes: primarily, they inform the agent about about the who/what/why of subsections of the codebase, allowing for better planning of new projects as well as knowledge gathering of existing material; secondarily, these documents also create an english readout of code, potentially opening the codebase to a wider array of understanding and further bridging the gap between PM and development. From a knowledge drift perspective, existing SDD frameworks earn their keep.

Our team has tested out existing SDD strategies but, while all present similar core pillars, these structures lack certain features that we at IBM have come to realize are integral to our development. While investigating solutions to these issues, our team has designed our own custom SDD framework, which is being tested and slowly rolled out to teams across IBM ALM. Our framework seeks to address the following existing issues with other SDD frameworks and with agentic development in general: rule/validation drift, agent efficiency, multi-repo understanding, and intention drift.

Rule and Validation Drift ​

Setting up an integrated coding agent with a company repository, while enticing, can lead to some disastrous consequences without guardrails, given that an agent has no concept of "unwritten" or "written" rules of development that human software developers learn as they onboard into their new teams. Existing SDD frameworks attempt to remedy this by introducing documents like constitution.md (we call it RULES.md), a file that holds non-negotiable coding standards along with repo and team conventions. This file idealy holds the agent to well-defined boundaries and should make sure that the non-determinicity of agent outputs has some bounds within which it settles. In theory, this concept holds merit. However, in testing, our team has found that despite instructions within AGENTS.md or other grounding documents, agents tend to go "off the rails", forgetting these "non-negotiable" instructions as they get burried further and further away in the agent's context window. This can lead to results that, when not reviewed closely, can directly contradict the very rules that teams spend so long iterating over in order to safeguard their development practices.

To fix this issue, our team has built a more structured validation system, focused on keeping instructions fresh in the "mind" of the coding agent. Our approach combines a few pre-existing methods for superior results:

  1. RULES.md, to define grounding rules of engagement for an agent to follow when working within the repository. This setup is similar to pre-existing SDD frameworks in that it defines non-negotiable rules, but is enhanced by the rest of the validation system in the points below. Additionally, we introduce a stacked RULES.md system, where GitHub Organizations keep up org-level rules in a central repository, while individual repos specify repo-specific principles at the repo root and optional sub-level RULES.md cover more granular regulations. In order to keep this multi-level rule system alive, our golden rule is: higher-level rules always win. See Multi-Repo Understanding for details on how we enable this multi-repo rules system. Multi-Level RULES.md Structure
  2. Lifecycle Hooks, for shorter reminders after certain important events. These can be hard checks like "Now validate generated code against RULES.md after event x and run test.py on uncommitted files" or softer reminders like "Always check plans against RULES.md before completing the planning phase". We use these to keep important instructions at the front of our agent's context window, and have found dramatic upgrades in expected behavior; our testing has shown that when instructions are injected into the context at important lifecycle hooks, such as the PostToolUse or Stop hooks, agents always treat these as top-level instructions, instead of reminders at the start of new conversations, which are often ignored. We have also been testing with validation shell and python scripts that can be run at the end of a turn, in order to run deterministic checks on the agent's outputs; these have performed well in limited testing.
  3. SDD Agent Mode, which augments the agent's role within the repository to add more system prompt reminders to follow the RULES.md. Modes can also be used to restrict certain tooling, so we are in continuous development in order to find the best ways to utilize this.

Along with these three mechanisms, we also rely on the AGENTS.md file for more behavioral instructions for the agent; these are less about non-negotiables for planning and coding and more about ways of interacting with humans, proper note-taking, and other reminders. The system we have outlined has markably improved our outputs, with the use of hooks being the primary source of these gains. We can track attention to instructions by following how (and how often) IBM Bob calls tools, runs validations, and performs other necessary checks we have instructed it to. Our system nearly guarantees that IBM Bob will run required checks, and we have seen a large increase in the agent's ability to check itself during processing as it validates its on-the-go decisions with the ruleset we have provided.

Agent Efficiency ​

One of the biggest problems when interacting with agents today, with SDD frameworks or without, is an agent's propensity to use expensive searching mechanisms like glob and grep searching (let's call these expensive searches). These are inaccurate and clog up the context window, expending valuable tokens in the process. To this end, we introduce the concept of yellow-paging; we have condensed the sdd framework files within the repository into an file optimized for agent comprehension, and have supplied IBM Bob with a skill (sdd-navigate) that instructs the agent how to traverse the file. In addition, we use a UserPromptSubmit hook to insert a reminder into Bob's context:

'SDD REMINDER: If answering this prompt requires gathering knowledge from this repo (its behavior, design, history, dependencies, interfaces, or pending changes), call use_skill sdd-navigate first, before reading yellow-pages.yaml, grep, or glob. Only use an expensive search such as grep or glob if you have already isolated the file you are looking for. If you do not need to gather repo knowledge to answer, do not use sdd-navigate.'

This small reminder tells Bob to always search using this skill first, which has dramatically reduced the amount of expensive searching the agent has done in our testing scenarios. The yellow-page document should have all the information and context that the agent needs to isolate which files it needs to look through in greater detail, instead of randomly searching for phrases or keywords. Our testing has so far been run on three separate scenarios:

  1. Small, one folder repository, with three diagnostic questions (separate session each): in this scenario, the average cost (in bobcoins) dropped by 45% between runs with yellow-paging versus without. It also took 30% less time to answer our questions, used 56% fewer tool calls, and used only 1 expensive search, compared to 13 in just three questions without yellow-paging.
  2. Larger, multi-folder, multi-level repo, comprising of source code, build pipelines, etc, with a barebones SDD structure installed (without our optimizations), with the same three diagnostic questions (separate session each): here, we saw a 17% reduction in bobcoin cost, a 22% reduction in time to answer, 58% fewer tool calls, and 3 expensive searches compared with 17 needed without yellow-paging.
  3. Same setup as #2, tested with 7 questions on purpose, architecture, request flow, dependencies, build conventions, and history, asked in one session to test longer conversations: in this scenario, we saw an 18% reduction in bobcoin cost, an 18% reduction in time to answer, 33% fewer tool calls, and 14 expensive searches compared with 25 without yellow-paging.

While testing is still on-going, these results are notable and suggest that developers can make huge gains in efficiency using this proposed yellow-page structure. Further work is required to optimize the yellow-page document further for agent understanding.

Multi-Repo Understanding ​

Current SDD frameworks generally assume that they will be installed in mono-repos, where all dependent/related code and information is packaged into one repository. This assumption is unrealistic for companies like IBM, whose development and deployment strategies rely on a complex web of interconnected repositories. Our current solution to this issue relies on the GitHub/GitHub Enterprise MCP Server, a free-to-use and readily available tool that GitHub exposes. Our IBM Bob mode (SDD Agent) instructs the agent to make calls to related repos when considering changes and planning new features, based on a dependency map that is built on installation of our framework. Bob can then reach into potentially-affected repositories and grab the remote yellow-pages file in order to find related information that could help Bob to come up with any items that may need to be changed in related repositories, as well as to gain a more complete picture of how the current repository is used in the product ecosystem. This also allows us to place certain org-level files in a central repository that all other repositories can reference through the GitHub Enterprise MCP, such as stacked RULES.md documents or other reference material.

Intention Drift ​

Agents excel at understanding what code is doing and writing solutions to accomplish a goal. However, they are much worse at understanding the why: why a particular exception to a rule exists, why a decision was made. We have therefore sought to preserve the intention behind code changes with a combination of an agreed-upon structure for project overviews from PM to development, as well as an archived version of completed projects. These are the first and last portions of our PM -> Dev pipeline, whose goal is to set guardrails and expectations around the design and implementation process in order to create better code solutions, identify test cases, and keep the intention behind changes for future reference. After a PM has handed off the project template, and the developer has worked with Bob to create a plan, update code and framework documents, and human review gates have been completed, the plan template is compacted into an entry in the historical changes section of the sdd docs folder, which is then referenced whenever Bob needs to understand more about a particular code segment. In this way, developers and Bob will be less likely to change an intentionally-written piece of code without understanding the downstream effects.

What's Next ​

As our team continues to iterate on these solutions, we hope to eventually onboard more and more repositories within IBM ALM to our new framework. The reality is that there are often preferences for particular pre-existing framework solutions, so our next phase will be transitioning the aforementioned implementations into a shell that can augment either our own original base framework, or any other option (speckit, OpenSpec, etc). We hope to be able to ship this as an improvement on any basic SDD structure, with all of the bells and whistles that have so far improved the areas of need that we have identified. Beyond this structural change, we are also working on more ways to evaluate these new methodologies, as well as a more defined end-to-end pipeline, in order to iron-out the lifecycle of a project from idea, to development, to testing, and finally to production.