ISJ hears exclusively from Jamie Boote, Principal Consultant at Black Duck about governing AI across the software development lifecycle.
The most important question in software security right now concerns the roles AI can perform within the software development lifecycle.
Every few years, a new technology arrives and reshapes the security conversation.
Generative AI is doing that now. Boardrooms want strategies. CISOs want frameworks.
Vendors are racing to establish positions around a technology whose long-term impact is still emerging.
Many discussions focus on the content AI can generate and the threats that content might create.
Those questions have value, but security teams gain more practical insight by examining which activities within the software development lifecycle can be disrupted, enhanced, or performed by AI.
This perspective shifts attention toward a familiar challenge.
Organisations have long evaluated contributors, determined appropriate levels of trust, defined access rights and established oversight mechanisms.
AI introduces a new type of contributor, which means many existing governance principles remain relevant.
Predictive versus generative: a distinction worth making
Any discussion of AI benefits from greater precision because the term covers technologies with very different capabilities.
Predictive AI, which has been embedded in security tooling for years, evaluates data against learned patterns to classify, score or flag activity.
Intrusion detection, code quality scanning and anomaly detection all fall into this category.
The system analyses available data, compares it with learned behaviour and produces an assessment.
Generative AI serves a different function.
It creates new content in response to prompts, including code, documentation, test cases, images and audio.
Since late 2022, advances in generative AI have attracted significant public attention because the quality of the output has reached a level that makes many forms of automation economically viable.
Both forms of AI affect the software development lifecycle and introduce distinct opportunities and risks.
Effective governance depends on understanding those differences.
Four roles AI now plays in the development lifecycle
Across the modern SDLC, including requirements, design, development, build and integration, delivery, deployment and production, AI is already operating in several distinct capacities.
As a contributor, generative AI can produce requirements documents, user stories, source code, test cases, build scripts and supporting content.
This use case has received considerable attention because it can shorten the path from an initial idea to a working prototype.
The output is often valuable as a first draft, but it remains constrained by training data and lacks the organisational context that shapes engineering decisions.
Human review and oversight remain essential elements of governance.
As an evaluator, predictive AI can assess artefacts at a scale and speed that would be difficult for human teams to match.
Vulnerability identification, code pattern recognition and dependency risk scoring are already integrated into many development environments.
Its strength lies in consistency and volume.
Its limitations arise when context is required.
An AI evaluator may identify a known risk pattern, yet determining the significance of that risk within a specific environment still requires human judgement.
As a repairer, AI is increasingly helping organisations move from problem identification to remediation.
Activities such as identifying an issue, analysing context, proposing a fix, implementing changes and validating outcomes can all benefit from AI assistance.
Self-healing software remains in an early stage of development, but the trajectory is clear.
Security leaders should already be considering the governance requirements associated with systems that can take corrective action with limited human involvement.
As a crown jewel, one of the most important AI roles receives relatively little attention in security discussions.
AI models themselves, along with the training data, model weights and intellectual property they represent, are valuable assets that require protection.
These assets should be developed securely, protected from tampering and managed alongside source code, trade secrets and other forms of intellectual property.
Supervision is not optional
Each of these AI roles introduces risks that security teams must address through governance and oversight.
Hallucinations remain a well-documented challenge.
Large language models can present incorrect information with complete confidence.
Names, dates, code libraries and security mitigations can all be fabricated convincingly.
Verification processes are therefore essential whenever AI-generated content influences business or technical decisions.
Training data risk deserves equal attention.
The quality and composition of training data directly influence model outputs.
Adversaries have a long-term incentive to manipulate publicly available datasets by increasing the prevalence of insecure coding patterns or reducing the visibility of secure ones.
Such efforts could shape future model behaviour in ways that create exploitable outcomes.
Trust boundaries also require careful consideration.
AI systems need access to information in order to generate useful outputs or perform meaningful analysis.
When those systems operate in the cloud, sensitive data may move beyond organisational control.
Information shared with a model becomes part of a broader trust relationship that organisations must evaluate carefully.
Access controls, vendor assessments and data protection requirements should be applied with the same rigour used for any third-party system handling sensitive information.
Development teams have always worked with contributors capable of making mistakes, introducing vulnerabilities or exposing sensitive information.
Organisations have established mature processes for managing those risks through code review, access controls, testing and governance.
AI fits within that broader framework and benefits from many of the same controls.
Extending the framework you already have
For security teams, the implications are significant but manageable.
Most of the governance capabilities required to address AI risk already exist within mature organisations.
Intellectual property protection, access management, vendor risk assessment, code review and secrets management all provide a foundation for governing AI models, training data and AI-generated artefacts.
At the same time, AI adoption across development environments continues to accelerate.
Developers are using coding assistants.
Pipelines are incorporating AI-generated test cases. Organisations are deploying models trained on internal data.
Security teams benefit from establishing governance early, while adoption patterns are still forming.
The software development lifecycle itself remains largely unchanged.
Requirements, design, development, integration, delivery and deployment continue to define how software is created.
What has evolved is the range of participants involved in those activities.
AI now performs tasks that were previously handled exclusively by people, creating new opportunities for efficiency while introducing governance responsibilities that security teams must address.
Organisations that approach AI through the lens of roles, responsibilities and trust relationships will find that many of the tools required for effective governance are already available.
The challenge lies in extending those practices to a new class of contributor and applying them consistently as AI capabilities continue to evolve.
