Executive Summary‍

Artificial Intelligence has fundamentally changed the economics of software development. Applications, integrations, dashboards, and complex business platforms can be created quickly, even by people with limited coding experience. However, even when AI-created code appears sophisticated and production-ready, there’s  organizational risk.

The answer to “Can we build it?” is usually yes, but the real question is “Should we build it if there’s an established tested, supported, and documented?”

Theres is an important distinction between software generation and software verification. AI has made it easy to produce code, but proving that software is correct, secure, and maintainable, is another story. AI-generated code may contain hallucinated libraries, incorrect APIs, insecure logic, vulnerable dependencies, or ineffective controls.

AI can generate code, recommend architectures, and even review its own output but the organization remains responsible for verifying that the output is correct, secure, and suitable for production. 

Human presence is not the same as human oversight.

The question is no longer build versus buy. It is what level of competent human oversight, verification, security, documentation, and long-term responsibility the organization is prepared to accept. AI dramatically reduces the cost to create software, but not the organizational responsibilities of owning it.

‍

1. The Build-or-Adopt Question

Historically, software development required deep technical knowledge of coding languages, databases, and architecture. Now, a user can describe a requirement in natural language and receive functional code almost immediately, including schemas, interfaces, APIs, and integrations.

AI has created extraordinary productivity improvements. However, an application can function during a demo and still have challenges demonstrating that it is:  

• secure, architecturally sound, and scalable
• recoverable and resilient to abnormal conditions
• maintainable by someone other than its original developer
• correctly licensed, documented, and compliant
• appropriately tested and free from unauthorized functionality

AI can help with these elements, but a human with the right experience and knowledge needs to verify that they have been achieved.

A department may ask, “Why buy a solution when AI can build one?” but that question often only looks at development cost, not total lifecycle ownership. An established solution typically bundles product, documentation, testing, maintenance, security updates, support, a roadmap, a knowledge base, trained personnel, a user community, and contractual accountability. An internally AI-built application may initially consist of little more than code, one developer, and an AI assistant. Those are very different solutions with very different risk profiles.

‍

2. Advantages of AI-Assisted Custom Development

Custom, AI-assisted development offers real advantages:

• Speed: Requirements can be converted into prototypes quickly.
• Lower initial cost: A single capable individual, supported by AI, can prototype what once required several developers.
• Alignment with business requirements: Custom systems can be built around an organization's own terminology, workflows, and data models rather than forcing the business to adapt to a commercial product.
• Control: Increased control over source code, hosting, architecture, security, and release schedules.
• Ongoing  innovation:  AI makes more experimentation economically viable.

‍

Risks can quickly compound when an experiments move to production without adequate human oversight from engineering or the required governance controls.

‍

3. The Verification Problem 

AI can produce answers that appear credible and inexperienced developers may not recognize errors, especially if they are using AI to verify the same code that AI created. 

This scenario is all too common:

• AI Generates Code 
• Developer Questions AI 
• AI Explains Why Its Code Is Correct
• Developer Accepts Explanation
• Code Enters Production

This can all happen without independent verification by someone technically capable of challenging the solution. 

Human oversight is the most important control in AI-assisted development 

There is a significant difference between having a human involved and having effective human oversight.  A person accepting AI-generated code without the competence to understand and challenge it provides no meaningful oversight. Effective oversight requires sufficient understanding, authority, independence, and accountability.

AI enables people to create applications and controls beyond their own technical competence. If a developer can’t explain the code without asking the AI, meaningful human oversight probably doesn't exist.

AI self-verification, i.e. using the same AI system to generate code and then to verify it, is not independent oversight. AI can easily repeat the same incorrect assumptions, hallucinations, or insecure decisions present in the original output. For higher-risk applications, oversight needs to be performed by someone capable of independently evaluating architecture and source code as well as data privacy, integrations, and regulatory requirements.

‍

The level of oversight needs to be proportionate to the application's risk. A simple internal productivity script does not require the same controls as an application processing sensitive information or supporting a critical business process. 

For higher risk applications, a stronger model is:

• AI-Assisted Developer
• Technical Peer Review 
• Security Review
• Testing
• Business/Process Owner Approval
• Production

AI-assisted applications require ongoing review of performance and security vulnerabilities. Every AI-generated modification is an opportunity for errors or unintended consequences to enter the environment. Human oversight is required for the full-lifecycle of the product, not just at the development stage. 

4. AI Hallucinations and Security Risk

Hallucinations can create vulnerabilities that extend well beyond incorrect answers. AI can generate technical content that appears sound but can have a number of underlying issues, including:

• nonexistent software libraries, incorrect package names, and hallucinated dependencies
• nonexistent or fabricated API functions and endpoints, and incorrect parameters
• outdated implementation methods and deprecated security functions
• invalid configuration options and incorrect database or authentication logic
• incorrect assumptions about data structures and ineffective error handling
• inappropriate encryption implementations and security controls that appear effective but can be bypassed

‍

Just because code runs correctly, doesn’t mean it’s correct.  It doesn’t take a crash for an application to create risks.  

Integration can fail to validate input. Database queries can expose more than intended. Logging function can omit what's needed to investigate an incident. Encryptions can encrypt data in an insecure way. 

5. Malicious Code, Dependencies, and Security Controls

AI-assisted development creates additional many ways for malicious or unauthorized code to enter the environment, especially when organizations introduce large volumes of machine-generated code without proper human oversight. Potential risks include:

• hidden backdoors, undocumented administrative accounts, and hard-coded credentials
• unauthorized external connections, hidden data transmission, and unnecessary telemetry
• excessive permissions, disabled security checks, and weakened authentication
• hidden execution routines and code executing only under specific conditions
• malicious functionality introduced through compromised third-party dependencies

‍

AI-generated code should be considered unverified code until appropriately reviewed and tested.

AI frequently recommends third-party libraries, packages, and frameworks, and these components expand the software supply chain. Organizations need to ensure each component originates from a legitimate publisher, is actively maintained, and has appropriate licensing. AI can easily hallucinate a package name and then a developer could install a malicious package deliberately registered under the hallucinated name. AI recommendations can’t be treated as verification of software dependency.

Risks increase when AI is used to generate security-sensitive code, such as authentication, authorization, encryption, or privacy controls. Encryption and authentication that appears to work may not be secure. This type of code needs a higher level of independent review and security testing.

‍

6. Ongoing Support and Documentation

An established, supported solution typically comes with technical support, escalation paths, service-level commitments, and documented recovery procedures. With internally generated code, who fixes it? The answer is often “the person who built it” but if that person leaves or forgets how it works, there’s a problem.  

AI makes code faster to produce than organizations can document it. Without strict governance, organizations will likely have applications without adequate:

• architecture documentation, data-flow diagrams, and API documentation
• database schemas, installation procedures, and configuration records
• operational and troubleshooting documentation, and disaster-recovery procedures
• security documentation, dependency inventories, and change histories

‍

Operating systems, browsers, and APIs change. Software must evolve as encryption and authentication requirements evolve. 

For internally developed AI-generated code, the organization becomes the software vendor.

The organization must decide who monitors vulnerabilities and dependencies, who evaluates and tests patches, who tracks external API changes, and who supports the application five years from now. AI can assist with this, but is not accountable for it. 

7. Developer Departure and Additional Risks 

Developers can resign, transfer, or leave inadequate documentation. This leaves the organization with source code that they don’t have the knowledge to safely operate or modify.

With AI-assisted development, developers can generate large amounts of code quickly without producing corresponding design documentation. Malicious functionality is also a possibility, including:

‍

• hidden administrative accounts, hard-coded credentials, and authentication bypasses
• backdoors, logic bombs, and time-triggered functions
• remote-access mechanisms and hidden data exfiltration
• deliberately weakened logging and malicious changes disguised as maintenance

‍

No business-critical application should depend upon the trustworthiness, availability, or knowledge of a single developer.

‍

AI-assisted risks can quickly compound in scenarios like this:

• Inexperienced Developer uses AI to Generate Large Amounts of Code
• AI Introduces Errors, Vulnerabilities, or Hallucinations 
• Developer Lacks Expertise to Detect Them
• Same AI Is Used to Verify the Code without  Independent Human Oversight
• Application Enters Production
• Business Becomes Dependent Upon It 
• Updates and Security Depend on One Person
• Developer Leaves
• Organization Inherits Code It Cannot Fully Understand

‍

8. Established Solutions Versus Custom Build

Established solutions have their own risks, but mature products do have advantages including:

• documented functionality, architecture documentation, and API documentation
• established security controls, penetration testing, and vulnerability management
• technical support, security updates, compatibility updates, and dependency management
• incident response, product roadmaps, and end-of-life management
• knowledge bases and user communities

Established software means you are purchasing full lifecycle capability. But this comes with licensing costs, vendor dependency, limited customizations, integration constraints, migration complexity, and roadmap dependency.

Understand which risks you are choosing and who owns those risks.

‍

AI reduces the effort required to write software. It does not reduce the responsibility required to own software.

9. The Hidden Cost and Regulated Environments

A prototype that appears 80 percent complete can easily look 100 percent complete. But verification testing, security, documentation, maintenance, and governance are still critical and often missing. For business-critical systems, this may cost more than creating the original application.

For organizations in regulatory environments, the risks are even greater.  ISO standards, privacy law, cybersecurity frameworks, financial or healthcare regulation, and government requirements must be followed.  Organizations must demonstrate where data resides, who accessed it, whether processing is correct, and what happens when failures occur. An auditor, regulator, customer, insurer, or management team may reasonably ask:

• Who approved and owned this application, and how much of it was AI-generated?
• Who independently reviewed the AI-generated code, and with what competence?
• How were hallucinations identified, and how were dependencies verified?
• How are vulnerabilities identified, and who manages updates and patches?
• How is the application recovered, and what happens if the developer leaves — could a departing developer have introduced malicious functionality?

‍

An application that cannot answer these questions has an assurance problem, regardless of whether it currently works.

10. Required Controls for AI-Assisted Applications

For business-critical internally developed applications, organizations should establish:

• Competent human oversight 
• Independent code review
• Risk-based approval that scales with application criticality and data sensitivity
• Independent verification of AI-generated APIs, libraries, and technical assumptions
• Central source-code ownership
• Change control
• Segregation of duties, so critical code is not created, approved, and deployed by the same person
• Validation of third-party components 
• Patch and update management
• Knowledge transfer and succession for developers
• Malicious-code review, with high-risk departures triggering targeted review 

‍

11. Management Questions

Before approving AI-assisted custom development where an established solution is an option, management should ask:

• What strategic advantage does building this ourselves provide?
• Are we comparing total lifecycle cost or merely development cost?
• How much of the application is AI-generated, and who possesses the competence to independently verify it?
• Who reviews security-sensitive code, and who supports the application when the original developer is unavailable?
• Who is responsible for ongoing security, maintenance, documentation, and enhancements?
• What happens if the original developer leaves?
• Can the organization defend the application before a customer, auditor, regulator, or insurer?

Conclusion

AI has changed software development, making it faster, cheaper, and more accessible but it has not eliminated the need for competent human oversight. Independent verification, security, documentation, support, maintenance,  governance, and lifecycle management are still critical and require involvement from humans with the right knowledge and expertise.

We know the answer to “Can AI build this for us?” is probably yes. The more important question is “Do we have people capable of understanding what AI built? And can they provide the validation, documentation, support, and governance needed to ensure it works correctly and securely?”  

AI can make almost anyone a software builder, but it does not remove the organization's responsibility for what gets built.

‍

Need Help Managing Your AI Your Risk?

Get in touch. We'd love to help.

‍

‍