Smoke Testing and UAT for Your AI-Built App

1. What is smoke testing

Smoke testing is a short set of checks that confirms that a new build works and is worth testing in more detail. The name comes from electronics development: you power the device on, and if no smoke comes out, you move on to real measurements.

A smoke test doesn't cover every feature. It checks only what the app can't be used without, like loading the home page, logging in, saving a record or sending an email. The whole run should take a few minutes at most, but you can often finish it in a few seconds.

2. Smoke testing vs. sanity testing vs. UAT

Smoke testing checks the whole app after a deployment, sanity testing checks one specific fix or change and UAT checks whether the app meets its users' requirements. They differ in who runs them and how long they take.

3. Smoke test checklist

Build your smoke test checklist from the main scenario you've described for the AI. If you wrote the spec with a PRD template, use the Main scenarios and Done when sections. Then run through the list after every change from the AI and after every deployment.

Here's an example for a web app with a login and a database. Adjust it to what matters for your app.

4. Smoke testing an AI-built app

Run a smoke test on an AI-built app after every larger change, not just before a release. While editing one screen, Lovable, Bolt or Cursor can change a shared component, the database schema or the login flow. The bug then shows up somewhere else in the app.

If a smoke test fails, tell the AI, or go back to the last working version and give the assistant the change again in a smaller scope, pointing out the problem the original solution caused. For that you need your code in Git and changes committed in small steps. You can also use the checklist to have the AI write automated tests from it, for example in Playwright. Even with automated tests, walk through the main scenario by hand now and then, because the AI may ignore situations that would bother you in the app.

5. User acceptance testing: how to let users test your app

User acceptance testing (UAT) is testing where real users or the client confirm that the app does what they need and approve the release. It runs in a test environment such as staging, not in production with real data.

1. 1

   Set up a test environment

   Deploy the version you want to verify to a dedicated environment. Switch payments to test mode and send emails only to test addresses or a dummy mailbox.

2. 2

   Create test accounts

   Prepare test accounts for testers with the roles they'll have in the live app. Test data should resemble production in volume, and you can anonymize sensitive data.

3. 3

   Write UAT test scenarios

   Tell testers what to go through. Don't describe click-by-click steps; give real scenarios, for example “book a meeting room for next Tuesday, then cancel the booking”. If something changed in the new version, let them know.

4. 4

   Agree on how to report bugs

   Pick one place for bug reports, such as a form or a task or bug tracker. For each bug, the tester should write the app version and the steps that led to it, and attach a screenshot or a short video.

5. 5

   Record the UAT sign-off

   When testing is done, write down which version users approved and which bugs need fixing before the release. Then deploy the approved version to production unchanged.

6. Keep the checklist, environments and tester links in AboutTheApp

Testers need to know the test environment URL, the test scenarios, the list of changes in the tested version and a place to report bugs. With AboutTheApp you can keep them together right in the app, so testers have everything in one place.

On the app's card, use Add resource to add the test scenarios (a document in your repo, Confluence, Google Docs, a project in test management software and more) under the Monitoring & Quality category and the bug report form or tracker under Issue Tracking. Mark links that not everyone should see as Private. They're shown only to signed-in team members and to people who enter the unlock code. Don't put test account passwords in AboutTheApp; share them with testers separately.

If the client tests the app, add them to AboutTheApp and assign the app with the test environment to them. Add links that belong only to that environment directly to it. Record the tested version in the Product versions window together with its list of changes so testers know what to check. If you add the About box to your app, testers get the version and the changes in it right in the app.

Frequently asked questions