1---
2title: "Post-mortem: A Backward Incompatible Database Migration"
3description: Val runs failed due to a database migration that was not backward compatible
4pubDate: 2025-04-02T00:00:00.000Z
5author: Sophie Houser
6---
7
8Today at 10:11am we experienced a 12-minute outage, which caused HTTP vals to return 503 errors and other types of vals to fail. In the end, the root cause was a deployment timing issue where database migrations were deployed successfully, but our application code deployment hung for several minutes. The new database migrations were incompatible with the old application code and crashed the process.
9
10We aim to make all database migrations maintain backward compatibility, but in this case, we only discovered through the delayed deployment feedback that the new migrations were not compatible with previous versions.
11
12## Timeline
27## Next Steps
28
29Reliability is important to us and we’ve taken steps to make sure this doesn’t happen again. We’ve added a test to ensure database migrations are backward compatible, which we’ll run before we deploy any new code that includes database migrations.
1import { generateCode } from "./backend/generate-code";
2import { createTables } from "./database/migrations";
3import { createProject, getCode, getNextVersionNumber, insertVersion } from "./database/queries";
4
5async function servePublicFile(path: string): Promise<Response> {
3import { generateShortCode } from "./shared/utils.ts";
4import {
5 initializeDatabase,
6 createShortUrl,
7 getRecentUrls,
8 findUrlByShortCode
9} from "./backend/database/queries.ts";
10
11export default async function(request: Request) {
14
15 try {
16 // Initialize database on each request
17 await initializeDatabase(import.meta.url);
18
19 // Serve client-side JavaScript
6
7/**
8 * Initialize the database schema
9 */
10export const initializeDatabase = async (metaUrl: string): Promise<void> => {
11 await sqlite.execute(`
12 CREATE TABLE IF NOT EXISTS ${TABLE_NAME} (
2import { generateShortCode } from "../shared/utils.ts";
3import {
4 initializeDatabase,
5 createShortUrl,
6 getRecentUrls,
7 findUrlByShortCode
8} from "./database/queries.ts";
9
10/**
12 */
13export default async function(request: Request): Promise<Response> {
14 // Initialize database on each request
15 await initializeDatabase(import.meta.url);
16
17 const url = new URL(request.url);
1# Database
2
3This directory contains database-related code.
4
5- `queries.ts` - Database operations for the URL shortener
4
5- `index.ts` - Main server entry point
6- `database/` - Database operations and queries
11## Project Structure
12
13- `/backend` - Server-side code and database operations
14- `/frontend` - React components and UI
15- `/shared` - Shared utilities and types
16- [ ] Rebuild as React Router?
17- [ ] opentownie as a pr bot
18- [ ] give it the ability to see its own client-side and server-side logs by building a middleware that shoves them into a SQL light database date and then give it a tool to access them
19- [ ] do a browser use or screenshot thing to give it access to its own visual output
20- [ ] Have it default to creating a new branch off main and then embedding and iframe to the resulting http val and give you a link to a pr opening url
7export const thinkTool = tool({
8 description:
9 "Use the tool to think about something. It will not obtain new information or change the database, but just append the thought to the log. Use it when complex reasoning or some cache memory is needed.",
10 parameters: z.object({
11 thought: z.string().describe("A thought to think about."),