Val Town Code SearchReturn to Val Town

1---
2title: "Post-mortem: A Backward Incompatible Database Migration"
3description: Val runs failed due to a database migration that was not backward compatible
4pubDate: 2025-04-02T00:00:00.000Z
5author: Sophie Houser
6---
7
8Today at 10:11am we experienced a 12-minute outage, which caused HTTP vals to return 503 errors and other types of vals to fail. In the end, the root cause was a deployment timing issue where database migrations were deployed successfully, but our application code deployment hung for several minutes. The new database migrations were incompatible with the old application code and crashed the process. 
9
10We aim to make all database migrations maintain backward compatibility, but in this case, we only discovered through the delayed deployment feedback that the new migrations were not compatible with previous versions.
11
12## Timeline
27## Next Steps
28
29Reliability is important to us and we’ve taken steps to make sure this doesn’t happen again. We’ve added a test to ensure database migrations are backward compatible, which we’ll run before we deploy any new code that includes database migrations.

ValTownForNotionexample-database-page.ts31 matches

@bradnoble•Updated 6 days ago

20  const data = payload?.data;
21  const pageId = data?.id;
22  const databaseId = data?.parent?.database_id;
23
24  // this endpoint is for a specific database page and is triggered by a button in that database page
25
26  // the button that calls this endpoint sends a payload with the id of the database page, and the id of the parent database
27  // (e.g., pageId and databaseId from the payload)
28
29  // different than other endpoints, this one does not get called by a reset button or the cron reset
30  // instead, the reset button and the reset cron target the database, not the page
31  // so, we don't have a reset call coming in with the x-blob-key custom header to this enpoint
32  // nor do we have a x-container-title header that holds the title of the target object coming from the button payload
33
34  // both the reset button and the reset cron delete all database pages, and then add back fresh pages
35  // the endpoint for that is /example-database-pages (note the "s" at the end)
36
37  // different than other endpoints, we get the id of the block to modify (ie, the database page) from the button payload
38  // so, we don't need to do the things we do at other endpoints to get the id of target block
39  // (e.g., search the blob cache, or traverse notion page objects)
40
41  // but, we do need to set the blob for the resets
42  // and for that we'll need the id of the page in which this database lives
43  // and an identifier for the database
44
45  // can that idenfifier be the databaseId rather than the database title?
46  // the reset button sends the database title to the .../pages endpoint in order to find the id of the database
47  // but the matching there will traverse the notion page database objects looking for a title match
48  // which has nothing to do with the database title in the blob
49
50  // note: if you're wondering why none of the notion buttons have hard-coded ids for the targeting
53  // as is, i can clone a page with examples for a guest and everything will work without anyone needing to edit button settings
54
55  // when a user adds a favicon to a database page, we need to capture in the blob cache that a page in this database has been updated
56  // so that resets know which database to reset
57  // but the payload of the favicon button in this page does _not_ send the id of the page in which the database lives
58  // nor does it send the database title
59  // so we'll need to get those to make the blob key
60
67    for (const item of items) {
68      const blobData = await blob.getJSON(item.key);
69      if (blobData?.id === databaseId) {
70        // console.log(item.key, blobData);
71        blobData.key = item.key;
82
83  // blob for cache
84  // get the parent page id of this child_database
85  // this is the user's page and we need it for the blob key
86  const parentPageId = blobject?.user
87    ? blobject?.user?.page_id
88    : await notionHelpers.getDatabaseParentPageId(databaseId);
89
90  // we also need the database title
91  // even though we have the database id and it's tempting to use it here
92  // we need the title in the blob key so that the reset button can work
93  // b/c the reset button can only send the title in the x-container-title header
94  // with no other way to tie the button to the database
95
96  // const databaseTitle = blobject?.key
97  //   ? (blobject?.key).split("--").pop()
98  //   : await notionHelpers.getDatabaseTitle(databaseId);
99  const databaseTitle = await notionHelpers.getDatabaseTitle(databaseId);
100
101  console.log("databaseTitle:", databaseTitle);
102
103  // store webhook data in blob storage for resets
105    "slug": slug,
106    "clientPageId": parentPageId,
107    // "containerId": databaseId, // slugify databaseTitle here?
108    "containerId": await helpers.slugify(databaseTitle),
109  });
110  await blob.setJSON(blobKey, {
113      page_id: parentPageId,
114    },
115    id: databaseId,
116    date: new Date(),
117    content: askingFor,

ValTownForNotionexample-database-pages.ts9 matches

@bradnoble•Updated 6 days ago

16  const slug = await c.req.url.replace("https://", "");
17  const askingFor = c.req.headers.get("x-asking-for") || "default"; // val.town or default
18  const containerTitle = c.req.headers.get("x-container-title") || "good web sites"; // should map to the database title
19  const payload = await c.req.json();
20  const pageId = payload?.data?.id;
38
39  // when the reset button is clicked, it sends the x-container-title header
40  // we use this header to a) find a blob or b) if no blob, find a database with a title that matches it
41  // a) find a blob
42  // b) find a database with a title that matches it
43
44  // blob
64  // console.log("blockId:", blockId);
65
66  const databaseId = blobject?.id || (await notionHelpers.findChildDatabaseBlocks(pageId, containerTitle))[0].id;
67  // console.log("databaseId:", databaseId);
68
69  // actions
70  await notionHelpers.deleteAllDatabasePagesParallel(databaseId);
71  // add back pages; fresh start for the next person to add favicons
72  // create new database pages from default object
73  await notionHelpers.createDatabasePagesParallel(databaseId, defaults);
74  // store webhook data in blob storage for resets
75  await blob.setJSON(blobKey, {
78      page_id: pageId,
79    },
80    id: databaseId,
81    date: new Date(),
82    content: askingFor,

ValTownForNotionnotionHelpers.ts23 matches

@bradnoble•Updated 6 days ago

7});
8
9export async function createDatabasePagesParallel(databaseId: string, pages: []) {
10  const creations = pages.map((page) =>
11    notion.pages.create({
12      parent: { database_id: databaseId },
13      properties: {
14        "Website": {
42}
43
44// delete all database rows
45export async function deleteAllDatabasePagesParallel(databaseId: string) {
46  const pageIds = await notion.databases.query({
47    database_id: databaseId,
48  });
49  // Step 2: Delete all pages in parallel
59}
60
61export async function getDatabaseId(databaseTitle: string) {
62  // getDatabaseId(databaseTitle)
63  const database = await notion.databases
64    .query({ filter: { property: "title", title: { equals: databaseTitle } } });
65  return database.results[0].id;
66}
67
68export async function getDatabaseTitle(databaseId: string) {
69  // getDatabaseTitle(databaseId)
70  const database = await notion.databases.retrieve({ database_id: databaseId });
71  // console.log(database.title[0].plain_text);
72  return database.title[0].plain_text;
73}
74
75export async function getDatabaseParentPageId(databaseId: string) {
76  let currentBlockId = databaseId;
77
78  while (true) {
121}
122
123export async function findChildDatabaseBlocks(rootBlockId: string, blockIdentifier: string) {
124  const matchingBlocks = [];
125
140      // }
141      if (
142        block.type === "child_database" && block.child_database?.title && (!block.archived || !block.in_trash)
143      ) {
144        const databaseTitle = await helpers.slugify(block.child_database?.title.toLowerCase()) || "";
145        const identifier = await helpers.slugify(blockIdentifier?.toLowerCase());
146
147        if (databaseTitle === identifier) {
148          console.log(databaseTitle, identifier);
149          matchingBlocks.push(block);
150        }

9- [ ] make it one click to branch off like old jp townie demos
10- [ ] opentownie as a pr bot
11- [ ] give it the ability to see its own client-side and server-side logs by building a middleware that shoves them into a SQL light database date and then give it a tool to access them
12- [ ] do a browser use or screenshot thing to give it access to its own visual output
13- [ ] Have it default to creating a new branch off main

7export const thinkTool = tool({
8  description:
9    "Use the tool to think about something. It will not obtain new information or change the database, but just append the thought to the log. Use it when complex reasoning or some cache memory is needed.",
10  parameters: z.object({
11    thought: z.string().describe("A thought to think about."),

192```
193├── backend/
194│   ├── database/
195│   │   ├── migrations.ts    # Schema definitions
196│   │   ├── queries.ts       # DB query functions
251  ```
252
253### Database Patterns
254- Run migrations on startup or comment out for performance
255- Change table names when modifying schemas rather than altering

jeevessetupTelegramChatDb.ts2 matches

@brontojoris•Updated 6 days ago

1// Script to set up the telegram_chats table in SQLite
2// Run this script manually to create the database table
3
4export default async function setupTelegramChatDb() {
25    `);
26
27    return "Telegram chat database table created successfully.";
28  } catch (error) {
29    console.error("Error setting up telegram_chats table:", error);

jeevesREADME.md3 matches

@brontojoris•Updated 6 days ago

13## Technical Architecture
14
15**⚠️ important caveat: the admin dashboard doesn't have auth! currently it just relies on security by obscurity of people not knowing the url to a private val. this is not very secure. if you fork this project and put sensitive data in a database you should think carefully about how to secure it.**
16
17Stevens has been designed with the utmost simplicity and extensibility, much like a well-organized household. At the heart of his operation lies a single "memories" table - a digital equivalent of a butler's meticulous records. This table serves as the foundation for all of Stevens' operations.
45- `dashboard`: the admin view for showing the memories notebook + visualizing imports
46- `dailyBriefing`: stuff related to sending a daily update via telegram
47- `dbUtils`: little one-off scripts for database stuff
48
49## Hiring your own Stevens
57- For the Google Calendar integration you'll need `GOOGLE_CALENDAR_ACCOUNT_ID` and `GOOGLE_CALENDAR_CALENDAR_ID`. See [these instuctions](https://www.val.town/v/stevekrouse/pipedream) for details.
58
59**important caveat: the admin dashboard doesn't have auth! currently it just relies on security by obscurity of people not knowing the url to a private val. this is not very secure, if you put sensitive data in a database you should think carefully about how to secure it.**
60
61Overall it's a simple enough project that I encourage you to just copy the ideas and run in your own direction rather than try to use it as-is.

jeevesREADME.md2 matches

@brontojoris•Updated 6 days ago

4
5* `index.ts` - this is the **entrypoint** for this whole project
6* `database/` - this contains the code for interfacing with the app's SQLite database table
7
8## Hono
26## CRUD API Routes
27
28This app has two CRUD API routes: for reading and inserting into the messages table. They both speak JSON, which is standard. They import their functions from `/backend/database/queries.ts`. These routes are called from the React app to refresh and update data.
29
30## Errors