logaritmisk/biggus-dickus
1
0
Fork 0
Code Issues 17 Pull Requests Actions Packages Projects Releases Wiki Activity

On-write search re-index after catalogue/visibility writes #17

New Issue
Closed
opened 2026-06-02 12:06:13 +00:00 by logaritmisk · 2 comments
logaritmisk commented 2026-06-02 12:06:13 +00:00
Owner
Copy Link
Copy Source

Context

The search crate (Plan 6) provides index_object / remove_object / reindex_all, but on-write sync is not wired — the index is only rebuilt via reindex_all. Plan 7 added set_visibility and the public read API, which makes index freshness more visible (a record published/unpublished should appear/disappear from search).

Meilisearch is not transactional with Postgres, so this is best-effort eventual consistency; reindex_all remains the recovery path.

What to do

At the API/service layer, after a catalogue create/update/delete/set_object_fields/set_visibility commits:

  • call search::SearchClient::index_object (re-project the object) or remove_object as appropriate;
  • log failures rather than failing the request (the write already committed);
  • consider: a record leaving public may need to drop from a future public-search index (public search itself is post-MVP — search already stores visibility as filterable).

References

  • crates/search/src/lib.rs — index_object / remove_object / reindex_all / build_document
  • crates/db/src/catalog.rs — the write paths
  • Plans: docs/plans/2026-06-02-search.md, docs/plans/2026-06-02-publishing-public-api.md (Notes for follow-on plans)
## Context The `search` crate (Plan 6) provides `index_object` / `remove_object` / `reindex_all`, but **on-write sync is not wired** — the index is only rebuilt via `reindex_all`. Plan 7 added `set_visibility` and the public read API, which makes index freshness more visible (a record published/unpublished should appear/disappear from search). Meilisearch is not transactional with Postgres, so this is best-effort eventual consistency; `reindex_all` remains the recovery path. ## What to do At the API/service layer, after a catalogue create/update/delete/`set_object_fields`/`set_visibility` **commits**: - call `search::SearchClient::index_object` (re-project the object) or `remove_object` as appropriate; - log failures rather than failing the request (the write already committed); - consider: a record leaving `public` may need to drop from a future public-search index (public search itself is post-MVP — `search` already stores `visibility` as filterable). ## References - `crates/search/src/lib.rs` — `index_object` / `remove_object` / `reindex_all` / `build_document` - `crates/db/src/catalog.rs` — the write paths - Plans: `docs/plans/2026-06-02-search.md`, `docs/plans/2026-06-02-publishing-public-api.md` (Notes for follow-on plans)
logaritmisk commented 2026-06-02 21:02:06 +00:00
Author
Owner
Copy Link
Copy Source

Scope note from the admin-CRUD work (merged main @ c4e0c4c): several new catalogue write paths now exist that mutate object data but do not trigger a search re-index, and should be covered by this issue alongside the publish/visibility writes:

  • POST /api/admin/objects (create)
  • PUT /api/admin/objects/{id} (update)
  • DELETE /api/admin/objects/{id} (delete)
  • PUT /api/admin/objects/{id}/fields (flexible-field replace)

All live in crates/api/src/admin_objects.rs. On-write reindexing should hook these in addition to the existing set_visibility path.

Scope note from the admin-CRUD work (merged `main` @ `c4e0c4c`): several **new catalogue write paths** now exist that mutate object data but do not trigger a search re-index, and should be covered by this issue alongside the publish/visibility writes: - `POST /api/admin/objects` (create) - `PUT /api/admin/objects/{id}` (update) - `DELETE /api/admin/objects/{id}` (delete) - `PUT /api/admin/objects/{id}/fields` (flexible-field replace) All live in `crates/api/src/admin_objects.rs`. On-write reindexing should hook these in addition to the existing `set_visibility` path.
logaritmisk commented 2026-06-02 21:30:58 +00:00
Author
Owner
Copy Link
Copy Source

Implemented and merged to main (d15afda, cosmetic follow-up 4921c73).

What landed:

  • search::SearchClient::sync_object(db, id) — one uniform path: re-project + index_object if the object exists, remove_object if it's gone. Covers create/update/delete/fields/visibility with a single existence check.
  • AppState.search: Option<SearchClient> + a best-effort reindex(state, id) helper that logs failures via tracing::error! and never fails the committed write (reindex_all stays the recovery path).
  • Reindex wired after commit on all five write paths: create_object, update_object (when existed), delete_object (when existed), set_fields, set_visibility — success branches only.
  • Server config: MEILI_URL / MEILI_MASTER_KEY / MEILI_INDEX (default objects); indexing enabled only when URL+key are set, else disabled with a warn (no-op).
  • Tests: search/tests/sync.rs (index-then-remove) and api/tests/reindex.rs (HTTP create → indexed, delete → removed). Full workspace green.

Follow-up filed for the inline-reindex latency (background task). Closing.

Implemented and merged to `main` (`d15afda`, cosmetic follow-up `4921c73`). **What landed:** - `search::SearchClient::sync_object(db, id)` — one uniform path: re-project + `index_object` if the object exists, `remove_object` if it's gone. Covers create/update/delete/fields/visibility with a single existence check. - `AppState.search: Option<SearchClient>` + a best-effort `reindex(state, id)` helper that logs failures via `tracing::error!` and **never fails the committed write** (`reindex_all` stays the recovery path). - Reindex wired after commit on all five write paths: `create_object`, `update_object` (when existed), `delete_object` (when existed), `set_fields`, `set_visibility` — success branches only. - Server config: `MEILI_URL` / `MEILI_MASTER_KEY` / `MEILI_INDEX` (default `objects`); indexing enabled only when URL+key are set, else disabled with a warn (no-op). - Tests: `search/tests/sync.rs` (index-then-remove) and `api/tests/reindex.rs` (HTTP create → indexed, delete → removed). Full workspace green. Follow-up filed for the inline-reindex latency (background task). Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
logaritmisk
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: logaritmisk/biggus-dickus#17