Export/Import is broken — 150MB file bloated by embedded vectors
In recent weeks, the export and import of ai-character-chat have become increasingly broken.
A file that previously compressed to around 40-50MB for many long chats across several characters now compresses to 150MB even for a single character with a single chat. This makes export nearly impossible: Chrome crashes, Firefox freezes for 30+ minutes without completing, and only Safari manages to finish — but the file it generates cannot be imported back into Chrome or Firefox.
I investigated the exported JSON and found the cause:
The lore table stores a 768-dimension embedding vector alongside every single lore entry.
Example of a single lore row:
text: "Hammer remembers the cavern where echoes were his only companions..."
embeddings: { "Xenova/bge-base-en-v1.5": [-0.0752, 0.0361, 0.0114, ... ] }
With 88,111 lore rows, each carrying a 768-float embedding, the math is:
88,111 × 768 floats × ~8 bytes = ~540MB of embedding cache
out of a total export size of 580MB uncompressed. The actual user data (characters, messages, threads) is only ~2MB.
These embeddings are a regenerable cache — they are recomputed automatically and serve no purpose in a backup file. They should be excluded from exports entirely, or at minimum made optional.
I also tried to manually strip the embeddings field from the exported JSON to test the import, but the importer rejected the modified file as invalid — so the fix needs to come from the export/import code itself, not from the user side.
Proposed fix
Strip the embeddings field from lore rows during export. This would bring the file size back to ~4MB, making export and import fast and reliable across all browsers.