Sluice

No catalog

Many photo apps keep a database. Your pictures sit in folders somewhere underneath, but the truth lives in the app. The albums, the tags, which file is which. Move a file behind the app’s back and the link breaks, and getting your photos out means an export.

The other common kind of tool is a viewer. It shows you what’s there and leaves the tidying to you.

Sluice does neither. It does the tidying, and it keeps no database while doing it. The library is plain folders:

  • Photos/2024/07 and so on, one folder per month
  • Videos/ the same way
  • one flat Funny/ folder

That’s all of it. Open it in Explorer or Finder, copy it to a drive, back it up with whatever you already use. Nothing in there needs Sluice to make sense. If Sluice vanished tomorrow, your library would still be a library.

Anything still waiting on a decision lives in a separate working folder, never in the library. Photos a vision model flagged for your review go there, and so do files with no usable date. The library only ever holds what you decided to keep.

The decision got written down on 26 August, during a review of the wording on the app’s validation messages. It opens this devlog because nearly everything later builds on it.

What it costs

With no database to fall back on, the folders themselves are the safety system. You meet that as validation, setups Sluice won’t accept. It rejects plenty, from a missing argument to an unreadable folder. Four of those checks come straight from having no database:

  • Your library and your Inbox may not overlap. The Inbox is the folder new photos land in. Neither may sit inside the other.
  • One unfinished sift per time range. A sift is the step where a vision model looks at your photos and sorts them into keep, review and so on. If one is still sitting on disk unfinished, a second over the same months is refused. Damaged ones count too, because they’re the ones most worth protecting.
  • The library can’t move while unfinished sifts are on disk.
  • A settings change that would move your library asks first what should happen to the duplicate index. More on that index at the end.

Every one of these is the design working, and every one is a wall somebody will walk into.

It also gives up things catalogs do well. A folder tree can’t put one wedding photo in “2019”, “family” and “portraits” at once. A catalog searches a big library faster, and can keep an edit history. Sluice doesn’t try to be where you browse and search. That’s a scope choice, and the losses are real.

The check that guards the only copy

The first check on that list matters most, and it exists because of a bug.

Sluice’s sort step deletes Inbox files whose exact bytes are already in your library. A file is identified by a SHA-256 hash of its full contents, never by its name or size. The duplicate from your phone backup goes away, and the copy in your library stays.

In late July, a full review of the codebase found what happens when the library sits inside the Inbox. Sluice records every photo it puts in the library. So when sort walks the Inbox, and the Inbox contains the library, each library photo finds its own record and looks like a duplicate of itself. Sort would delete them. Each one was the only copy.

Two setups. Allowed: the Inbox and the library side by side. Rejected: the library inside the Inbox, where sort reaches every library photo and treats each as a duplicate of itself. Allowed: side by side Inbox Library Rejected: the library inside the Inbox Inbox Library Sort walks the Inbox, library included. Each matches itself, so each is deleted.
Sort deletes Inbox files whose bytes are already in the library. With the library inside the Inbox, every library photo matches itself. Sluice now rejects this setup in either direction.

Nothing checked for that. Now Sluice refuses the setup outright. It compares the two folders by where they really are on disk, following symlinks and Windows junctions, not by how their paths are spelled. The check caught something on its very first run. One of the app’s own tests used the current folder as the library, the Inbox and the working folder all at once. The check rejected that setup, and it was right to.

That fix raised the question the rest of this entry is about. If the safety system rests on “are these the same folder”, how many ways can that question be answered wrong?

Three ways to get “same folder” wrong

Over the next four weeks, at least three. An AI agent wrote Sluice’s code, a separate review agent read each change, and I signed off every commit. Each of these bugs got past at least one of those three. None reached a user, since all of it happened before release.

A lock that let go of itself. Sluice locks its working folder so two copies of the app can’t work in it at once. In mid-August, a change made a settings save hold both the old and the new folder’s lock until the save succeeded. Sensible. But the settings code decided whether old and new differed by tidying the path text. The lock decided “same folder” by resolving where the path really points.

On a Mac, letter case doesn’t count, so /Users/me/Sluice and /Users/me/sluice are one folder. A symlink and the folder it points to work the same way. Change one spelling to the other in settings, and the save treated it as a move. It asked the lock for the “new” folder, which the lock already held, so that did nothing. Then it released the “old” one, which was the only lock there was.

One folder spelled two ways. The settings code saw different text and treated it as a move. The save locked the new folder, which was already held, then released the old one, which was the same lock. Afterwards no lock was held. /Users/me/Sluice /Users/me/sluice Settings: the text differs, so the folder moved. Lock: one folder on disk, already held. Before the save 1 lock held 1. Lock the “new” folder Already held, so nothing happens 1 lock held 2. Release the “old” folder Same folder, so the only lock goes 0 locks held Sluice keeps running with no lock at all.
The save took the new folder's lock before releasing the old one. With one folder spelled two ways, both steps reached the same lock.

The app kept running with no lock at all, which was exactly the state the change was meant to prevent. Before review, the agent had tested its own work. It ran five checks that each undid part of the fix on purpose, to confirm a test would notice. Then it ran the full test suite and the IDE’s inspections. Everything passed, and it had to. Those checks can only test rules the code already has. Here the problem was two parts of the code disagreeing about what “the same folder” means, and no rule covered that.

The review agent caught it in its first round, before the change was ever committed. The fix counts lock holders per real folder, so a second spelling can’t release the lock anymore. The settings code still compared paths as text, though, and that came back three days later.

A username that only exists on someone else’s machine. Three days later, a test failed on the Windows machine that runs the project’s automated builds. It passed everywhere else. That machine’s user folder is C:\Users\runneradmin. Windows still keeps an eight-character short name for long folder names, a leftover from DOS, so the same folder can also be written C:\Users\RUNNER~1. My own username is short enough to need no short name. On my machine both spellings are the same string, so this test could never fail there.

On the build machine, two paths reach one folder: the long name runneradmin and the short name RUNNER~1. With a short username, both forms are the same string, so there is only one path. The build machine C:\Users\runneradmin C:\Users\RUNNER~1 one folder Two spellings, so two pieces of code can disagree. A short username C:\Users\dev one folder Its short name is the same string. The bug can't appear.
Windows still gives a long folder name an eight-character short name, a leftover from DOS, so the build machine's user folder answers to two paths. A username short enough to need no short name has only one.

The test itself was wrong, but fixing it uncovered a real bug in the same spot. When you save your settings, Sluice checks whether you moved your library by comparing the old and new paths as text. Say the library was saved as C:\Users\RUNNER~1\Photos, and the next save named it C:\Users\runneradmin\Photos. Same folder, different text, so Sluice treated an ordinary save as a library move. It then asked what to do with the duplicate index, when the library wasn’t going anywhere.

The fix resolves both paths to their real form at the moment they’re compared, and changes nothing else.

A file visited twice. About a week later, the same DOS short name broke something else on the build machine. You can give import whole folders, single files, or both, and Explorer’s search view lets you pick a folder and a file inside it together. The test’s temp folder came with the short spelling, so import got:

  • the folder C:\Users\RUNNER~1\Trip
  • the file C:\Users\RUNNER~1\Trip\beach.jpg

Walking the folder, import found the file under its real path, C:\Users\runneradmin\Trip\beach.jpg. The file named directly kept its short spelling. One photo, two paths, so import treated it as two files. When it was moving files rather than copying them, the first visit moved beach.jpg. The second visit then tried to read a file that was already gone.

The agent’s first test for this gave import a folder and a file inside it, and checked that the file was imported once. The agent said the test would catch the bug on any machine. Then it removed the fix and ran the test again, and the test still passed. My user folder’s name is five letters, short enough that Windows gives it no short name. On my machine, walking the folder and naming the file both give the same path, so import sees one file with or without the fix. That test could only ever fail on a machine like the build server.

The replacement test doesn’t wait for the bug to happen. It checks that import looks up a file’s real path before comparing it with the files it found in the folder. Remove the fix and that test fails, on my machine too.

Three bugs, one cause. In each, two parts of the code had different ideas of when two paths are the same folder. A catalog avoids this by giving every file one ID and comparing IDs. Without a catalog, every piece of code that compares folders has to get it right on its own.

The one file that describes your library

Sluice does keep one file of its own about your library. It’s a list of hashes, library-hashes.csv, in the working folder and never in the library itself. It’s what makes deleting a duplicate safe, because it records which exact bytes are already in your library. It holds no tags, no albums and no thumbnails. You don’t need it to open or browse your library.

It doesn’t trust itself blindly either. Each hash is stored with the path where Sluice put the photo, and a hash only counts if a file still exists at one of those paths. Say you delete a photo from the library by hand, and a copy of it later turns up in the Inbox. Sluice won’t delete that copy as a duplicate, because the original is gone. It checks that the file is still there, but not that its contents are unchanged. That’s why Sluice assumes nobody edits photos inside the library.

But nothing can rebuild it. It’s only ever added to, one row per file Sluice puts in the library. Delete it and every photo is still there and still in order, but duplicate detection starts from zero. The next time you import the same phone backup, the copies land beside their twins as IMG_1234 (2).

The folders hold everything needed to rebuild it, and keeping the two in step is on my list of possible future work. I don’t know yet when, or whether, I’ll get to it. Until then, losing that file costs you duplicate detection, never your photos.