Refcount One

chunk

Everything that arrives at Boston Spa is cut into pieces before anyone looks at it.

Not at round numbers. We slide a forty-eight-byte window along the file, computing a rolling fingerprint at each position, and wherever the bottom thirteen bits of that fingerprint come up zero we declare a boundary. Nothing shorter than two kilobytes, nothing longer than sixty-four. On average that gives us eight kilobytes a piece — four pages of a paperback, near enough.

The boundaries are arbitrary but they are arbitrary in a way that depends on nothing except the bytes in the window. The same passage cuts in the same places wherever it turns up, in whatever it turns up inside. If we chopped at fixed offsets instead, inserting one paragraph at the front of a document would shove everything after it out of alignment and we’d match none of it. As it is, the insertion spoils one piece and the file re-synchronises on its own.

Then SHA-256 over each piece. The digest is the piece’s name — not a name we assign, a name it has. Two pieces with the same name are the same piece, for every purpose we can afford to care about. We ask the store whether it already holds that name. Usually it does.

What we keep, after that, is a manifest: an ordered list of names and lengths. A recipe. The store holds one copy of each ingredient however many recipes call for it.

That is the whole of the machine. Everything below follows from it and from nothing else.

price

Statutory deposit is free to the depositor, which is the point of it. Private deposit is not. Estates, parish councils, defunct societies, anybody who wants their material kept in the same conditions as the legal intake without having a statutory right to it — they pay, and we are obliged to charge at cost.

Our cost is bytes we did not previously have.

Nobody designed that as a policy. It falls out of the store. If a publisher sends us the twenty-second edition of a pharmacology textbook, the manifest is four thousand names long and three of them are new, and the quote goes out at eleven pence. If they send a first novel, every name is new and the quote is the full weight of the file.

So the invoice is a measurement, and it is a measurement of something other than storage. We have had two solicitors’ letters about it. Neither went anywhere, because we don’t publish the invoices and we don’t say what matched. We just say the number.

The number is enough, if you know what it is.

index

The other thing that falls out of the store is the sweep.

We used to keep a count against each piece: how many manifests name it, increment on ingest, decrement on withdrawal, delete at zero. Counts drift. Two ingests touching the same piece at the same moment, one of them retried after a timeout, and by the end of the quarter you have pieces claiming three references that have none and pieces claiming none that four manifests are relying on. The second kind is the one that loses you a book.

So we stopped counting and started walking. Sunday night, every manifest in the archive, name by name, marking each piece as reachable. Anything unmarked at the end of the walk was named by nothing and is swept. It takes about nine hours.

While you are walking every manifest anyway, building the reverse map costs you nothing extra — piece to the list of manifests that name it. We built it because the sweep occasionally goes wrong and you want to know what a piece belonged to before you decide it belonged to nothing.

It sits there all week. Nobody much queries it.

Kellow

Ruth Kellow came up from Leeds on a Tuesday with her father’s manuscript on a stick, and a form she had filled in wrongly, which is normal.

“He’d been at it thirty years,” she said. “The parish, the mills, all the way back to the enclosures. He kept saying he’d finish it and then he’d find something else in the record office and that was another year.” She had the flat, unembarrassed manner of somebody who has already done the crying part elsewhere. “I don’t want it published. I just want it somewhere that isn’t my loft.”

I explained the charge. She said she’d expected that, and asked roughly what sort of figure.

The ingest took under a minute. Two point one megabytes of plain text — he’d typed it in a word processor two decades out of support and she’d exported it as flat text because a man at the library had told her to, which was the best advice anybody gave her that year. Had she sent the PDF, it would have shared nothing with anything, because a PDF of a text and the text are not the same bytes and the store does not know they are related. It sees bytes. It has never seen a word.

The quote came back at four pence.

I checked it twice before I said anything, and then I said the number, and she said, “That can’t be right.”

“It’s right. It means we already hold most of it.”

“Hold it how?”

“Some other deposit contains the same text. Word for word, in long runs. Our system can’t see borrowings shorter than about four pages — anything at that scale slips straight through it. What it caught here is nearly all of the book.”

She sat with that for a moment. “He’d have told me if he’d copied it.”

I said I couldn’t tell her whether he had.

priority

This is the part I have had to explain often enough that I have a form of words.

The store can tell you that two texts share a body. It can tell you which pieces and in what order and how much of each text is accounted for. It is very good at that and it is never wrong about it, in the way that a ruler is never wrong.

It cannot tell you who wrote them first. It holds one copy of a piece and that copy has no history; there is no first copy and second copy, there is the piece. What we have besides that is deposit dates, and a deposit date is the date a file arrived at Boston Spa. It is evidence about a delivery.

The reverse map gave me two manifests for her father’s pieces.

The first was The Parish of Kirkby Wyke: A History, by Alan Prentis, statutory deposit, eight years ago. Ninety-one per cent of her father’s manuscript sat inside it.

The second was a run of the Kirkby Wyke Local History Society Bulletin, thirty-one issues, digitised under a small-press grant and deposited in a single batch eleven years ago by a volunteer whose name I won’t have been the first to forget. Chapters two through nine of her father’s text ran through it in instalments, three years before Prentis.

Under Denzil Kellow’s byline, she said, when I read her the titles. Every one.

I told her what that was and what it was not. It was not proof that her father wrote first; it was proof that a document containing his chapters, over his name, was in a public archive three years before Prentis deposited. What she did with that was a matter for somebody who was not me. She said she understood, in the tone of a woman who intended to go and find that somebody.

She asked, before she left, whether the sweep would take any of it. People hear the word and assume the worst.

“No. A piece is only swept when no manifest names it. We don’t delete manifests.”

“So his is safe because the other man’s exists.”

“His is safe because his exists.”

She asked one more thing, and it was the only question anybody had ever asked me that the reverse map was actually shaped to answer. She wanted to know which parts were only his.

It is a single pass. You walk the manifest and you keep the names that appear in no other manifest in the archive. Eleven pieces came back. I wrote the byte offsets on the back of the storage quote, in a column, because the terminal wouldn’t print to anything and I wasn’t going to make her wait.

Eighty-two thousand bytes out of two point one million. Forty pages, thereabouts. His last three summers, at a guess, and whatever he had found in the record office that made him keep going.

She folded the quote twice and put it in her coat pocket, and paid the four pence by card because the system will not accept a deposit that has not been paid for.