Meta TIL (2 blogmarks)

I set up an internal process in the Rails app that powers still to run all of my TIL posts through the AWS Bedrock embeddings API. It used the Titan Text Embeddings V2 model and stored the result of each in this app's PostgreSQL database in an embeddings table using pgvector.

Why

I'm experimenting with ways to surface links to similar posts at the bottom of each TIL page. This seemed like both a fun experiment and what would probably end up being fairly accurate and novel results. Using pgvector, I can do comparisons across embeddings of all the posts to find the ones that are most similar in vector space using cosine similarity. That's computed using the <=> (cosine distance) operator provided by pgvector and subtracting it from 1 to get the similarity.

UI

Here is the prototype interface for comparing TF-IDF with Vector Embedding Similarity. I'm still going through and spot checking results for various posts to see which approach I'm happier with.

Cost

It's not quite 2000 posts. I knew it would be cheap, but wasn't exactly sure what it would come out to. Maybe like 10 or 20 cents is what I would have guessed. It was a lot cheaper than that. AWS's billing portal rounded up the cost to $0.01.

There is something really cool about seeing other people adopt a practice of writing TIL-style posts and coming to the same realizations and conclusions that I did with mine.

TILs are great learning resources and reference resources:

I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page.

In Simon Willison's 1 Year TIL Retrospective, he points to how they reframe writing and publishing something:

The thing I like most about TILs is that they drop the barrier to publishing something online to almost nothing... The bar for a TIL is literally “did I just learn something?”—they effectively act as a public notebook.

and on the value of always having a learning mindset:

They also reflect my values as a software engineer. The thing I love most about this career is that the opportunities to learn new things never reduce—there will always be new sub-disciplines to explore, and I aspire to learn something new every single working day.

It was fun to read through both of these posts having just myself reflected on A Decade of TILs.