I Added Search to the Site

Because of the site’s unusual architecture for storing articles and text sections - the site is built around text and JSON files connected to each other in a special way - I had to get creative with the search process.

As usual, I started by reinventing the wheel - my favorite thing to do, because I’m interested in how everything works and how everything is built. I made a script to index the language sitemaps, crawl all the pages, and build an index of tokens that I generate based on the title, the preview text, and the article text, while not forgetting the keywords either.

Then I save all the tokens and links to the corresponding articles into a separate file for each language. I wrote the token search script in JS, and it quickly searches text across the tokens using simple Damerau-Levenshtein transformations. This makes it possible to avoid complex semantic search, but it is enough to find the desired text by similar-sounding words and partial matches right while the user is typing. It works on the fly and consumes only the resources of the local computer, so there is no load on the server.

I can’t say the search is perfect, but for my tasks it is sufficient. Feel free to use it.