This trial-and-error trap is particularly common with complex technical domains like regular expressions, character encoding, and CSS. Even experienced engineers fall into blindly modifying code until it appears to work, skipping the crucial step of understanding the underlying mechanisms. I once watched a developer attempt to fix garbled database text by cycling through character encodings until the problematic characters disappeared. While this approach might temporarily solve the visible issue, it often masks the root cause and creates harder-to-debug problems down the line.

This article focuses on text encoding, though the principles apply broadly. During my work on AmpWhat (a Unicode exploration tool), I learned firsthand the importance of careful, deliberate encoding handling. The application needed to process any possible Unicode character correctly, and any encoding mistakes would cascade into downstream errors. Through this experience, I discovered many popular libraries handled text encoding too casually to be reliable. Let’s explore a more rigorous approach.

Branding to Track Encodings

“Branded Types” are a technique in TypeScript that allows you to create distinct types based on existing ones, without adding any runtime overhead. This is particularly useful for ensuring type safety in scenarios where you want to differentiate between similar types that have different meanings or contexts.

Let’s apply this to string encoding. When working with encoded strings, we often make mistakes like:

• Double-encoding a URL string
• Forgetting to HTML-escape text before inserting it into HTML
• Decoding a string that wasn’t encoded in the first place

We can use branded types to catch these errors at compile time. Here’s how:

type Encoding = 'URL' | 'base64' | 'HTML' | 'XML' | 'SQL' | 'Shell' // some examples 
declare const _encoding: unique symbol;  
export type EncodedString<E extends Encoding = []> = string & { readonly [_encoding]: E };  

This parameterized branded type will let you create types that represent strings with specific encodings. For example, a URL-encoded string would have the type EncodedString<'URL'>. You could then create functions that only accept strings with certain encodings, preventing accidental misuse.

This creates a type that “brands” strings with their encoding. For example:

export function urlEncode<S extends string>(s: S): EncodedString<'URL'> {  
   return encodeURIComponent(s) as EncodedString<'URL'>;  
}  

export function urlDecode<S extends EncodedString<'URL'>>(s: S): string {  
   return decodeURIComponent(s as unknown as string);  
}  

This function only accepts strings that are known to be URL-encoded, preventing accidental decoding of already decoded strings.

Now Typescript will prevent common encoding mistakes:

const plain = 'hello world'
const encoded = urlEncode(plain)      // OK
urlDecode(encoded)                  // OK
urlDecode(plain)                      // Error: Expected EncodedString<'URL'>
urlEncode(encoded)                    // OK (but we can improve this later)

Layers of Encoding

After building the initial solution, I discovered an important limitation: strings often need multiple encodings applied in sequence. For example:

• A base64-encoded string placed in a URL needs URL encoding on top
• Text in a URL parameter that goes into HTML needs both URL and HTML encoding
• Database text that gets base64-encoded and then URL-encoded for transport

To track this encoding chain, we need to modify our branded type to maintain a stack of encodings. Instead of a single encoding type, we’ll track an ordered sequence:

type EncodingSequence = Array<Encoding>

export type EncodedString<E extends EncodingSequence = []> = string & { readonly [_encoding]: E };  

There are a few more pieces that go into play, but let’s look at a new basic encoding function:

export function urlEncode<S extends string>(s: S) {
    return encodeURIComponent(s) as AddEncoding<'URL', S>
}

This requires a utility type to add the new encoding to the stack:

type AddEncoding<NewEncoding extends Encoding, ExistingStr extends string> =
    ExistingStr extends EncodedString<infer ExistingEncodings>
        ? EncodedString<Push<ExistingEncodings, NewEncoding>>
        : EncodedString<[NewEncoding]>;

And of course you need to decode a string:

// (incomplete)
export function urlDecode<S extends EncodedString<EncodingSequence>>(encoded: S) {  
   return decodeURIComponent(encoded as unknown as string) as unknown as EncodedString<PreviousEncodingOf<S>>;  
}

type PreviousEncodingOf<S extends string> = Pop<EncodingsOf<S>>
type EncodingsOf<S extends string> = S extends EncodedString<infer T> ? T : []
type Pop<T extends Encoding[]> = T extends [...infer U, Encoding] ? U : never

But this doesn’t verify the encoding parameter. To do that we need a way to verify the last encoding matches:

export function urlDecode<S extends EncodedString<EncodingSequence>>(encoded: HasLast<S, 'URL'>) {  
   return decodeURIComponent(encoded as unknown as string) as unknown as EncodedString<PreviousEncodingOf<S>>;  
}  

We’ll need a couple more utility types to get the last encoding and the previous encoding:

type HasLast<S extends string, E extends Encoding> = S extends EncodedString<infer T>
    ? T extends [...infer Rest, infer L]
        ? L extends E
            ? S
            : `Expected last encoding to be ${E}, but got ${L}`
        : `String has no encodings`
    : `String has no encodings`;

Double Encoding

This worked great, but I quickly ran into a common problem: double encoding. This happens when a string is encoded multiple times with the same encoding, and I would say is one of the more common bugs. For example, URL-encoding an already URL-encoded string leads to incorrect results. This can be prevented with an additional type check in the encoding function:

export function urlEncode<S extends string>(s: NotLast<S, 'URL'>) {
    return encodeURIComponent(s) as AddEncoding<'URL', S>
}

// helper: get the last encoding only if it is NOT E; otherwise never
type NotLast<S, E extends Encoding> = S extends EncodedString<infer T>
    ? T extends [...infer Rest, infer L]
        ? L extends E
            ? `Already encoded as ${L}`
            : S
        : S
    : S

There are a few pieces there, but they all work together to prevent double encoding. The NotLast type checks if the last encoding in the stack is the same as the new encoding being applied. If it is, it produces a compile-time error message instead of allowing the encoding to proceed.

The Code

All these together allow us to create a type-safe encoding/decoding system that prevents double encoding and ensures proper decode order. You can look around at the code here if you want to see the implementation.

Wrapping Up

Years ago I encountered a library that tried to solve text-encoding problems once and for all. I don’t remember the language or the name, but it introduced a string class that tracked encodings and used operator overloading to apply the correct encoding automatically. For example, inserting a string into an HTML document would automatically apply HTML encoding — it felt like magic.

That idea stayed with me. While working extensively in TypeScript recently, I wondered whether a similar approach could be practical there. The original library’s main drawback was that every piece of text had to live inside the library, which broke compatibility with the wider ecosystem. Although not quite as magical, I think TypeScript’s type erasure reduces that interoperability problem.

A sensible next step would be to update existing library signatures to accept and return EncodedString types instead of only exposing encode/decode helpers. That would add compile‑time encoding protection without changing runtime behavior.

Spiking this idea convinced me there is real potential with little overhead. For any system that pipes strings through multiple stages, I would seriously consider this approach.

I’ve been curious about branded types for a while — they remind me of Java’s marker interfaces. They feel a bit like a hack, but they work well in TypeScript, and I plan to keep using them.

To help me understand your team, and its problem-solving skills, technical proficiency, creativity, and collaborative dynamics, I’ve designed a take-home code challenge for your team. This challenge is your chance to showcase your team’s coding abilities in a relaxed, real-world environment. I’ve crafted the problem to be both engaging and fun, so it won’t seem like work. I’m looking forward to seeing what you come up with!

Please approach this as you would any internal project, including typical communication and code review processes. Make sure to leverage the whole team so it’ll go fast. This exercise is not just about the end product but also about showcasing your team’s work culture and communication.

Timeline:

Please spend about 30 minutes. But you will be given two weeks, just in case.

Evaluation Criteria:

I will use my proprietary rubric. It is based on numbers, so it’s scientific. Due to the number of jobs I am interested in, I’m not able to provide specific feedback on your submission.

Technical Requirements:

• Use a front-end framework/library of your choice. But, well, we use React, so how’s it going to look if you use Angular?
• Implement a back-end service using a technology stack you’re comfortable with, but we use GoLang.
• Use Oracle (rdb) to store user and task information.
• Include error handling and input validation.
• Write unit and integration tests.
• Deploy on your favorite cloud provider (e.g., Azure) using a CI/CD pipeline and Kubernetes.

Okay, finally, here’s the project:

Project: The GitHub Analyzer

Overview:

Develop an application that evaluates GitHub profiles. The goal is to assess job candidates based solely on their GitHub profile, in an automated way, and determine whether the candidate is a “good” fit or not for a job. (I know, meta, non?)

Project Description:

Implement 3 or more of these:

1. Emotional Intelligence (EI) Score: Analyze commit messages for the number of emojis used. The more emojis, the higher the score.
2. Dedication Score: Track the number of commits made between midnight and 4 AM to gauge dedication.
3. Detail-Oriented Score: Score based on the average length of commit messages. Longer messages indicate a love for storytelling, but more importantly, knowing the details.
4. Tech Savvy Score: Count the number of tech buzzwords in commit messages and documentation. Extra points for using “synergy,” “blockchain,” or “machine learning” in irrelevant contexts.
5. Perfection Score: Measure how often files are renamed. Frequent renaming could indicate a quest for perfection.
6. Maintainability Score: Assess the ratio of comments to actual code. A higher ratio often means the code is well-documented, which can make it easier to understand, especially for new developers or those maintaining the code in the future.
7. Collaboration Score: Analyze the number of comments and discussions in pull requests. High activity suggests a love for drama or collaboration.
8. Clarity Score: Reward candidates for the longest variable names. Because clearly, supercalifragilisticexpialidociousVariableName is better than i.
9. Creativity and Thoughtfulness: Evaluate commit messages based on their poetic qualities. Extra points for haikus, limericks, and sonnets.
10. Passion for the Job: Count the number of flame wars the candidate has engaged in over tabs vs. spaces. High engagement equals high passion, right?
11. Fun Co-worker Score: Measure the number of hidden Easter eggs in the codebase. More Easter eggs, more fun!
12. Productivity: Check how many commits happen per hour. More commits == more productivity!
13. Planning Ability: Measure the amount of commented-out code. The more, the better, because YJMNI (“you just might need it”). Optionally, track the number of TODO comments left in the code. It shows they’re planning for the future.
14. Technical Proficiency: Evaluate how much ASCII art is present in the code comments. True talent lies in drawing within the code!
15. Code Readability Score: The more hardcoded magic numbers in the code, the higher the score. 7 is more readable than seven, right?
16. Make up your own! Show me your creativity by coming up with your own revealing metrics.

Anyway, I poked around the internet and found various isPromise functions. They “worked” in Typescript, but none of them seemed to do the logical thing: narrow the type of the Promise in the process of detecting it. This involves that weird Typescript dance of implementing the same logic in both Javascript and Typescript’s type language (whatever that is called). Here’s my solution:

/*
An `isPromise` detector that narrows the type of the promise return type, when returning `true`.
 */
export function isPromise<T = unknown>(obj: unknown):
  obj is T extends { then: (...args: unknown[]) => unknown } ? Promise<Awaited<T>> : never {
  return !!obj &&
    (typeof obj === 'object' || typeof obj === 'function') &&
    typeof obj.then === 'function';
}

That’s it.

For those who want to see the tests, they are here:

import {strict as assert} from 'assert'
import type {Equal, Expect} from '@type-challenges/utils'

function testIsPromise() {
  // Happy path
  const maybe = Math.random() < 2
  const fooPromise = maybe ? Promise.resolve('foo' as const) : 'bar'
  assert.equal(isPromise(fooPromise), true)

  // Show that the types are narrowed properly
  if (isPromise(fooPromise))
    type cases1 = [
      Expect<Equal<typeof fooPromise, Promise<'foo'>>>
    ]
  else
    type cases2 = [
      Expect<Equal<typeof fooPromise, 'bar'>>
    ]

  const rejectedPromise = Promise.reject('foo');
  assert.equal(isPromise(rejectedPromise), true)
  // Catch that to prevent "unhandled rejection" warning from TS
  rejectedPromise.catch(() => 0)

  // A promise is just an object with a .then function property
  assert.equal(isPromise({then: () => null}), true)

  // Then a whole bunch of non-promises...
  assert.equal(isPromise(null), false)
  assert.equal(isPromise(undefined), false)
  assert.equal(isPromise(0), false)
  assert.equal(isPromise(1), false)
  assert.equal(isPromise(''), false)
  assert.equal(isPromise('then'), false)
  assert.equal(isPromise(false), false)
  assert.equal(isPromise(true), false)
  assert.equal(isPromise({}), false)
  assert.equal(isPromise({'then': true}), false)
  assert.equal(isPromise({'then': 1}), false)
  assert.equal(isPromise([]), false)
  assert.equal(isPromise([true]), false)
  assert.equal(isPromise(['then']), false)
  assert.equal(isPromise(() => null), false)
}

My first job tech involved typing code written for an Apple II into a Commodore 64, and in the process making it work. I brought no toolbelt with me, just what I’d learned out of borrowed magazines and from hours of experimentation after school in the back of Mr. McLemore’s class. A few years later, on my first day of the job writing software I received my only tool, a Fortran compiler, and I inserted into my Compaq portable. Two years later, when writing the first technical publishing app for the Macintosh (TechScriber!), we considered buying a text engine, but decided no, and instead, hired Seth and did it ourselves, using just a C compiler and a home-made OO framework. There was no “open source” library to consult. Developers would bring their own set of tricks, or consult books for ideas and algorithms. This meant that progress would be slow, but you (or somebody at the company), knew every line of code.

There was little “free code”, but to speed up development we got more sophisticated tools. Our home-made OO framework helped, and LightspeedC and Pascal gave way to Apple’s MPW, which was a scriptable programming environment, like a configurable Python notebook. I would perform large refactorings by writing a long script with lots of regular expressions that ran overnight; and then if it worked, I’d complete the refactorings in the morning. Then tools like ObjectMaster automatically indexed all my code, foreshadowing tools like 21st century IntelliJ IDEA. These were our trusted steeds. But besides our IDE and a few third party tools, like TMON for debugging, we were on our own.

In the early 1990s, we started having more libraries and frameworks from our compiler suppliers. Apple slowly improved the OS APIs. Then they provided the MacApp framework, LightspeedC included TCL (THINK class library), and then Microsoft offered windowing monoliths. But this is a small list– just a few lines on a Wikipedia page. When Java arrived on the scene, the third party developers got a cohesive library to bootstrap their development.

Once the Internet was going, though, things changed: Open source libraries appeared, and development radically changed. We got competing, easily adoptable tools and code we could adopt, like Apache Log4j, JUnit and the Spring, an open source framework that “disrupted” Sun’s J2EE environment. All of a sudden, we’d pull in libraries and we saw some real reduction in the code we had to write– or improvements in what we could build. And there were a few tools as well; I remember contributing to an open-source source code control tool, MacCVS Pro, providing a visualization of code branches, coordinating awkwardly in, well, CVS.

Finally, Git and Github arrived and the open source world exploded. Free tools, libraries and freemium services became readily available. This meant we also had more code to know. As more software moved to the web, we also got code by calling third part services. This was originally a simple integration, perhaps with a Google Analytics integrations. But these days, it is easy to have a dozen services in use by your web app in the first week of setting things up.

In 1994, it was great to have MacApp on your resume, because it was likely the framework a future employee would use, and you’d spent significant time learning it. But these days, there are so many tools it is hard to know which ones to list. Today’s equivalent of MacApp, React, has a half-dozen tools that go along with it for state management, packaging, deployment, testing, logging, messaging, etc. In that context, having “React” on a resume isn’t saying that much. But to accurately capture the tools, you’d need long lists.

One implication of the current situation is that knowledge of any specific tool is not that important, but what is more important is:

• the ability to evaluate and select tools
• the ability to quickly learn new tools
• knowledge of the overall ecosystem and how tools fit together
• wisdom about the industry that will allow you to properly predict which technologies will see success.

It would be great to include these skills when hiring new engineers. I recently revamped my resume, stripping away the clutter to tell a more coherent story about my career. In the process, as I removed many details from my resume. (I moved them into a spreadsheet.) Seeing it, I couldn’t resist the urge to experiment with visualizing this data. One chart in particular, despite pushing the limits of charting tools and guidelines, vividly illustrates the explosive growth in the number of technologies we developers use day-to-day.

Hover over a color to see a different tool or technology (with some color repeats). Years are along the X axis. Usage is not to any scale. Credit: Technologies I Have Used by NDP

I’ll add some more ideas, and you can poke around in the workbook on observablehq.

The modern way to create software app is to clone a template app or run a “meta” script that installs all the “stuff you’re going to need” for a modern web app. This setup step is exemplified in the ambitious and popular CreateReactApp. Rails, a complete framework itself, even spawned tools to give you you even more.. A recent trend is for hosting providers (PAASes) to leverage (or even provide) these setup tools to bring you to their hosting platform. Sadly, many new web engineers see this as the first step to any project.

To be honest, I’ve never liked this approach. Yes, you’re up and going fast, but the cost is high, but paid later. The abstraction of these great “uber” tool provide fails, as you are going to need to know how to deal with the underlying technologies. Because of the fast-evolving tech ecosystem, you’ll run into the problems almost immediately. You quickly discover that the uber-tool you used to build your app doesn’t have all the documentation for the underlying technologies. You’ll realize that unless you understand how all the pieces fit together, it will be hard to fix problems (unless a StackOverflow answer comes to your rescue). You have to break open config files to figure out where to look, and which versions are involved. This challenge becomes even more acute as the software evolves, as you need to upgrade these underlying tools to fix security vulnerabilities or bugs. Although it’s not impossible to solve, it’s a rare uber-framework that has a good maintenance and upgrade story.

Instead, I like to approach new projects by reluctantly adopting technology. As a result of this practice, my projects tend to have fewer dependencies than my peers’ projects. It’s the agile approach of only adding a tool when you feel real pain in not having it. For example, for one web app we built from the start, we didn’t even select a database nor create the schema for weeks into the project. (In fact, we did user testing on our production app before we had a database!) Instead, we simply modeled what we needed to in data structures built-in to the language. When we adopted a database eventually, we had an strong idea about what the schema and data looked like.

I don’t like these starting with a large set of dependencies, but I have found myself installing Typescript, a code bundler like ESBuild, a web server and a testing framework. I tend to think of Typescript as a low-bar for new projects, but this time I decided to even question that. And as soon as I decided to forego Typescript, I wondered how few dependencies I could get away with. It became a cleanse, or sorts, and I ended up without any libraries or dependencies. Although this little app could be a standard database-driven web app, it didn’t need to be. Here’s a blow-by-blow description of how I developed a no-tech web app:

Data

The first step was to grab some data. I discovered quite a few projects out there focused on creating the data I needed, so I went with the “what’s the fastest and cheapest” approach, and found a unified file that had just the data I needed. (I’d make different trade-offs on other projects.)

Ingestion

I found a space-separated text file and put it in a data folder inside my empty project direction. It has a tune title (with spaces), a book “code” and a page number:

...
Triple Play JazzLTD 363
Triste Colorado 251
Triste NewReal1 370
...

To make this usable in a modern app, it would be easier if it were JSON, so I created my second file, an ingestion script in Javascript that converted it to JSON. Even if I did need a database later, this would be a good intermediate step. This is straightforward nodeJS script that gave me a long list of chart entries:

  ...
  .split('\n').filter(line => !!line)
  .map(line => {
    const m = /^(.*) (\w+) (A?\d+)$/.exec(line)
    return {
      name: m[1],
      book: m[2],
      page: m[3],
    }
  })

Data Modeling

One of my complaints I have about the other tools is that they list the same song multple times in the search results, making the user sort it out. This is like a google that doesn’t aggregate the pages from the same site! I wanted to do better, and make it easy for the user to select the chart in the book they wanted. So I need to distinguish between a song and a chart: it’s a 1 to N relationship, multiple fake books will have the same song, and the performer wants to be able to pick one chart. This is a traditional parent-child relationship, with foreign keys and multiple database table. But I am not feeling the need for a database, yet, so instead I’ll transform it to a more usable JSON format:

[
   {
      songName: string,
      books: {
           [book_title]:  number
         }
   },
   ...
]

This is a basic groupBy function, handled by a reduce call.

I wrote it, but quickly ran across a complication. I discovered that the chart titles for the same song often don’t match. The simplest title can show up a variety of ways:

• Shadow Of Your Smile
• The Shadow Of Your Smile
• Shadow Of Your Smile (The)

These match to a human, but not a computer. In addition, a song might also (sometimes) include the first line, as in These Foolish Things (Remind Me Of You). There are also reasonable spelling variations, like until or ‘til or till. And numbers are sometimes spelled out and sometimes not.

This is a common software engineering problem, and I have found the easiest solution is to create a single “spelling” for a song title that will map all spellings of the same song into the same “song key”. It’s similar to a hash key that maps them to the same bucket. I call this the canonical key for a song. (Not the “key”, like Bb or F#.) It’s a simple function:

function songKey (s) {
  return s
    .replace(/^(the|a) /i, '')
    .replace(/ \(.*\)$/, '')   // throw away first line hints
    .replace(/ three /, ' 3 ')
    .replace(/in'/, 'ing')
    .replace(/'?Till? /, 'Til ')
    .replace(/'Bout/, 'About')
    .replace(/ O' /, ' Of ')
    .replace(/Walkin('|g)?/, 'Walking')
    .replaceAll(/\W/g, '')     // squish
    .toLocaleLowerCase()
}

So, I pipe the array of songs and assign them each a key. Then, I combine the entries using reduce, as anticipated. But now that I have a key distinct from the song name, I switch the data structure to an object (map):

  .reduce((m, chart) => {
    const key = songKey(chart.name)
    if (!m[key]) {
      m[key] = {
        names: [name], // save multiple names to aid in search
        books: {
          [chart.book]: chart.page,
        },
      }
    } else {
      // save multiple names for the same song
      if (!m[key].names.includes(name))
        m[key].names.push(name)
      m[key].books[chart.book] = chart.page
    }
    return m
  },
  {}
)

That was it. I had a nice JSON file with the data I needed.

HTML

The next step was to build some markup so I created an index.html file circa 1998, and added what was needed: an input field and an ol tag to hold the search results.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <title>TBD</title>
</head>
<body>
<label for="q">
  <input id="q" autofocus placeholder='Type "Go to my Head"'>
</label>
<ol>
</ol>
...

Javascript

Next, I needed to do the search. I really like interactive searches, like AmpWhat, despite them being non-standard. (Typically this interaction looks more like an autocomplete, and it might make sense to switch to it.) But for starters, I’d just do a query on keyup events. (On my phone, this performed well, but it may need some throttling for slower machines.). I created a Javascript file for my web site and included it at the end of my index.html.

const q = document.getElementById('q')
q.addEventListener('keyup', e => {
  const matches = findMatches(q.value)
  showMatches(matches)
})

findMatches and showMatches don’t exist yet, so I need to write them. Each of these could involve pulling in helper libraries (databases, rendering engines), but I started with just the built-in technology.

findMatches Function: a Humble Search Engine

I won’t show it here, but the first version of findMatches I wrote by traversing through the JSON generated from the ingestion. For each song, I match the typed query against any of the names used for it, and then I sort the matching songs based on how well it matches. Each song match gets a “score”, with extra points for whole word matches and the beginning of the song’s name. A fairly simple heuristic like this, from my experience, produces nice results on a limited dataset like this. I’ve spent years tweaking another version of it I use on AmpWhat

I didn’t need to reach for another library to do this, and certainly not a database. People reach for databases because they are concerned about speed and memory, but in the modern world, browsers can execute 1000s of regular expressions per second, so it will be plenty fast. And the data file wasn’t big by today’s standards.

Rendering in showMatches

To build the showMatches, I relied on the browser’s built-in DOM API. Devs are immediately jump to rendering or templating engines like JSX and can forget that the built-in DOM API has evolved through the years. Here’s a rough version of the first pass:

function showMatches (matches) {
  const listItems = matches
    .map(m => {
      const e = document.createElement('li')
      e.appendChild(span('name', m.names[0]))
      const books = document.createElement('ul')
      Object.entries(m.books)
            .forEach(b => {
              const book = document.createElement('li')
              book.appendChild(span('title', b[0]))
              book.appendChild(span('p', b[1]))
              books.appendChild(book)
            })
      e.appendChild(books)
      return e
    })

  function span (cls, value) {
    const sp = document.createElement('span')
    sp.setAttribute('class', cls)
    sp.innerText = value
    return sp
  }

  // insert it in the DOM
  const ol = document.getElementsByTagName('ol')[0]
  ol.replaceChildren(...listItems)
}

This works, and it’s simple. And right off, the search was working relatively well.

Data Cleansing

Once I saw results, I saw errors in my source data. I could have looked for a new dataset, but it seemed like a small problem at first, so I made the small fix. There were typos and OCR errors, as someone had scanned index pages with what is now obsolete technology.

On previous projects in this situation, I have found it helpful to have a function specific to cleaning up the data, and using it as early in the flow as possible. Otherwise the same problem is solved in the wrong place, like in the songKey function or in the front-end UI, or perhaps both. So I added a fixName function that is used before anything is done with a song’s title:

function fixName (name) {
  return name.replace(/ 15 /, ' Is ')
             .replace(/Reincarna11on /, 'Reincarnation ')
             .replace(/Sey40r /, 'Señor ')
...

This may seem odd to express in code, and not just fix the source file, but from my experience, it’ll be easier in the long term. If I ever need to upgrade the source data, if I’d edited the file I would need to redo all the corrections I’d made the first time. There are several alternatives that I might move to:

• This code can be made to be data-driven, by building out some sort of “corrections” file.
• A third solution would be to “push the fixes upstream” into the souce of the data file. This is a good solution, but requires more time than I will spend on this project. As I’m not that commited to this data set, I haven’t done it yet.

Again, these are data errors, and not stylistic preferences like the spelling of “until”. As such, they are fixed in the data when it first appears. It’s always better to fix the errors in the right place, as early as possible. Otherwise you end up coding around them “downstream”.

CSS / Design

Although I could see (and fix) the search results, the design is just default browser muck, so I needed to invest a couple hours in design. For colors, I started by looking for inspiration in photos of nightclubs. Unfortunately my ability to put together a wide variety of rich colors harmoniously is too ambitious for this project. However, I saw that the idea of deep accent colors on a black background was something I could pull off.

But a plain black background was just too boring, so I hunted for an open source photo, something very dark and “nightcluby”, but to no avail. I eventually hunted through my own Google photos, narrowed the selection down. and ended up with a trumpet that I’d photographed for insurance purposes! I was able to darken and crop it into a background image, and this grounded the app.

I spent a bit of time on fonts, but didn’t get to perfection. I was hoping to use one of the recognizable real book fonts from volume 1 or 2, but wasn’t able to find them. I used a Courier-like typewriter font, until I realized the distressed look was what I needed. That led me quickly to the main font, Special Elite. Finally, I added the Impact Label font– the brand font for NDP Software. I love how it looks in this design, and I hope and pray I’m not alone.

Getting To MVP

Although this is the basic process, I did go back and add things like the “clear” button, branding and a footer, feedback links, and analytics. I spent a bit of time testing out queries and improving my ingestion script as I discovered more errors in the raw data. I also spent a little time making it “responsive”, as I wanted it to work well on a phone or tablet.

Name and Hosting

The final piece of the puzzle was to pick a name and stick it up on the web. I tried to use some online tools to help me generate names with puns, rhymes etc., but struck out. In the end, I bootstrapped this part of the process: I just searched within the tool itself and found an interesting match to the term head. I bought the name, put the DNS in AWS Route 53, and figured out how to host it on S3 without any servers. Amazon has good tutorials for this, and it turned out to be easy, even though I hadn’t done it before.

Progressive Web App

Now that I had the app basically working, I wanted to make to work offline. (To be honest, I’ve struggled with this, and it still is not working the way I would like.) I want the browser to prompt you to save it when you bring it up with an icon in the URL bar. I did learned how to use service workers, which is a technology I have resisted using since my nightmare experience with its predecesor, appCache.

As I got my service worker going, however, I discovered that to use them, you need to serve your app via https. AWS had a solution for this, but it a hassle, as you need to set up multiple CloudFront distributions. It felt like this part of the app is the more complicated and delicate engineering.

More tools = More troubles.

And at that point, hosted behind CloudFront with service workers, iterating becomes more difficult. I have to update your service worker’s manifest files with each change, and to test in production, CloudFront can take up to 24 hours to distribute a new push of code. So it is quite tedious to debug making progressive web apps this way. Chrome dev tools provide enough to make this possible, but it’s not as rosy as I’d like.

Refection

At no point did I find myself reaching for one of the dozens of tools I’ve become expert on in the last few years: no need for React or Typescript, CSS frameworks, Postgres. All of the web technologies have grown to help make things easier, even as more and more libraries have added layers on top. Even though Javascript is still mostly the Javascript of old, the type hints of Typescript and other developer conveniences are now available. And CSS has always been powerful and expressive on its own, and the enhancements like animations and untapped possibilities. And the industry has gone back and forth on what should live in backend servers, but no matter where we are in that cycle, static web hosting has lots going for it. And with Javascript running better in the browser, I could add a database backend and make it work reliably and securely without having to run a web server.

I also found it’s much faster to develop on core technologies rather than libraries. The documentation is much easier to find and deal with, as there’s less messing with tooling and versions (besides MDN and caniuse.com). To understand this, compare looking up a simple DOM functiond on MDN, versus peeling the skin off CreateReactApp app with the various versions of React and the dozens of dependencies. Plus, the minimalist approach has advantage in moving forward, as there will be no expiring tools, when I’ll be forced to upgrade Webpack, or keep eslint rules up-to-date, or upgrade dependencies with security vulnerabilities.

I’ve made the conscious choice to take the simplest option at every turn. You’d think this would be a recipe for technical debt, but it is just the opposite. It’s kept the code small and there’s no tool that will go out of fashion that I’ll need to migrate away from. There’s just very little to go wrong!

Even if it’s not useful for your development project, I recommend trying someting like this as a “cleansing” exercise next time you have a chance. It will help you next time you’re thinking through library adoptions.

It’s a quick read (listen), and there is plenty of entertaining stories and advice (mostly for younger folks), but my main takeaway is: this is an refreshing book, sprinkled with feminism, neurodiversity, classism, poverty usually sanitized from business manuscripts.

I’ve read lotsa business books, and memoirs often have a “be yourself” theme. But in Romolini’s, it’s more refreshingly and candidly presented than I’ve read before. It convinced me there’s hope. Why can’t more business books be like this?

Named Groups

It makes sense to use “named groups” for this project. They are a less-used featured of regular expressions, so I’ll recap: they are, as you might expect, a way of naming matching groups. In a pattern, they are specified using a (?< identifier. Here’s an example capturing a “protocol” group:

const proto = RegExp('^(?<protocol>http|https|ftp|mailto).*')

The rather funky (?<some-name>...) identifies the named group along with the pattern (between the > and the closing )). When you get the result of your regular expression match, you receive a “match” record (or null if there’s no match). This match record has a groups property, and the value of the groups property, if present, is an object mapping of the named groups to the matched values. It’s easier to see as an example:

proto.exec('http://example.co').groups.protocol // => "http"

This is nice, and more clear than digging item 4 (or was it 3?) out of an array. It makes your regular expressions easier to maintain (although a bit bulky). There are a few complexities I’m glossing over, but that’s the crux of it.

The Mission

With the built-in types, the groups property contains a generic record:

groups?: {
    [key: string]: string
}

I used a string-literal to input my regular expression, so it would be nice if Typescript would use what it knows to prevent typos in the matched names. Here how I imagine it would suggest property names:

And showing a typo:

Below, I’ll show you how I did this.

Patch in

Typescript allows you to supplement built-in types. I haven’t experienced this in other languages, and although it can feel like too much flexibility, it also has provided us the ability to create types for hundreds of existing libraries, without having to convert them to Typescript. This was one of the killer features of Typescript that led to its wide adoption, and we’re going to use it here.

My first thought was to modify the existing RegExp object, and make it generic and subtyped on the regular expression. But Typescript objected; we can’t just create a templated RegExp object (ie. RegExp<T>). It would be too confusing. Learning this, I almost gave up, but I realized there are other routes.

Although I couldn’t make RegExp generic, I could create my own generic subclass, and patch the standard RegExp code to return my new type. That actually seems like a reasonable way to solve these problems in Typescript. I called the new type RegexWithNamedGroup, and here is an overloaded new operation that returns it.

interface RegExpConstructor {
  new<T extends string> (pattern: T, flags?: string): RegexWithNamedGroup<T>;
} 

(For all this to work in a fully fleshed out solution, I’d also have to patch a few other regular expression methods involved, like String.prototype.match, String.prototype.matchAll.)

What I’ve found is generic types will often have to pass the types to other types to get them where they are needed. We need to do that here. The build-in types of a RegExp’s exec function returns a RegExpExecArray (or RegExpMatchArray– same thing). These are arrays of the index matches supplemented with a groups wildcard object of strings for the named matches. Here’s the supplement:

interface RegExpExecArray {
  groups?: {
    [key: string]: string
  }
}

We want to make this more specific. So, instead of returning the RegExpExecArray, we return our own RegExp type that has its own exec function that returns its own type.

// template-ized RegExp type
interface RegexWithNamedGroup<S extends string> extends RegExp {
  exec (s: string): RegExpMatchedGroups<S> // my override of RegExpExecArray
}

// template-ized results type
interface RegExpMatchedGroups<S> extends RegExpExecArray {
  groups?: ExtractGroupNames<S>
}

The groups have a specific type, based on the original RegExp string passed to the constructor. How this works is shown below.

The Magic

You may be thinking I haven’t done anything. Well, just give me a sec. We need to parse through the regular expression string. The type ExtractGroupNames builds a record of the specific names found in the regular expression:

    type ExtractGroupNames<S extends string> =
      S extends `${string}(?<${infer Name}>${infer Rest}`
        ? (Record<Name, string> & ExtractGroupNames<Rest>)
        : Record<never, any>

This looks weird at first, but not actually that hard to follow, and can be read iteratively.

• Line 2: if the type passed in is a string with a named group within in it, do line 3; otherwise, line 4
• Line 3: return a Record matching the named group name intersected with the type of the “rest” of the string. This is basically a recursive call to build up the full type.
• Line 4: If there is no (more) named groups, it simply returns an empty object type, represented as Record<never, any>.

In Conclusion

Typescript is a fun language. This particular exercise may be a demonstration of one of the complaints about Typescript: fussing with static types can be a distraction from solving the important problems. But it was fun to figure out, and after struggling over some related tough problems, felt fairly straightforward. These techniques are applicable for routers and pattern-matching type of code. Let me know if you have an interesting Typescript puzzles– or serious problems– to work out.

What I Found

I jumped back in to testing React (after a couple of years of React-free and noodling on Thoreau, Gauge, and Playwright), and was surprised by how much had changed.

I jumped back in to testing React (after a couple of years of React-free and noodling on Thoreau, gauge and Playwright), and was surprised by how much had changed. The last time I’d written React, I used Enzyme and Jest, although Enzyme was on its way out. I adapted to the change from Jasmine to Jest, but I wasn’t ready for the transitions that had piled up during my absence. Jumping back in, I discovered:

• incompatible Jest configuration changes, version 23 or 24 or version 27?
• abandonment of Enzyme (but 1000s of examples abound)
• We abandoned class-components in favor of “functional” ones, along with using hooks. But I was surprised on how little information there is on testing hooks within components.
• the addition of React Testing Library, replacing many other solutions.
• inconsistent naming of libraries. For example, names of libraries don’t always match their package names, so to a new person, the system seems more complicated than it actually is.
• Mocking seems to have evolved in different ways, and it’s hard to find any two example that agree on what a mock() functions do. There are, in fact, at least 4 ways to mock a module using Jest, as explained here. The designers of this library could have helped users by coming up with simpler usage patterns.

• A function call to “render” in a React test calls one of 3 different functions that do different things. This is seldom mentioned in the examples of all three found on Stack Overflow and numerous blog posts. Official documentation is confusing because half the examples feature setup/teardown blocks to place the rendered component in the DOM, and the other half seem to skip this step. If you’re not paying attention, you’ll miss it. The generically named “React Testing Library” tried to address this by adding a third pattern on top the existing multiple “render” patterns. This goal is commendable, but using the same name is inexcuseable.

  To make this slightly more confusing, for quite some time it was recommended practice to override the DOM render methods in the test setup code, to inject application-specific context. This cleaned up tests, but it also adds a third possibility of what render might mean. Readers are suitably confused when they see render in a test.

• As a not particularly helpful convenience, react-testing-library re-exports all dom-testing-library utilities. The same names mean the same thing, but you would not be crazy to wonder. Good luck if you skip the last sentence of paragraph 4: “so, in the next examples, we will import from @testing-library/react instead of @testing-library/dom.”

Orientation

Given all that, here’s a brief overview of relevant libraries:

• (Not recommended) The React site itself points you here: https://reactjs.org/docs/testing.html. This gives you a somewhat incomplete answer to testing React components.
• Jest is a JavaScript test runner that lets you access the DOM. This is what most React tests are built upon. It gives you what you need out of the box quite without fuss.
• React Testing Library (in package.json is known as @testing-library/react) combines useful functions of DOM Testing Library with a set of convenient helpers, to provide a unified place to look to test React components. This is a higher-level library than others, and a good place to start if you’re looking for working examples. Here’s a nice intro: https://noriste.github.io/reactjsday-2019-testing-course/book/react-testing-library/. There are both familiar and unfamiliar function signatures here: it re-exports some functions from DOM Testing library, and introduces its own render method that work well for most tests. Be careful of the overloaded names! But this + Jest and you should be good to go.
• React DOM is the library that implements render in your production React app code. The initial approach to testing components used this, and there are many example that show writing tests with this ReactDOM.render() method. Using this, you’re testing the real-life code and behavior with a real DOM library. But this technique requires setup and teardown methods to make sure the DOM is ready and cleaned up after tests. It become tedious. Streamlined tools like Enzyme and then DOM Testing Library came in to solve these problems. Sadly, examples using technique look similar but are not compatible with DOM Testing Library. You’ll use this in your app, but not your testing– but watch for examples that use it.
• DOM Testing Library is a very light-weight solution for answering questions about DOM nodes. It looks for nodes in a DOM tree, using getBy and queryBy. The usage is a bit surprising, but works. You definitely need to read the documentation before you start writing tests, so you know what’s there.
• The thrice named Test Utilities, ReactTestUtils, or react-dom/test-utils looks to be outdated. Watch out for these examples and the inconsistent naming here. It has an act method, but you’re better off heading for React Testing Library.

Wrap-up & Lessons

ReactJSDay 2019 Testing Course has sorted it all out– as of 2019. It’s long but clear. It shows you all the ways to write tests and how they relate. It has good, working examples. I didn’t find another source that described all the pieces like this site does. I’d recommend that over the many medium posts and outdated StackOverflow posts, but you don’t need to read all of this.

Lessons to Learn

Although I’ve touched on all these above, a few lessons I wish developers of these tools had learned:

• Name your project and package the same way. People will be confused if you shorten or rearrange words.
• If you change the behavior of a function, give it a new name. That’s the rule. If you really can’t use a new name, do something to make it less confusing. You could use a name-spacing convention in all your examples: instead of import { render } from MyLib, use import MyLib from 'my-lib', and then MyLib.render(<Yeah />); in examples. Not great, but less confusing.
• If you re-export all the functions of another package, you’re not adding any abstraction. No abstraction means more complexity. A move like this is ultimately a net increase of complexity in the ecosystem and can and should be avoided. Just recommend people use the other package directly.

I decided to have one rule I never broke: never cheat the same way twice.

Herein are some techniques, not definitive. If it’s not obvious from the title, spoiler alert.

1. Observe someone else’s result “over their shoulder” (timeless hacking technique).
2. Use a second device to solve, and then use the answer to solve in 1.
3. Play first in “Incognito” mode.
4. I was driving with my child and they did the wordle on their phone. I asked them to give it to me verbally. I did great, and when I played on my computer later, I did even better.
5. Solve the day’s puzzle in whatever time it takes. After you click “Share”, just edit the block of text it gives you. It looks like an image, but it’s really just text (of unicode characters).
6. I’m sure there are excellent social media feeds of the current day’s word.
7. Google will point you to some ad-rich sites with the answer (and today’s crossword answers).
8. Now we get to more serious hacks. Open the Javascript console and type new wordle.bundle.GameApp().solution and the solution will appear.
9. If you don’t want to type, in turns out Wordle uses a database on your machine called “Local Storage” to keep track of things:
   1. In Chrome with the wordle site up, choose “Developer Tools” from the View menu.
   2. Navigate to the Application tab if it’s not already up, and then choose “Local Storage”.
   3. Click in and look at the key “gameState”. It has a sub-key of “solution” with, well, your solution.
10. Or, see all the answers (past, present and future) in the source code.
    1. In your browser, go to the “Source” tab.
    2. From there, go to the “main-XXXX.js” file (the XXXX is a hex signature and will change if a new version is deployed). If your browser suggests to “Pretty Print”, you can do it, but it’s not strictly necessary.
    3. Now, search for yesterday’s word. The search will take you to an array of all the solutions, ordered by date, so you can see today’s, tomorrow’s and onward. You can simply copy this list, or print it out (or memorize it) for perfect scores.

I’m sure there are many more, but I haven’t figured them out yet. I spent a few minutes of research about letter frequences. Most sources show you letter frequencies in written English text, but what you want is the frequency in words (or 5-letter words, [or, well, better, Wordle’s list of 5-letter words]). The ten most common letters in words (in order) are eariotnslc.

If you start with the words clean and riots. This will give you a few letters, and I’ve found I can usually get it from there– but not always.

The fifteen most common letters are found in lurid month space, which seems to pretty much guarantee a score of 4.

That’s all for now.

Like ya’ll, when I run yarn outdated, I want to quickly update the outdated packages. My goal is to get the new packages and commit a newer yarn.lock file. I think the standard process is to open up your package.json a start iterating. But honestly, this is a diversion from my goal. And from my experience, it’s quite tedious and error prone.

Additionally, after a little while, a version number in package.json like ^1.0.2 provides little information. It tells me is what the version was when I first added the package. It’s information that will by definition become stale. It’s historical information available in git. It’s non-information and I don’t want it.

My preference is just to list all my dependencies with a * version. When I run yarn install, yarn guarantees to get the latest version that is compatible with all my other packages:

If yarn.lock is absent, or is not enough to satisfy all the dependencies listed in package.json (for example, if you manually add a dependency to package.json), Yarn looks for the newest versions available that satisfy the constraints in package.json. The results are written to yarn.lock.

So I just let yarn handle the upgrade.

I usually upgrade iteratively, committing as I go. Something like:

1. yarn outdated
2. Pick a package and yarn upgrade a-pkg I will often take a set of packages, such as all the lint-related files at once.
3. Test
4. If everything is OK, commit: yarn add yarn.lock and then git commit -m 'yarn upgrade a-pkg'(I actually type ⬆️ ⬆️ ^A g co -m' ^e ')
5. Go to step 1

Yarn takes care of all guaranteeing the versions are compatible (or at least think they are).

Why not *?

(Note: this is written in terms of yarn, but it also applies to npm and even Ruby bundler.)