Taming the Data Fetching Beast: Our Journey at PayFit

Ever tried to solve a jigsaw puzzle with pieces from different sets? That’s pretty much what we faced with our data fetching setup at PayFit. Our types didn’t match the API results, and we were drowning in a sea of runtime errors. But fear not, fellow developers! We’ve charted a course through these treacherous waters, and I’m here to share our tale.

#

The Problem: When Types and Reality Don’t Align

Picture this: You’re confidently coding away, trusting your type definitions, when suddenly — BAM! 💥 Runtime errors start popping up like whack-a-mole in production. Why? Because the data we were getting from our APIs didn’t always match what our types were expecting.

We had two main issues:

• Legacy endpoints with unpredictable data structures
• Modern endpoints that were out of sync with our codebase because they are on different monorepos

How do you solve a problem when you don’t even know what you’re getting? That’s the question that kept us up at night.

#

A Soft Touch for Hard Problems: Runtime Validation

For our legacy endpoints, we needed a solution that was both flexible and informative. Enter Zod — our knight in shining armor for runtime validation.

Here’s what we did:

1. Implemented soft validation at runtime using Zod, meaning that we try parsing, and we don’t error out if it doesn’t match
2. Added error monitoring to capture and log type mismatches
3. Used these logs to refine our types over time

Is it perfect? No. It’s still a lagging indicator — we only know about issues after they hit production. But it’s a massive improvement over flying blind.

#

Bringing Order to Chaos: OpenApi and Type Generation

For our more modern endpoints, we took a different approach. We leveraged Orval to generate types from OpenAPI files. But here’s the catch: our product and API were in separate monorepos. How did we bridge this gap?

We set up a nightly sync of OpenAPI files. It’s not real-time, but it’s a heck of a lot better than manual updates. From these files, we generate result schemas that strike a balance between type checking and practical usage.

And what we found was surprising: OpenApi files were lying to us. They weren’t always accurate! Thanks to this, we found types issues on our APIs that expected data from other api to be present, but weren’t!

#

The Nuts and Bolts: Implementing Our Solution

Now, let’s get into the nitty-gritty. For our HTTP requests, we chose ky — a fantastic library that’s type-safe by default. Unlike fetch, which defaults to any and can lead you down a rabbit hole of type issues, ky defaults to unknown. It has a slim API that makes it easy to use, while still using fetch under the hood.

We created a utility function that parses API results using our generated schemas. It looks something like this:

import type { ResponsePromise } from 'ky';
import type { z } from 'zod';

/**
 * Parses the result of a Ky response promise using a Zod schema.
 * @param kyMethod - A function that returns a Ky response promise.
 * @param schema - The Zod schema to use for parsing the response data.
 * @returns A promise that resolves to the parsed data
 */
type ParseZodResult = <
  TDataPromise extends () => ResponsePromise,
  TSchema extends z.ZodType<unknown>,
>(
  kyMethod: TDataPromise,
  schema: TSchema,
) => Promise<z.output<TSchema>>;

const parseZodResult: ParseZodResult = async (kyMethod, schema) => {
  const response = await kyMethod();
  const data = await response.json();

  if (import.meta.env.MODE === 'production') {
    const result = schema.safeParse(data);

    if (!result.success) {
      window.datadog.addError(result.error, {
        apiURL: response.url,
        prettyError: result.error.format(),
        zodSchemaName: schema.description,
      });
    }

    return result.success ? result.data : data;
  }

  // On development, we want to throw an error if the schema fails to parse
  return schema.parse(data);
};

And so, when we use it, it looks like this:

import ky from 'ky';
import { z } from 'zod';

const postData = await parseZodResult(
  () => ky.get('https://jsonplaceholder.typicode.com/posts/2'),
  z.object({
    userId: z.number(),
    id: z.number(),
    title: z.string(),
    body: z.string(),
  }).describe('PostSchema'),
);

And postData will be properly typed! 🎉

We built libraries specific to each API, exporting queryOptions and useQueries from React Query. For fetching, with ky if you don’t parse, with zod, you won’t be able to use the data given it will be unknown, making the best path, the one with the zod schema!

#

Bringing It All Together: A Type-Safe Ecosystem

The beauty of our approach? It creates a cohesive, type-safe ecosystem:

• OpenAPI files generate schemas
• Schemas are used to infer types
• Types are used in MSW for mocking in tests and Storybook
• The same schemas power our useQueries hooks
• These hooks become the public API of our libraries within our monorepo

It’s like a well-oiled machine, with types flowing smoothly from API definition to client-side usage.

#

The Trade-Offs: Nothing’s Perfect

Now, I won’t sugarcoat it — this approach isn’t without its downsides:

We also tried to use orval to generate msw mocks, but the generated data wasn’t representative of the real data.

For us, the benefits far outweigh these costs. We’ve dramatically improved our type safety and reduced runtime errors, all while maintaining flexibility.

#

The Road Not Taken: Alternative Approaches

Let’s talk about the elephants in the room - there are other paths to type safety that we considered. GraphQL, with its strong typing system and schema-first approach, was an obvious contender. But migrating our entire API ecosystem? That would’ve been like rebuilding the ship while sailing it!

gRPC also came up in our discussions, but it felt like bringing a tank to a bike race - powerful, but not quite what we needed for frontend operations.

Then there’s tRPC. We initially passed on it due to concerns about scaling, but with their recent addition of lazy routing, it’s starting to look more appealing. Maybe it’s time to reconsider this choice?

The reality is, while these alternatives promise a type-safe utopia, they would require massive architectural changes that we weren’t ready to undertake. Sometimes, the perfect solution isn’t the practical one, and that’s okay! Our current approach might not be the most elegant, but it gets the job done without requiring a complete overhaul of our systems.

#

Looking Ahead: Future-Proofing Our Approach

We’re not resting on our laurels. We’re constantly evaluating our setup, ready to optimize when needed. Right now, the performance overhead isn’t a bottleneck, but we’re keeping an eye on it.

So, there you have it — our journey through the treacherous seas of data fetching at PayFit. It’s not a one-size-fits-all solution, but it’s working wonders for us. Have you faced similar challenges? How did you tackle them?

Remember, in the world of software development, there’s always room for improvement. Keep exploring, keep refining, and most importantly, keep sharing your experiences. After all, a rising tide lifts all boats! 🚀

Special thanks to Vincent Capicotto and Cyril Lopez for the work we did on this ❤️.️