How to model a Typescript app using Effect: docgen

Reading and understanding open source code is one of the best way to learn about a library.

It can also be a daunting task, especially when you are not familiar with the structure and organization of the code.

In this post we explore the source code of the docgen library, written using Effect:

Here are some of the key takeaways:

• By using services and layers it becomes easy to read and understand code written using Effect
• The complexity is isolated in small and readable functions, allowing you to focus on a specific part of the code without the need to know everything about the rest of the implementation
• Effect automatically collects all the dependencies and errors in the type signature. This allows to get a sense of the result of any function without knowing the details of its implementation

In summary, Effect is a great solution to improve the structure and readability of your codebase. Here is how 👇

Project structure

All the code is contained inside the src folder:

All the implementation is contained inside the "src" folder

From package.json we can also see all the dependencies used inside the project:

"dependencies": {
    "@effect/platform-node": "^0.22.0",
    "@effect/schema": "^0.43.0",
    "chalk": "^5.3.0",
    "doctrine": "^3.0.0",
    "effect": "2.0.0-next.48",
    "fs-extra": "^11.1.1",
    "glob": "^10.3.3",
    "markdown-toc": "github:effect-ts/markdown-toc",
    "prettier": "^2.8.8",
    "ts-morph": "^19.0.0",
    "ts-node": "^10.9.1",
    "tsconfck": "^2.1.2"
  },

These information are generally enough to get a sense of how the project is implemented.

You may also want to read some of the tests inside the test folder:

All the tests are implemented inside the "test" folder

In the docgen package most of the other files are used to configure linting, type checking, testing, and bundling.

Entry point: bin.ts

The entry point is bin.ts. This is an executable file that imports the program main and run the final effect using runPromise (Runtime):

import chalk from "chalk"
import { Effect } from "effect"
import { main } from "./Core"

Effect.runPromise(main).catch((defect) => {
  console.error(chalk.bold.red("Unexpected Error"))
  console.error(defect)
  process.exit(1)
})

The type of main is Effect.Effect<never, never, void>. When you run the final program Effect should have never in the first two parameters:

• The first never means that you correctly provided all the dependencies
• The second never means that you handled all possible errors

In case of unexpected errors (defects) you catch any possible error after executing the Promise.

chalk is a library to style strings output on the Terminal

Provide dependencies and handle errors

The main method is defined inside Core.ts:

const runnable = program.pipe(Effect.provide(MainLayer))

export const main: Effect.Effect<never, never, void> = runnable.pipe(
  Effect.catchAll((error) => Console.error(chalk.bold.red("Error:"), error.message))
)

Notice how the type of main is manually provided. As explained above, main must have never in both the dependencies and and error channels (first two generic types).

Providing the type definition manually ensures that Typescript will report an error if we forget to provide a dependency or handle an error.

The other two variables defined in the code reported above are runnable and program.

program contains the full definition, with all dependencies and errors. It has the following type:

const program: Effect.Effect<Config.Config | FileSystem.FileSystem | Process.Process | ChildProcess.CommandExecutor, Error, void>

runnable then takes the full program definition and provides all the dependencies using Layer:

const MainLayer = Layer.mergeAll(
  Logger.replace(Logger.defaultLogger, SimpleLogger),
  ChildProcess.CommandExecutorLive,
  FileSystem.FileSystemLive,
  Process.ProcessLive
).pipe(
  Layer.provideMerge(Config.ConfigLive)
)

const runnable = program.pipe(Effect.provide(MainLayer))

Finally, main is responsible to handle errors, in this case using catchAll to handle all of them:

export const main: Effect.Effect<never, never, void> = runnable.pipe(
  Effect.catchAll((error) => Console.error(chalk.bold.red("Error:"), error.message))
)

This is a common pattern in an Effect app: a full program definition, runnable to provide all dependencies, and main to handle all errors:

/** Full program definition, with all dependencies and errors */
const program: Effect.Effect<Config.Config | FileSystem.FileSystem | Process.Process | ChildProcess.CommandExecutor, Error, void>

/** Provide all the dependencies makes the first generic parameter `never` */
const runnable: Effect.Effect<never, Error, void>

/** Handle all the errors makes the second generic parameter `never` */
const main: Effect.Effect<never, never, void>

Program implementation

program is implemented as follows:

const program = Effect.gen(function*(_) {
  yield* _(Effect.logInfo("Reading modules..."))

  const sourceFiles = yield* _(readSourceFiles)
  
  yield* _(Effect.logInfo("Parsing modules..."))
  
  const modules = yield* _(parseModules(sourceFiles))
  
  yield* _(Effect.logInfo("Typechecking examples..."))

  yield* _(typeCheckAndRunExamples(modules))

  yield* _(Effect.logInfo("Creating markdown files..."))
  
  const outputFiles = yield* _(getMarkdown(modules))
  
  yield* _(Effect.logInfo("Writing markdown files..."))

  yield* _(writeMarkdown(outputFiles))

  yield* _(Effect.logInfo(chalk.bold.green("Docs generation succeeded!")))
}).pipe(Logger.withMinimumLogLevel(LogLevel.Info))

The final program is usually the execution of a series of steps using Effect.gen.

Effect.gen will collect all possible errors and dependencies from all the steps:

const program: Effect.Effect<Config.Config | FileSystem.FileSystem | Process.Process | ChildProcess.CommandExecutor, Error, void>

All the steps in the effect are implemented using the same logic. Take a look at readSourceFiles for example:

const readSourceFiles = Effect.gen(function*(_) {
  const config = yield* _(Config.Config)
  const fs = yield* _(FileSystem.FileSystem)

  const paths = yield* _(fs.glob(join(config.srcDir, "**", "*.ts"), config.exclude))

  yield* _(Effect.logInfo(chalk.bold(`${paths.length} module(s) found`)))

  return yield* _(Effect.forEach(paths, (path) =>
    Effect.map(
      fs.readFile(path),
      (content) => FileSystem.createFile(path, content, false)
    ), { concurrency: "inherit" }))
})

• Use Effect.gen to construct an Effect
• Collect all required dependencies (config and fs)
• Execute the core logic of the function (paths and final return)
• Log information during the execution (Effect.logInfo)

The complexity of the program is isolated in simple and testable functions. These are then composed using the methods provided by Effect to construct the final program.

Service and Layer

An app in Effect is made of multiple services composed together.

In the readSourceFiles example above the effect used the FileSystem service:

export interface FileSystem {
  /**
   * Read a file from the file system at the specified `path`.
   */
  readonly readFile: (path: string) => Effect.Effect<never, Error, string>

  /**
   * Write a file to the specified `path` containing the specified `content`.
   */
  readonly writeFile: (path: string, content: string) => Effect.Effect<never, Error, void>

  /**
   * Removes a file from the file system at the specified `path`.
   */
  readonly removeFile: (path: string) => Effect.Effect<never, Error, void>

  /**
   * Checks if the specified `path` exists on the file system.
   */
  readonly exists: (path: string) => Effect.Effect<never, Error, boolean>

  /**
   * Find all files matching the specified `glob` pattern, optionally excluding
   * files matching the provided `exclude` patterns.
   */
  readonly glob: (
    pattern: string,
    exclude?: ReadonlyArray<string>
  ) => Effect.Effect<never, Error, Array<string>>
}

Notice how you do not need to read the implementation of each function to understand the program. The type signature using Effect tells you enough:

• The first parameter tells you if the function has any dependency
• The second parameter specifies if the function can fail
• The third parameter is the return type in case of success

Note: Effect does not distinguish between sync and async code (no need of Promise nor IO vs Task like fp-ts).

When you execute the Effect at the very end you need to use runPromise if you know that your program executes async code (or use runSync instead).

You can then create a Layer that defines a concrete implementation of the service:

export const FileSystemLive = Layer.effect(
  FileSystem,
  Effect.gen(function*(_) {
    const fs = yield* _(PlatformFileSystem.FileSystem)

    const readFile = ...

    const writeFile = ...

    const removeFile = ...

    const exists = ...

    const glob = ...

    return FileSystem.of({
      readFile,
      writeFile,
      removeFile,
      exists,
      glob
    })
  })
).pipe(Layer.use(PlatformFileSystem.layer))

Once again the pattern is the same:

• Collect all the dependencies (PlatformFileSystem)
• Define all the functions of the service
• Return a valid instance of FileSystem (FileSystem.of)
• Provide the final required dependency (Layer.use)

The complexity is contained inside the definition of each single function. The final Layer is a composition of smaller functions in a service.

Utility functions

Some function do not return Effect:

import * as NodePath from "path"

const join = (...paths: Array<string>): string => NodePath.normalize(NodePath.join(...paths))

These are pure functions used as helpers in different places in the app. These are not contained inside any service and they can instead be tested in isolation (if necessary).

How to read the source code

The docgen app is implemented following some conventions to help the make the code more readable.

Imports

Here is an example from Core.ts for imports:

import chalk from "chalk"
import { Console, Effect, Layer, Logger, LogLevel, ReadonlyArray, String } from "effect"
import * as NodePath from "path"

import type * as Domain from "./Domain"

import * as ChildProcess from "./CommandExecutor"
import * as Config from "./Config"
import * as FileSystem from "./FileSystem"
import * as Parser from "./Parser"
import * as Process from "./Process"

import { SimpleLogger } from "./Logger"
import { printModule } from "./Markdown"

All the imports of internal services are in the form import * as X from "./X".

This allows to have a clear separation of functions and errors (even in case on naming conflicts):

import * as FileSystem from "./FileSystem";

Effect.Effect<
  never,
  ConfigError | FileSystem.ReadFileError | FileSystem.ParseJsonError,
  A
>;

Here we know that the errors come from FileSystem, so we know at a glance where to search for their definition.

Notice how also path is imported using import * as NodePath. This unlocks the same benefits: we can reference all path functions from NodePath.

The Logger and Markdown modules export a single function instead of a service. Therefore the import * as is not necessary in this case.

Finally, the chalk library exports a single chalk function that itself contains all the methods provided by the library.

Errors definitions

The errors in docgen use the standard Typescript's Error interface:

interface Error {
    name: string;
    message: string;
    stack?: string;
}

For example the CommandExecutor service:

export interface CommandExecutor {
  readonly spawn: (command: string, ...args: Array<string>) => Effect.Effect<never, Error, void>
}

The Error interface is enough when all we need is to provide a readable error message (string):

const { status, stderr } = yield* _(Effect.try({
  try: () => NodeChildProcess.spawnSync(command, args, { stdio: "pipe", encoding: "utf8" }),
  catch: (error) =>
    new Error(
      `[CommandExecutor] Unable to spawn command: '${command} ${String(args)}':\n${
        String(error)
      }`
    )
}))

When you need more information about the context of the error (and not a simple string message) you should use a class or use Data (Data.tagged):

export interface ComplexError extends Data.Case {
  readonly _tag: "ComplexError";
  readonly metadata: Record<string, any>;
}

export const ComplexError = Data.tagged<ComplexError>("ComplexError");

export class UnexpectedRequestError {
  readonly _tag = "UnexpectedRequestError";
  constructor(readonly error: unknown) {}
}

Schema parsing errors

The Config service uses @effect/schema for parsing the configuration from a json file.

The validateJsonFile function uses the TreeFormatter.formatErrors method to output a readable error message in case of failures:

const validateJsonFile = <I, A>(
  schema: Schema.Schema<I, A>,
  path: string
): Effect.Effect<FileSystem.FileSystem, Error, A> =>
  Effect.gen(function*(_) {
    const content = yield* _(FileSystem.readJsonFile(path))
    return yield* _(
      Schema.parse(schema)(content),
      Effect.mapError((e) =>
        new Error(`[Config] Invalid config:\n${TreeFormatter.formatErrors(e.errors)}`)
      )
    )
  })

Effect.gen instead of pipe

docgen uses generators to define Effects (Effect.gen):

Effect.gen(function*(_) { ... });

Using Effect.gen makes the code linear and more readable.

This syntax is similar to async/await: it allows to execute any function returning Effect, extract and use the success value while automatically collecting possible errors.

That's it!

Once you got a sense of how the project is structured you can then focus on the details of its implementation. The good news is that generally all projects using Effect will follow a similar pattern.

Using Effect forces you to better organize your code: thinking about all the services and their methods, defining all the errors, and executing the final program only at the end.

In the next posts we are going to focus more on how to use Effect to implement your own project. If you are interested you can subscribe to the newsletter below for more updates 👇

Thanks for reading.