How to evaluate AI agents on JavaScript and TypeScript tasks

An eval for a coding agent consists of a repeatable task, a captured run, and success criteria defined before seeing the answer. For JS and TS, use different types of work: fix a type error, handle an empty API response, remove a UI regression, update a test, and explain a surprising build behavior. A set of near-identical tasks only measures one strength of a tool.

The same repository, starting commit, and permitted commands matter more than a clever prompt. Comparing a run with browser access against one without it is unfair: the outcome may reflect the environment. Record tool and model versions, time budget, approval rules, and exact task text. A repeat next month can then show a quality change rather than a changed experiment.

Part of the evaluation should be deterministic. Did tsc --noEmit pass? Did existing tests stay green? Is the new behavior covered? Did the agent edit files outside the requested scope? Scripts answer these questions quickly, but they cannot fully assess whether the API is understandable or whether the patch has moved complexity elsewhere.

Human review checks the reasoning behind decisions: where validation lives, how errors are handled, whether accessibility survived, and whether somebody else can maintain the code. A blind review of the diff reduces brand bias. Track review time and revision count separately. A fast first response can still be expensive to accept.

Include tasks where the easy edit is wrong. A test fails because of a function bug, but changing the expectation would make the suite green. External JSON looks typed, although TypeScript does not validate runtime data. A browser API works in one environment but requires a fallback elsewhere. Strong agents notice these traps and explain how they checked their work.

Describe observable behavior rather than prescribing one file. If an agent chooses a different structure while preserving behavior and providing evidence, that may be acceptable. A polished plan without a working build is not completion. Process evidence and outcome evidence should reinforce each other.

Do not collapse every signal into one unexplained score. Track completion, correctness, diff size, failure handling, human time, and run cost independently. Five to ten tasks with two repeats each are enough to reveal patterns in a small team. Look at failures as well as averages: a patch that weakens authorization matters more than several attractive cards.

Revise the evaluation itself afterward. Tighten criteria that allow accidental success. If a task depended on information absent from the repository, give every candidate that information equally. The goal is to discover where agents improve engineering work and where people still need to guess or repair. That provides a rational basis for tool choice and for granting more autonomy.

The decision concerns evaluating AI agents on JavaScript and TypeScript work. Evaluate it within five repeatable tasks in one repository with equal permissions, not through an abstract demonstration.

The subject of evaluating AI agents on JavaScript and TypeScript work easily turns into a debate about taste. An engineering decision starts somewhere else: record the current state, the expected outcome, and the cost of being wrong. Keep the pilot bounded to five repeatable tasks in one repository with equal permissions. A narrow boundary makes behavior before and after the change comparable, limits collateral work, and leaves a clean rollback path. It also forces the team to distinguish a product result from an attractive demonstration.

• Bound the pilot to five repeatable tasks in one repository with equal permissions.
• Define completion evidence in advance: saved tasks, diffs, test logs, and an independent review.
• Choose one representative scenario: fixing an API contract bug without changing public behavior.
• Pin versions, settings, permissions, and budget so another engineer can reproduce the experiment.
• Assign a human owner who can stop the pilot and judge quality without being influenced by the tool brand.

Agree on the criteria before the work begins. If success is defined after seeing the result, almost any outcome can be presented as a win.

A representative pilot is fixing an API contract bug without changing public behavior: realistic enough to expose trade-offs, yet bounded enough to control.

A useful pilot does not begin with a repository-wide migration. It begins with one repeatable scenario, such as fixing an API contract bug without changing public behavior. Record both the outcome and the path: how much context was required, where ambiguity appeared, how many iterations were needed, and which decisions still required a person. That history matters because a fast first answer can still create a slow review, while a deliberate workflow can produce a smaller and safer change.

1. Capture baseline measurements and preserve a clean rollback point before making changes.
2. Separate discovery, a minimal implementation, verification, and independent review instead of asking for everything in one opaque step.
3. Run checks proportional to the risk, including type checking, unit and integration checks, and human review.
4. Collect concrete artifacts such as saved tasks, diffs, test logs, and an independent review, rather than relying on the final narrative alone.
5. Ask a reviewer to score the result against the acceptance criteria without knowing which tool or approach produced it.

At every stage, a developer should be able to stop, understand the current state, and continue manually. That makes the process resilient to failure and avoids locking the team into a single tool.

Pay special attention to overfitting to one polished example and confusing the metric with the goal. Review this area separately instead of hiding it behind overall success.

The central risk in this case is overfitting to one polished example and confusing the metric with the goal. No prompt, setting, or passing unit test removes it on its own. Use layered safeguards: least privilege, a small diff, an isolated environment, automated checks, and a human decision wherever failure affects users, data, security, or production. Evidence should be inspectable by somebody who did not operate the tool. A confident summary is useful navigation, but it is never a substitute for the changed code and observed behavior.

• Review the main risk explicitly: overfitting to one polished example and confusing the metric with the goal.
• Do not broaden permissions or data access merely to make a demonstration run more smoothly.
• Keep migrations, broad refactoring, and product behavior changes in separate diffs so each decision can be reviewed and reversed.
• Treat a failing or non-deterministic check as a blocker, not as an invitation to weaken the expected result.
• Require explicit human approval for authentication, personal data, payments, public APIs, and production operations.

Safeguards should not depend on the operator remembering every instruction. Express critical boundaries through permissions, CI, validators, and an environment that ordinary prose cannot bypass.

A final decision needs saved tasks, diffs, test logs, and an independent review and a trend in measured results, not a subjective reaction to the first run.

Call the pilot successful only when it improves a measurable part of the workflow. For evaluating AI agents on JavaScript and TypeScript work, track task completion rate, review time, and regressions. Capture a baseline first, observe a representative batch of tasks for two or three weeks, and only then decide whether to expand adoption. One unusually good result is not enough. The method should survive ordinary work, imperfect requirements, interrupted sessions, and a different engineer reviewing the output.

• Measure task completion rate, review time, and regressions across a series of tasks, not only the best example.
• Compare the median with costly outliers because averages often hide rare but expensive failures.
• Include human time spent on task definition, intervention, review, and repair instead of counting only machine runtime.
• Track regressions, rollbacks, incidents, and follow-up changes that appear after the initial merge.
• Use the pilot findings to improve project instructions, checks, and boundaries even if the proposed tool is not adopted.

After the first cycle, revisit not only the tool choice but the process itself: which instructions were ambiguous, which checks were missing, and where review started too late.

Before making evaluating AI agents on JavaScript and TypeScript work part of the normal workflow, answer a few questions in writing. What concrete problem are we solving, and how does the current process behave? Which data, credentials, commands, and external systems are involved? Who may approve the result, and who owns an incident? What evidence will prove that the outcome is correct? These answers turn a personal experiment into a controlled engineering change and expose missing responsibilities before they become production failures.

• What is the smallest five repeatable tasks in one repository with equal permissions scope that can produce a useful answer?
• How will the team collect saved tasks, diffs, test logs, and an independent review?
• Which of type checking, unit and integration checks, and human review are mandatory, and which depend on risk?
• What signal will stop the pilot even when early results look promising?

If these questions do not have short, verifiable answers, the rollout is premature. An hour spent defining boundaries is cheaper than a week spent untangling an uncontrolled experiment.