How Many Tokens Is a Codebase? 60 Repositories Measured

Every tool that turns a folder into LLM context tells you it removes the noise. None of them tells you how much noise there was, what the rest is made of, or what is left after. The number people quote for their own repository is usually the one their editor's status bar showed them once.

So we measured it. Sixty public repositories, ten languages, three size bands, every file routed through the same engine the tool ships, every bundle assembled and tokenized whole, and the same 60 commits measured again each time a default changed, so every number here is the shipped build's. Three results change what you should do with your own folder.

There is no single answer, and the distribution is the finding. Half the repositories in our sample sit between 60,831 and 1,201,507 tokens. That is a factor of 20 inside the middle half alone.

The largest is certd/certd, an SSL certificate renewal tool whose monorepo carries a web front end, a server and one deployment plugin per hosting provider, at 5,535,907 tokens. The smallest bundles come from single-purpose libraries with a handful of source files. In the first two runs the largest was KDE/ghostwriter at 9,373,193, and 8,633,558 of that was a vendored 3rdparty directory our defaults did not know the name of; it is 861,866 tokens now, and the fix is in the list of our own defects below.

We checked what is already published before measuring. Two kinds of claim exist and neither answers the question.

The first is tool marketing. Repomix, the most widely used tool in this category, documents its Tree-sitter compression as "~70% token reduction" in its own README. That is a claim about an optional compression mode, measured against the tool's own uncompressed output, not a claim about how much of a repository is worth sending. Gitingest and the rest publish no figure at all.

The second is the long-context literature, which measures what happens to a model as input grows but never says how big a repository is in the first place. Liu et al. found that performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts". Chroma's 2025 evaluation across 18 models found that "across all experiments, model performance consistently degrades with increasing input length". Both are about the cost of a large context. Neither tells you whether your repository produces one.

Nobody has published the funnel: files in, text out, filters applied, tokens produced, over a sample large enough to show a distribution. That is what this is.

The sample was pre-registered. One GitHub search query per cell, written before any result was seen, and the first two results taken in the order they came back. No repository was added, dropped, or swapped after seeing what it measured.

language:<L> size:<band> stars:100..5000 pushed:>2026-03-07
archived:false is:public mirror:false template:false
sort=stars order=desc

Ten languages: TypeScript, JavaScript, Python, Go, Rust, Java, C#, PHP, Ruby, C++. Three size bands from GitHub's own size field, in KB: small 0.1 to 2 MB, medium 2 to 20 MB, large 20 to 200 MB. Two repositories per cell, 60 in total. Every one of them cloned at depth 1 on 2026-09-07, with the commit recorded, and every rerun since has measured those same 60 commits.

The star ceiling is the filter doing the most work. A floor of 100 keeps abandoned experiments out. A ceiling of 5,000 keeps the sample from becoming a list of exceptional mega-projects and curated link collections, which are not what anyone drops into a context window. The full list, the query, the replacement rule for a failed clone, and the biases are in packages/cli/scripts/repo-sample-2026-09-07.txt.

Two repositories in the draw turned out to be documentation collections rather than codebases. They were kept, because removing them after seeing them would turn a query into a choice. They are named in the results, at the top of the documentation share, exactly where they were predicted to land.

Every decision in the measurement comes from the shipped engine. There is no second implementation of the classifier, the filter, or the tokenizer, because a reimplementation measures a tool nobody uses.

1. 1

   Clone at depth 1 and record the commit

   A shallow clone, so the measurement describes repository content and not version control history.

2. 2

   Walk every file, including hidden ones

   .git is excluded because it is the version control database rather than repository content. Counting it would drown every other number.

3. 3

   Decide what is text the way the product decides

   Content signature routing first, then the encoding classifier. Documents such as PDF and DOCX go through the same parser registry the CLI uses, so a PDF that yields text counts as text.

4. 4

   Apply the defaults, and record which one fired

   Hidden files, the default ignore list, and any .gitignore in the tree. Each excluded file is attributed to the first rule that removed it.

5. 5

   Assemble the bundle and tokenize it whole

   The artifact is built with the product's own assembler, in the default XML style, and tokenized in full. This is what a reader pastes into a model, wrapper and file tree included.

The category map (source, tests, documentation, configuration, assets, lockfiles, generated, vendored) is ours and is a judgment call, not a standard. Nothing in a repository declares that a path is a test. The rules are ordered and first match wins, because a vendored test is vendored and a lockfile is a lockfile before it is JSON configuration. Print the full map with pnpm --filter @fileconcat/cli measure-funnel --rules. Assets are images in a text encoding such as SVG: the defaults keep them, and until the map was revised on 2026-09-10 for the tool comparison they were counted as source. Every composition figure below is cut under the current map.

The funnel

The model tail describes the median repository rather than the pooled total: 236,218 tokens is 23.6% of Claude Sonnet 5's 1,000,000 token window, and costs about $0.34 to send once at the catalogue's cheapest listed provider on 2026-09-11.

Two things in that funnel are smaller than people expect.

A fresh clone is almost entirely text. 37,293 of 39,107 files, 95.4% pooled and a median of 97.1% per repository. Binary assets are rarer in a repository than in a working folder, and the engine reads more formats than "text file" usually means.

The defaults remove a fifth, and unevenly. A median of 73.7% of files survive. The token distribution is heavily skewed: the 25th percentile loses 2.2% and the 90th loses 70.0%. Thirty-nine of 60 repositories lost more than 10%. Two lost nothing at all.

The same exclusion, counted two ways

Here is the result that changed how we describe our own product.

A tool that says "removed 40% of files" and a tool that says "removed 40% of tokens" have not told you the same thing, and neither has told you which one they meant.

The .gitignore row is the one we did not expect. Honoring .gitignore is a feature we built and describe; on a fresh clone it fires in 6 repositories out of 60, because the things a .gitignore names are the things that were never committed.

What the bundle is made of

Of the tokens that reach the bundle, 70.7% are source, 8.6% documentation, 7.1% assets in a text encoding, 6.5% configuration and 6.4% tests. The hatched segments never arrive, and the largest of them is the test suite the defaults drop by file name.

Pooled shares hide the variation, and the variation is the useful part:

The tests row is what survives the naming rule, and its tail is the rule's limit: one repository in ten is still more than a third tests, and jarro2783/cxxopts is 85.1% tests, because its suite lives under names the convention does not cover. If you are pasting a repository to ask about behaviour rather than to review the suite, that remainder is yours to remove, because a test is legitimate source code and only the named part of it can be dropped by a default.

The two documentation repositories the sample drew are the top of the docs column exactly as predicted: gopl-zh/gopl-zh.github.com at 99.5% and SwiftOldDriver/iOS-Weekly at 99.2%.

By language

Six repositories per language is a spread check, not a league table. Read the shape, not the ranking.

Go keeps the fewest files, because a _test.go sits beside most source files and the defaults drop it by name. The C++ repositories are nearly ten times the size of the TypeScript ones and almost entirely source. Ruby's median repository still carries the highest test share after the naming rule, which is the rule's limit showing: what it does not name, it does not remove.

Against a context window

The four dashed lines on the figure above are these: 23 of 60 fit 128,000 tokens (38.3%), 26 fit 200,000 (43.3%), 43 fit 1,000,000 (71.7%) and 50 fit 2,000,000 (83.3%).

More than half of these repositories do not fit in a 200,000 token window. A million-token window takes 72% of them, which is the number that makes the long-context literature above relevant rather than academic: the repositories that need the big window are exactly the ones sitting deep inside it, where measured accuracy is worst.

The measurement was built to describe repositories. It described our product instead: three times on the day, and once more when the same sample was walked beside three other tools. All four are fixed, and every number on this page comes from the fixed build, rerun on the same 60 commits.

1. The default ignore list was missing go.sum

In the run before the fix, 95 lockfiles reached bundles carrying 886,586 tokens, and every one of them was a go.sum. testcontainers-go alone shipped 789,477 tokens of dependency checksums, roughly three times the median repository's entire bundle, in a file whose contents no model can use.

Fourteen other lockfile names were already on the list. This one was an omission, not a decision. go.sum is now excluded and go.mod is explicitly kept, because the manifest is the part a reader wants. Pipfile.lock and gradle.lockfile went in alongside as siblings of the same oversight, though neither appeared in this sample and we claim nothing about them.

2. The token counter was not counting the artifact

The readout tokenized the file contents joined together. The XML root, the per-file tags, the header and the file tree are all sent to the model and all charged for, and none of them were in the number.

Measured over the sample, the wrapper is a median 2.5% of the bundle, 5.1% at the 75th percentile, and 18.6% on sintaxi/harp, where 147 small files mean the tags outweigh what they wrap. The counter now counts the same string the preview shows and the copy button copies.

3. Above 1 MiB the count was a guess with the wrong ratio

Tokenizing a multi-megabyte bundle in a browser is slow enough to freeze a tab, so above 1 MiB of text the tool stopped tokenizing and forecast the count as characters divided by four. That ratio is English prose. It was applied to Chinese, to minified assets, to anything.

Twenty-seven of the 60 repositories cross that threshold. Scored against the real count of the same bundles, the old figure was a median 11.6% off, 48.2% off at the 90th percentile, and 62.5% low on SCIR-HI/Huatuo-Llama-Med-Chinese, where a character is closer to one token than to four. Five of those 27 repositories were within 5% of the truth.

The fix keeps the reason the threshold exists and drops the assumption: 64 evenly spaced 4 KiB slices are tokenized for real, and the count is scaled by the ratio those slices show. The cost is a flat 256 KiB of tokenizing, whatever the bundle's size. How that budget is spent was tuned against the truth rather than picked, on the four repositories the first configuration did worst on:

The error is coverage, not sample volume: the same 256 KiB spread over more positions reads a heterogeneous bundle far better. 64 x 4 KiB is the shipped setting rather than the marginally better 128 x 2 KiB, because every slice cuts two tokens in half at its edges, a bias that grows with the slice count, and the gain past that point sits inside the noise of a four-repository comparison.

Across the same 27 estimated repositories, the shipped setting is a median 0.8% off, 2.1% at the 90th percentile, and 3.2% at worst. All 27 are now within 5% of the truth, against 5 of 27 before. Counting the whole sample, including the 33 repositories small enough to be tokenized exactly, 49 of 60 readouts are now within 1% of what a paste actually costs. Before both fixes, one was.

4. Two more names were missing: bun.lock and 3rdparty

On 2026-09-10 the same 60 commits were walked beside Repomix, gitingest and code2prompt for the tool comparison, which reads what each tool carries path by path, and two of our omissions were only visible from the outside. bun.lock reached our bundle in four repositories, 490,966 tokens of resolved dependency graph between them, and on KDE/ghostwriter a directory named 3rdparty put 8,633,558 tokens of vendored code into a bundle of 9,374,075. vendor had been on the list from the start; the other name the same thing travels by was not. Both went in on 2026-09-10, third_party and thirdparty beside them as spellings, and the rerun on 2026-09-11 moved those five repositories and no other: ghostwriter to 861,866 tokens, the four bun.lock repositories by between 20,499 and 400,854 each, and the remaining 55 reproduce to the token.

Ask what your repository is before asking which model fits it. The median here is 236,218 tokens and the 90th percentile is 3,044,662. Those two repositories have nothing in common as context problems. The first fits anywhere; the second does not fit anything, and no filter setting will change that.

A filter removes what is named, and the rest is a decision. On a clone the defaults remove a median 22.7% of tokens, and 11.9% of every token in the sample was a test file dropped by its name; before that rule the same defaults removed 3.5%, because the node_modules a filter is famous for was never committed. What moves the number past that is deciding that a whole category does not belong in this particular question: the documentation at 8.6% of kept tokens, the tests the rule does not see at 6.4%. That is a choice about intent, and a default cannot make it for you.

Read the unit on every claim, including ours. 12.4% against 1.6% is the same exclusion described twice. Whenever a tool tells you how much it removed, the first question is what it counted.

A bigger window is not a free pass. The repositories in this sample that need a million-token window are the ones that land where Liu et al. and Chroma both measured the worst accuracy. Fitting is not the same as being read.

• A fresh clone is not a working folder. No node_modules, no build output, no local env files. The generated and vendored shares are a floor and the real reduction on a developer's machine is larger. We did not measure it, and we are not going to guess it.
• The sample spreads over language and size, not popularity. Sorting by stars descending under a 5,000 star ceiling means every repository sits near that ceiling. A well-known project carries more CI configuration, more tests and more lockfiles than a private work repository, so the non-source share here likely overstates what a solo developer's folder looks like.
• Six repositories per language. Enough to show that ecosystems differ, not enough to rank them.
• The category map is ours. Nothing in a repository declares a path to be a test or generated. The rules are published and ordered, and a different reasonable map would move the composition figures.
• The tokenizer is OpenAI's. o200k_base via the o1-preview-2024-09-12 encoding. Claude and Gemini tokenize differently, so every token figure here is an approximation for those targets. That is true of the product's own readout as well.
• The composition split is per-file, the bundle figure is the whole artifact. They differ by the wrapper and by the joins between files, which is why the composition bar's total is slightly below the bundle total of 55,092,425. Do not mix the two.
• The measured wrapper is a floor. The shipped tool adds one more header line naming the image and binary files it left out, capped at ten paths. The measurement never fills that list, so a repository carrying binaries pays a slightly larger wrapper than the 2.5% median here.
• This measures what reaches the model, not what helps it. Whether a filtered bundle produces better answers than a raw one is a separate experiment that needs a model in the loop. It is not this one.

Everything needed is in the repository. The sample list is tracked next to the script that consumes it, so the selection is auditable rather than described.

git clone https://github.com/CeamKrier/file-concat
cd file-concat && pnpm install
pnpm --filter @fileconcat/cli measure-funnel --repos scripts/repo-sample-2026-09-07.txt
pnpm --filter @fileconcat/cli analyze-funnel

Add --pin <a previous run's JSON> to the first command to measure the commits that run measured instead of each repository's current HEAD, which is how the reruns below were made.

The first command writes a JSON record with a row per repository: the three stages, the token figures, the per-category split with kept and excluded separated, which rule excluded each file, and the skip counts. The second prints every cut quoted on this page but two, and both are marked where they appear: the go.sum figures describe the run before that fix, which the fixed build cannot produce again, and the slice tuning table comes from a separate comparison over four repositories. The commit measured for each repository is recorded in the output, and a rerun pinned to it reproduces the file counts and the token totals. The first run, on 2026-09-07, could not be reproduced that way, because the file tree was rendered in directory-walk order until the day it was published and two clones of one commit could order siblings differently; the tree has been sorted since.

The defaults changed twice after the first run, and the sample was re-run at the same 60 commits each time to say by how much. On 2026-09-09 the default ignore list stopped ignoring directories named __tests__ and started ignoring test files by each ecosystem's own naming convention: api.test.ts and api.spec.js, handler_test.go, test_parser.py, parser_spec.rb, UserServiceTest.java, OrderTests.cs, PaymentTest.php, ParserTests.swift. On 2026-09-10 bun.lock and the 3rdparty family joined the list. The page above is the third run; this is what each change moved:

The first change moved 42 of the 60 down and 18 up, the latter by 3,158 tokens in total, which is the folder-grouped tests coming back now that __tests__ is not ignored for its name. The drops landed where the language table predicts: the five largest were Ruby and PHP repositories, the two languages with the highest test shares in the first run, led by simplecov-ruby/simplecov at 52.8% and catfan/Medoo at 52.1%. The second change moved five repositories and no other, KDE/ghostwriter by its 3rdparty directory and four by a bun.lock.

The last column also recuts every file under the eight-category map, which is why source falls from 80.8% to 70.7% of kept tokens while nothing was removed from it: 7.1% of kept tokens are text-encoded assets that the seven-category map had counted as source, and ghostwriter's departure changed the pool every other share is taken over.

The limit of the test rule is that it is naming, not detection. A tests/ directory whose files carry no convention survives, Rust's integration tests in tests/*.rs survive, and a suite that does not follow its ecosystem's spelling survives. Tests at 6.4% of kept tokens is what remains after the named part is gone, not proof that 6.4% is all the test code there is.

The numbers above are a distribution. Yours is one point in it, and the only one that decides anything. Drop a folder here and the same engine runs on your machine: nothing is uploaded to us, the whole thing runs in your browser. If the destination is a ChatGPT Project, whose limit is a file count, the bundle takes one of its slots; the per-plan caps are on the ChatGPT Projects page.

How many tokens is my codebase?

If it is a typical public repository, somewhere between 34,860 and 3,044,662, with a median of 236,218. That range is the honest answer, which is why the tool counts your actual folder instead of guessing. Language matters more than you would think: the median C++ repository in our sample was 790,553 tokens and the median TypeScript one 82,762.

Does removing node_modules and lockfiles shrink a repository a lot?

On a fresh clone the defaults remove a median 22.7% of tokens, and more than half of that is test files dropped by name; before that rule existed the same defaults removed 3.5%. node_modules is not committed, so it is not there to remove. On a working folder the same filters remove much more, because that is where the installed dependencies and build output live. What does shrink a clone is a lockfile or a vendored directory, which is why one missing entry in our ignore list cost 886,586 tokens across four Go repositories and another cost 8,633,558 on one repository.

Will my repository fit in a 1M token context window?

Seven in ten of our sample did: 43 of 60. But fitting is not the same as being used well. Both studies in the references below found accuracy falling as input grows, so a repository that just barely fits a window is sitting in the worst part of it.

Should I strip tests before pasting a codebase into an LLM?

It depends on the question you are asking, and it is the biggest lever available. Tests were 17.5% of all kept tokens in the first run over this sample and more than 41.5% in the top quarter of repositories. Since 2026-09-09 our default ignore list removes test files by each ecosystem's naming convention, which takes that share to 6.4% on the same 60 repositories: 11.9% of every token in the sample is a test file dropped that way. That is naming rather than detection: a tests/ directory whose files carry no convention still comes through, as do Rust's tests/*.rs, so strip the rest yourself when the question is about behaviour rather than about the suite.

Why does the token count differ between tools?

Because tools count different things. Some count file contents only and leave out the wrapper they add, which we measured on ourselves at a median 2.5%. Some estimate from character counts rather than tokenizing, which we also did above 1 MiB and which ran a median 11.6% off. And any tool using an OpenAI tokenizer is approximating when you paste into Claude or Gemini.

• Lost in the Middle: How Language Models Use Long Contexts, DOI 10.1162/tacl_a_00638
• Context Rot: How Increasing Input Tokens Impacts LLM Performance
• Repomix
• Token estimation, how the count on the result screen is produced
• File filtering, what the defaults remove and how to change them
• XML vs Markdown for LLM context, what the wrapper format costs
• Context window costs, what filling 100K, 500K or 1M tokens costs across the model catalogue