lolspider / notes
Clipping · 6 分钟

How OpenAI ingests petabytes of logs every day

In this blog we will see where OpenAI's logs go, and how one line of code saved its logging system. In June 2025, ClickHouse wrote about OpenAI's logs.

aiengg.dev Gaurav Sen
#clippings#OpenAI#ClickHouse#observability#logging#performance#optimization#fastmod#profiling#indexing#bloom-filter#database-internals#sharding#replication#tiered-storage

📝 摘要

本文基于 ClickHouse 2025 年 6 月关于 OpenAI 日志系统的公开案例，梳理了 OpenAI 每天摄入 PB 级日志的架构与一次关键性能优化。OpenAI 的训练与推理负载产生海量日志，经负载均衡器均匀分发到由 90 个分片组成的 ClickHouse 集群；每个分片保留两个副本，并按 tiered storage（分层存储）策略把最近两天的数据留在本地磁盘上（约 80% 的查询只访问这部分），更早的数据下沉到对象存储。2025 年 3 月 25 日 GPT-4o 图像生成（Ghibli 风格）爆火，日志量一夜之间上涨 50%，原有的 50% CPU 余量被瞬间吃光；OpenAI 采用激进采样、为每个分片增加第三个只服务查询的副本来止损，但仍不足以支撑。工程师随后对 ClickHouse 本身做性能剖析，发现超过一半的 CPU 时间花在构建 Bloom filter 上——每次插入都要做一次「哈希值 % 过滤器大小」的除法取模。由于过滤器大小固定不变，可以用预先算好的常数把除法替换为「乘法 + 位移」（即 Lemire 的 fastmod 技巧），语义完全等价。这个「几乎只有一行」的改动让 CPU 使用率下降 40%，系统在爆红事件中稳定下来，改动随后被上游合入 ClickHouse，所有用户受益。

核心概念： 稳定的除数可以预计算，用乘法与位移替代除法取模；在海量写入路径上，一个看似微小的算术操作就足以吃掉半个集群的 CPU。

关键技术栈 / 关键要点：

  1. 90 分片 + 副本的 ClickHouse 集群 — 负载均衡器把新日志均匀打到 90 个分片，每分片 2 副本保障可用性并支持水平扩展。
  2. Tiered storage — 热数据（最近两天，覆盖 ~80% 查询）留在本地磁盘保证快查询，冷数据放对象存储保证低成本。
  3. 爆红流量事件 — 2025-03-25 GPT-4o 图像生成上线，日志量一夜 +50%，超出预留的 50% CPU headroom。
  4. 第一轮应对 — 激进采样 + 新增第三个「只读查询」副本，使查询不再拖慢写入，但仍不够。
  5. Bloom filter 的隐藏成本 — 它是让查询跳过不可能命中数据块的跳过索引，但每次插入都要计算 hash % (8 * size) 的除法。
  6. 除法 → 乘法 + 位移 — 除法比加法/位运算慢约 30 倍；size 恒定，因此可用常数做 fastmod 等价替换（文中给出了逐位验证的 Python 实现）。
  7. 结果与回馈社区 — CPU 使用率下降 40%，故障平稳度过，改动 upstream 到 ClickHouse 使全体用户受益。

关键结论 / 实践建议： 先测量再优化——不要凭直觉猜热点，剖析（profiling）会指出真正吃掉 CPU 的路径；对于固定除数，优先使用 fastmod 之类的乘法-位移替换；同时把「热数据留近、冷数据下沉」的分层存储与读写副本分离作为日志/可观测性系统的默认架构。

In June 2025, ClickHouse wrote about OpenAI's logs.[1] OpenAI ingests petabytes of log data every day.

Its engineers faced two problems:

  1. The logs had to be kept, and searched quickly.
  2. A viral launch could push the load up by half overnight.

In this blog we will see where OpenAI's logs go, and how one line of code saved its logging system.

1. Where do the logs come from?

OpenAI's two major technical problems are training large language models and serving customer queries, both of which scale to millions of GPUs, generating logs in the order of petabytes of data.

These logs are then routed to a central repository, where a load balancer evenly distributes them amongst 90 shards.

Logs go through a load balancer to the shards. Recent logs stay on disk and older ones go to blob storage, each shard is a group of replicas, and a third replica is added to every shard.

The central repository is a ClickHouse cluster. A shard holds a part of the logs, and the load balancer spreads new logs evenly over all 90.

2. Which logs do people read?

Do you read last month's logs? Most people read today's.

Since most users want to see recent logs, the last two days' worth of data is kept on disk for fast access, and older logs go into blob storage.

Each shard has two replicas which can be used if things go wrong and every shard can be horizontally scaled as per requirement.

ClickHouse's post puts a number on it: about 80% of the queries touch only the last two days. Keeping recent data close and old data cheap is called tiered storage.

3. The day image generation launched

What happens to the logs when a feature goes viral?

A recent incident with the logging system was on the 25th of March when millions of users started generating Ghibli images on ChatGPT.

That was 25 March 2025, the day OpenAI launched image generation in GPT-4o. People turned their photos into pictures in the style of Studio Ghibli.

The same dog in a park, photographed on the left and redrawn in the style of Studio Ghibli on the right. Both pictures were made with OpenAI's image model for this post.

Log volume surged by 50% overnight. The cluster had gone to bed with 50% headroom in CPU, and by morning it was gone.

OpenAI sampled the logs aggressively, keeping only some of them. It also added a third replica to every shard, for queries only. Lookups stopped slowing down ingestion.

But this wasn't enough.

4. One line of code

Why wasn't it enough? The engineers profiled ClickHouse itself to find where the time went.

More than half the CPU time went into building Bloom filters. A Bloom filter is an index that lets a query skip blocks of data that can't hold what it's looking for.

Each log added to the filter sets a bit. The bit's position is the log's hash modulo the filter's size. That modulo is a division, done once for every insert.

ClickHouse's post says a division is about 30 times slower than an addition or a bitwise operation. But the size never changes. The division can be replaced with a multiplication and a bit shift.

The Bloom filter's function in ClickHouse. The line that takes the position with a modulo, % (8 * size), is struck out, and the line that calls fastMod replaces it.

Here is that trick in Python. The constant is worked out once from the size. After that, every insert multiplies and shifts.

python
import random

size = 8 * 1_000_003  # bits in the filter: never changes
constant = (1 << 64) // size + 1  # worked out once

def by_division(h):
    return h % size

def by_multiplication(h):
    low = (constant * h) & ((1 << 64) - 1)  # the low 64 bits
    return (low * size) >> 64  # a multiplication and a bit shift

random.seed(0)
hashes = [random.getrandbits(32) for _ in range(100_000)]
print("same bit for every hash:", all(by_division(h) == by_multiplication(h) for h in hashes))
plaintext
same bit for every hash: True

The answers are the same, with no division. In ClickHouse it was "almost a one-line change". CPU usage fell by 40%.

So, did the logging system hold?

The result was the logging system stabilized, and the viral event of Ghibli images went through successfully.

A teammate then upstreamed the change to ClickHouse. Every user of it got the fix.

Summary

  1. OpenAI ingests petabytes of logs every day into a ClickHouse cluster of 90 shards.
  2. A load balancer spreads the logs evenly over the shards.
  3. The last two days stay on disk, where about 80% of queries look. Older logs go to blob storage.
  4. Each shard has two replicas. After the image generation launch, a third was added for queries only.
  5. The launch of 25 March 2025 raised log volume by 50% overnight. Sampling and the extra replica weren't enough.
  6. Half the CPU time went into Bloom filters, and one division per insert was the cost.
  7. A multiplication and a bit shift gave the same answer, and CPU usage fell by 40%.

Sources

  1. Why OpenAI chose ClickHouse for petabyte-scale observability, ClickHouse, 30 June 2025.
  2. BloomFilter.cpp, ClickHouse source code.
  3. Faster remainder by direct computation, Daniel Lemire and others, 2019.
返回 Clippings