Loading...
Manual, repetitive operational work that scales with the system but adds no lasting value.
Quick Definition
In site reliability engineering, toil is operational work that is manual, repetitive, automatable, and grows in proportion to the size of the system, while adding no lasting value: restarting stuck services, rotating credentials by hand, copying data between environments, approving routine requests. It keeps systems running but builds nothing. SRE practice caps toil, commonly at half of an engineer's time, and treats anything above that cap as a signal to automate, which frees engineers for higher-value work and improves both reliability and efficiency.
Toil is the term site reliability engineering uses for operational work that is manual, repetitive, automatable, and grows with the size of the system: restarting stuck services, rotating credentials by hand, copying data between environments, approving routine requests. It keeps the lights on but builds nothing.
Toil matters because it compounds. A task that takes ten minutes a day costs a week per year, and as systems grow, toil grows with them until entire teams do nothing else. SRE practice caps toil, commonly at half of an engineer's time, and treats anything above the cap as a signal to automate.
Example. An engineer spends two hours every Monday cleaning up orphaned storage volumes. A scheduled script does the same job in minutes, recovering a hundred hours a year and deleting the volumes faster than any human would.
Toil and cloud waste are cousins: both are recurring losses that feel too small to fix individually and too large to ignore in aggregate. Automating cost hygiene, cleanup of zombie resources, rightsizing reviews, tag enforcement, removes toil and spend together. The DevOps guide covers the automation mindset behind this.
No, and the distinction matters. Work that requires judgment, such as incident response, architecture and capacity decisions, and design reviews, is engineering, even when it is urgent or unglamorous. Toil is specifically the mechanical, repeatable subset that a script could do identically every time. Google's SRE definition adds that toil also scales linearly with the service, so work that stays constant as you grow usually is not toil.
SRE practice suggests keeping toil under 50 percent of an engineer's time, with the freed half spent on engineering work that permanently removes toil. The cap matters more than the exact number: without one, toil expands to fill the available time, because operational work always feels more urgent than automation work. Teams that measure toil usually find it higher than they assumed.
Much cost hygiene is toil when done by hand: finding and deleting idle resources, resizing over-provisioned instances, chasing missing tags, reviewing commitments. Because it is manual, it happens occasionally rather than continuously, so savings are captured once and then quietly erode. Automating cost hygiene removes the toil and makes the savings continuous, which is why toil reduction and cloud waste reduction are usually the same project.
The common method is to have engineers categorize their time, usually through ticket tagging or a periodic survey, into toil versus engineering work, then track the percentage per team over time. The precise number matters less than the trend and the cap: if toil is rising quarter over quarter, the system is growing faster than the automation around it.
Automate the most frequent tasks first, since toil cost is frequency multiplied by duration, not duration alone. Then remove the source where possible, for example fixing the bug that causes the weekly restart rather than scripting the restart. Self-service tooling also reduces toil by removing the human approval step, and a formal toil budget keeps the work funded rather than perpetually deprioritized.