Usage

Every example in this guide is given in both YAML and TOML: pick your format in any of the tabs, and the whole page follows.

Installation

pip install action0-celery-sched                # TOML schedules
pip install "action0-celery-sched[yaml]"        # YAML schedules too (PyYAML)
pip install "action0-celery-sched[solar]"       # solar schedules (ephem)
pip install "action0-celery-sched[yaml,solar]"  # all of it

TOML is read with the standard library’s tomllib, so it needs nothing beyond Python 3.11 and Celery. YAML needs PyYAML, which comes with the yaml extra. Without it, reading a YAML schedule stops with:

ImportError: YAML schedules need PyYAML: pip install 'action0-celery-sched[yaml]'

A first schedule

A schedule is a mapping of entry names to entries. Each entry names the task to send and when to send it:

# beat.yaml
"Poll feed":
  task: myapp.feeds.tasks.poll
  schedule:
    every: 5m

"Nightly report":
  task: myapp.reports.tasks.nightly
  kw:
    recipients: [ops@example.com]
  params:
    - daily
  schedule:
    crontab: "0 3 * * *"
  options:
    queue: reports

from celery import Celery
from action0.celery_sched import load_beat_schedule

app = Celery("myapp")
app.conf.beat_schedule = load_beat_schedule("beat.yaml")

# beat.toml
["Poll feed"]
task = "myapp.feeds.tasks.poll"
schedule = { every = "5m" }

["Nightly report"]
task = "myapp.reports.tasks.nightly"
kw = { recipients = ["ops@example.com"] }
params = ["daily"]
schedule = { crontab = "0 3 * * *" }
options = { queue = "reports" }

from celery import Celery
from action0.celery_sched import load_beat_schedule

app = Celery("myapp")
app.conf.beat_schedule = load_beat_schedule("beat.toml")

load_beat_schedule() reads either file into exactly what Celery’s beat_schedule setting expects, the same for both:

{'Poll feed': {'task': 'myapp.feeds.tasks.poll',
               'schedule': <freq: 5.00 minutes>,
               'args': (),
               'kwargs': {},
               'options': {}},
 'Nightly report': {'task': 'myapp.reports.tasks.nightly',
                    'schedule': <crontab: 0 3 * * * (m/h/dM/MY/d)>,
                    'args': ('daily',),
                    'kwargs': {'recipients': ['ops@example.com']},
                    'options': {'queue': 'reports'}}}

Load it wherever the app is configured, so that it is in place before beat starts.

YAML and TOML

An entry has the same keys and values in both formats. The loaders take a path (a str or any path-like), an open stream, or a mapping, and tell the format of a file by its suffix: .yaml and .yml are YAML, .toml is TOML. Streams are judged by their name. Files opened with open() have one, in text mode or binary mode (the mode tomllib users are used to). For anything else, say which format it is:

load_beat_schedule("conf/beat.conf", format="yaml")  # or Format.YAML
load_beat_schedule(io.StringIO(text), format="toml")  # or Format.TOML

Without a known suffix and without format=, loading refuses to guess:

ValueError: cannot tell the format of 'beat.conf' (known suffixes: .yaml, .yml, .toml): pass format='yaml' or format='toml'

Differences between the formats

The formats themselves differ in a few places, each explained where it matters:

  • Environment variables. YAML has tags, so !ENV is one: !ENV ${VAR}. TOML has none, so there !ENV is a prefix of the string: "!ENV ${VAR}". See Environment variables.

  • Shared settings. YAML anchors let entries share settings, and .-prefixed template entries carry them. TOML cannot refer to another table, so there the settings are repeated. See Templates and anchors.

  • Duplicate names. A name used twice is an error in both. In YAML it is reported as a duplicate entry. TOML forbids it in its syntax already, so there it is a syntax error. See Errors.

  • Empty values. In YAML, a kw: with nothing after it means no keyword arguments. TOML has no null, so there you leave the key out.

  • Long values. YAML nests by indentation. TOML inline tables (kw = { ... }) must fit on one line, so a longer one becomes a sub-table:

    ["Nightly report".kw]
recipients = ["ops@example.com", "reports@example.com"]

  • Quoting. YAML strings rarely need quotes; a cron string does, because * means something to YAML. TOML quotes every string, every = "5m", and no numbers or booleans, every = 300, enabled = false.

Mappings

A mapping is taken as the already parsed schedule. That way, a schedule can live inside a larger configuration file, as one part among others:

# settings.yaml
database:
  url: postgres://db/myapp

beat:
  "Poll feed":
    task: myapp.feeds.tasks.poll
    schedule:
      every: 5m

import yaml

with open("settings.yaml") as f:
    settings = yaml.safe_load(f)
app.conf.beat_schedule = load_beat_schedule(settings["beat"])

# settings.toml
[database]
url = "postgres://db/myapp"

[beat."Poll feed"]
task = "myapp.feeds.tasks.poll"
schedule = { every = "5m" }

import tomllib

with open("settings.toml", "rb") as f:
    settings = tomllib.load(f)
app.conf.beat_schedule = load_beat_schedule(settings["beat"])

The same works for a dict from JSON, or from whatever settings system the app uses. The mapping is validated exactly like a file. Only !ENV doesn’t apply, because the data is parsed already, and error messages can’t name a file.

Entries

Key

Required

Meaning

task

yes

the name the task is registered under, usually its dotted path

schedule

yes

when to run it — see Schedules

params

no

a list of positional arguments

kw

no

a mapping of keyword arguments

options

no

a mapping passed on to apply_async(): queue, priority, expires, …

enabled

no

false keeps the entry in the file but out of the schedule

The entry’s name becomes its name in beat_schedule, so it must be unique. Quote it if it contains spaces or punctuation: "Nightly report": in YAML, ["Nightly report"] in TOML.

Any other key is an error. That is deliberate: a misspelled key would otherwise be ignored, and the task would quietly never run.

"Nightly report":
  task: myapp.reports.tasks.nightly
  shedule:
    crontab: "0 3 * * *"

beat.yaml: entry 'Nightly report': unknown key 'shedule' (allowed: enabled, kw, options, params, schedule, task)

["Nightly report"]
task = "myapp.reports.tasks.nightly"
shedule = { crontab = "0 3 * * *" }

beat.toml: entry 'Nightly report': unknown key 'shedule' (allowed: enabled, kw, options, params, schedule, task)

Schedules

The schedule of an entry is a mapping with exactly one key naming its kind: every, crontab or solar.

Intervals: every

schedule:
  every: 5m

schedule = { every = "5m" }

The interval can be written in three ways:

YAML

TOML

Meaning

every: 30

every = 30

seconds; fractions are fine

every: 1h30m

every = "1h30m"

<amount><unit> parts

every: {hours: 1, minutes: 30}

every = { hours = 1, minutes = 30 }

timedelta arguments

The units of the duration string are w, d, h, m, s and ms, and parts may have spaces between them (2d 12h) and fractions (1.5h). Units are lowercase only, so m is never mistaken for months. The mapping takes weeks, days, hours, minutes, seconds and milliseconds. The interval must be positive.

Celery’s relative flag goes next to every:

schedule:
  every: 1h
  relative: true   # rounded to the interval: runs on the hour

schedule = { every = "1h", relative = true }   # rounded to the interval: runs on the hour

An interval becomes a celery.schedules.schedule.

Crontabs: crontab

schedule:
  crontab: "30 7 * * mon-fri"   # 07:30 on workdays

schedule = { crontab = "30 7 * * mon-fri" }   # 07:30 on workdays

A five-field cron string is read in the usual cron order: minute, hour, day of month, month, day of week. The nicknames @hourly, @daily (or @midnight), @weekly, @monthly and @yearly (or @annually) work too.

Alternatively, name the fields with Celery’s own argument names. The fields you leave out mean *:

schedule:
  crontab: {minute: 30, hour: 7, day_of_week: mon-fri}

schedule = { crontab = { minute = 30, hour = 7, day_of_week = "mon-fri" } }

The fields are minute, hour, day_of_week, day_of_month and month_of_year. Each takes what Celery takes (*/15, 1-5, mon,wed, a number), and in the mapping form also a list of numbers: minute: [0, 30] in YAML, minute = [0, 30] in TOML.

Celery validates every field while the file loads, so an impossible value fails at startup, pinned to its field:

"Nightly report":
  task: myapp.reports.tasks.nightly
  schedule:
    crontab: "0 25 * * *"

beat.yaml: entry 'Nightly report': schedule.crontab.hour: invalid value '25': Invalid end range: 25 > 23.

["Nightly report"]
task = "myapp.reports.tasks.nightly"
schedule = { crontab = "0 25 * * *" }

beat.toml: entry 'Nightly report': schedule.crontab.hour: invalid value '25': Invalid end range: 25 > 23.

A crontab runs in the app’s configured timezone.

Solar events: solar

schedule:
  solar: {event: sunset, lat: 48.21, lon: 16.37}

schedule = { solar = { event = "sunset", lat = 48.21, lon = 16.37 } }

event is one of dawn_astronomical, dawn_nautical, dawn_civil, sunrise, solar_noon, sunset, dusk_civil, dusk_nautical and dusk_astronomical (also available as the SolarEvent enum); lat and lon are the observer’s position in degrees.

Celery computes the event times with ephem, which comes with the solar extra (see Installation).

Environment variables

Values that differ between environments can come from environment variables. In YAML, !ENV is a tag in front of the value. TOML has no tags, so there it is a prefix at the start of the string:

"Nightly report":
  task: myapp.reports.tasks.nightly
  schedule:
    crontab: !ENV ${REPORT_CRON:-0 3 * * *}
  options:
    queue: !ENV ${REPORT_QUEUE}
  enabled: !ENV ${REPORTS_ENABLED:-true}

["Nightly report"]
task = "myapp.reports.tasks.nightly"
schedule = { crontab = "!ENV ${REPORT_CRON:-0 3 * * *}" }
options = { queue = "!ENV ${REPORT_QUEUE}" }
enabled = "!ENV ${REPORTS_ENABLED:-true}"

Only strings that start with !ENV followed by whitespace are substituted, and only values, never keys.

${VAR} is replaced by the variable’s value, and loading fails if it is not set. ${VAR:-fallback} uses the fallback instead. !ENV applies to single values, not to lists or mappings, and can mix text with several variables: !ENV ${QUEUE}-high in YAML, "!ENV ${QUEUE}-high" in TOML.

The error says where the unset variable is used. TOML substitutes after parsing, so it can name the entry and key. YAML substitutes while parsing, before any entry exists, so it names the line:

beat.yaml: environment variable 'REPORT_QUEUE' is not set (referenced via !ENV) on line 6

beat.toml: entry 'Nightly report': options.queue: environment variable 'REPORT_QUEUE' is not set (referenced via !ENV)

The result is always a string. It is not parsed again, so a value like 0123 or yes cannot turn into a number or a boolean by accident. Where a number or a boolean is expected (intervals, crontab fields, coordinates, relative, enabled), the string spelling is accepted. For booleans that means true/false, yes/no, on/off and 1/0. Values of params, kw and options stay strings, because nothing says what type they should be.

Disabling entries

"Nightly report":
  task: myapp.reports.tasks.nightly
  schedule:
    crontab: "0 3 * * *"
  enabled: false

["Nightly report"]
task = "myapp.reports.tasks.nightly"
schedule = { crontab = "0 3 * * *" }
enabled = false

A disabled entry is still validated, but left out of the beat schedule. Combined with !ENV (see above) it switches entries on and off per environment.

Templates and anchors

Entries often share settings: the same task, the same queue. YAML can say that once, with anchors and merge keys. Entries whose name starts with a . are templates: they are there to carry an anchor and are never scheduled, so they don’t need to be complete entries. TOML has no way to refer to another table, so there the shared settings are written out in each entry. Both files below define the same two entries:

.reports: &reports
  task: myapp.reports.tasks.build
  options: {queue: reports}

"Daily report":
  <<: *reports
  kw: {period: day}
  schedule: {crontab: "@daily"}

"Weekly report":
  <<: *reports
  kw: {period: week}
  schedule: {crontab: "@weekly"}

Keys merged in with <<: may be overridden. Arguments shared through an anchor are copied per entry, so a task that mutates its kw cannot affect another entry.

["Daily report"]
task = "myapp.reports.tasks.build"
kw = { period = "day" }
schedule = { crontab = "@daily" }
options = { queue = "reports" }

["Weekly report"]
task = "myapp.reports.tasks.build"
kw = { period = "week" }
schedule = { crontab = "@weekly" }
options = { queue = "reports" }

.-prefixed tables are skipped in TOML too, but nothing can refer to them.

Apart from merged keys, a key that appears twice in the same mapping is an error. Plain YAML would silently keep the last one, so copying "Daily report" to make the weekly one, and then forgetting to rename it, would replace the original. TOML forbids the duplicate in its syntax already:

beat.yaml: duplicate entry 'Daily report' on line 10 (first on line 5)

beat.toml: invalid TOML: Cannot declare ('Daily report',) twice (at line 7, column 16)

Several files

Pass several sources to merge them in order. They may be of different formats, and mappings too:

app.conf.beat_schedule = load_beat_schedule("beat/common.yaml", "beat/reports.toml")

An entry name defined in two files is an error, naming both. With replace=True the later definition wins instead and keeps the position of the first. Use this to let an environment-specific file override or disable entries of a shared one:

files = ["beat/common.yaml", f"beat/{environment}.yaml"]
app.conf.beat_schedule = load_beat_schedule(*files, replace=True)

# beat/staging.yaml
"Nightly report":
  task: myapp.reports.tasks.nightly
  schedule: {crontab: "0 3 * * *"}
  enabled: false

files = ["beat/common.toml", f"beat/{environment}.toml"]
app.conf.beat_schedule = load_beat_schedule(*files, replace=True)

# beat/staging.toml
["Nightly report"]
task = "myapp.reports.tasks.nightly"
schedule = { crontab = "0 3 * * *" }
enabled = false

A replacement is a whole entry, not a patch of the earlier one.

Checking task names

Beat sends tasks by name, and a misspelled task goes unnoticed until the first run is due, and then shows up only as an error on a worker. check_tasks() compares every scheduled name with the tasks the app has registered, and raises UnknownTaskError listing all unknown ones:

from celery.signals import beat_init
from action0.celery_sched import check_tasks


@beat_init.connect
def check_schedule(sender, **kwargs):
    check_tasks(sender.app)
    # if nothing is registered as myapp.feeds.tasks.poll, beat stops with:
    # UnknownTaskError: unregistered tasks: 'Poll feed' -> myapp.feeds.tasks.poll

The app only knows the tasks whose modules were imported. Beat imports them (through imports/include and autodiscovery) just before it sends beat_init, which is why the check belongs there and not next to load_beat_schedule. By default it checks the app’s own beat_schedule, so entries configured some other way are covered too. Pass a schedule as the second argument to check that one instead.

Errors

Everything this package raises on purpose derives from ScheduleError:

  • DefinitionError: a file or an entry is malformed, including YAML and TOML syntax errors. Its message starts with the location, and the pieces are also available as the attributes source, entry, path and reason.

  • DuplicateEntryError (a DefinitionError): an entry name is used twice, in one YAML file or across sources. Within one TOML file it is a syntax error instead (see Templates and anchors).

  • UnknownTaskError: raised by check_tasks, with the offenders in missing (entry name → task name).

The messages are the same in both formats, apart from the file name:

"Nightly report":
  task: myapp.reports.tasks.nightly
  schedule:
    crontab: "0 3 * *"

beat.yaml: entry 'Nightly report': schedule.crontab: expected 5 fields (minute hour day-of-month month day-of-week) or a nickname like '@daily', got 4 in '0 3 * *'

["Nightly report"]
task = "myapp.reports.tasks.nightly"
schedule = { crontab = "0 3 * *" }

beat.toml: entry 'Nightly report': schedule.crontab: expected 5 fields (minute hour day-of-month month day-of-week) or a nickname like '@daily', got 4 in '0 3 * *'

A few errors are not about the schedule’s content:

  • a file that doesn’t exist raises the usual FileNotFoundError;

  • a file or stream whose format can’t be told raises ValueError (see YAML and TOML);

  • a YAML schedule without PyYAML, or a solar schedule without ephem, raises an ImportError naming the extra to install (see Installation).

Working with entries

load_entries() returns the parsed Entry objects instead: disabled entries included, each knowing the file it came from. This is handy for tooling, or for a test that checks the schedule file itself:

from action0.celery_sched import load_entries

entries = load_entries("beat.yaml")
print(entries[0])
# Entry(name='Poll feed', task='myapp.feeds.tasks.poll', schedule=<freq: 5.00 minutes>,
#       args=(), kwargs={}, options={}, enabled=True, source='beat.yaml')
print(entries[0].to_celery() == load_beat_schedule("beat.yaml")["Poll feed"])
# True

from action0.celery_sched import load_entries

entries = load_entries("beat.toml")
print(entries[0])
# Entry(name='Poll feed', task='myapp.feeds.tasks.poll', schedule=<freq: 5.00 minutes>,
#       args=(), kwargs={}, options={}, enabled=True, source='beat.toml')
print(entries[0].to_celery() == load_beat_schedule("beat.toml")["Poll feed"])
# True

Entries compare equal whichever format they were read from; only source differs, and it is left out of the comparison.

The parsers behind the file format are public as well, each taking the parsed value of its key, whatever format it came from: parse_entry(), parse_schedule(), parse_duration(), parse_crontab() and parse_solar().

from action0.celery_sched import parse_duration, parse_schedule

parse_duration("1h30m")  # datetime.timedelta(seconds=5400)
parse_schedule({"crontab": "@daily"})  # <crontab: 0 0 * * * (m/h/dM/MY/d)>

Trust

YAML files are parsed with a safe loader and TOML files with tomllib, so neither can construct arbitrary Python objects. They can, however, schedule any registered task with any arguments, so load only files you would trust with that.