links: System Design MOC
Requirements
Functional
- Should be able to take 1000 rps
- It should persist the jobs and run during the scheduled time
- The system should be read heavy
- Results of the job should stored and can be queried
Non Functional
- Scalability: Thousands or even millions of jobs can be scheduled and run per day
- Durability: Jobs must not get lost → we need to persist jobs
- Reliability: Jobs must not be executed much later than expected or dropped → we need a fault-tolerant system
- Availability: It should always be possible to schedule and execute jobs → (dynamical) horizontal scaling
- Jobs must not be executed multiple times (or such occurences should be kept to a minimum)
Domain Analysis: Concepts
Job:
- Represents a Job to be executed
- Properties
- Id, name, JobExecutorClass, Priority, Running, LastStartTime, LastEndTime, LastExecutor, Data(Params)
Trigger (based on the concept Quartz Scheduler uses):
- Defines when a job is executed
- We can define different triggers like One Time or Cron Time
- Based on the type we have properties like
- id, type, start time, end time, one time, cron job, interval
Executor:
- is a single job executor/worker node
- can have properties like:
- id, last heartbeat
High-Level Design
Microservices that want to schedule a non-/recuring Job_: Can send a Message (or produce in Kafka terminology) to the corresponding Kafka Queue (precisely a Topic).
Job Scheduler Service: Consumes the Jobs from the Kafka Queue and puts in the Database
RDBMS: Save the Jobs here
Job Executor Service:
- On Startup picks the pending jobs to execute
tags: job-processing cron asynchronous scheduled-tasks sources: