Distributed Locks using Shedlock
Purpose
The purpose of this document is to implement distributed locking across multiple JVMs using Shedlock.
Before reading any further, kindly consider reading distributed locks post, it transitions nicely into this one.
Use Case
With spring-integration's version of distributed locks, we don't know how long a lock is grabbed for.
If the lock owner fails to release a lock because it is a spot instance or it's JVM crashed or the GC run is taking longer than usual - no other JVM can grab the lock.
Shedlock solves this problem by grabbing the lock only for a specified amount of time.
In the above example ip-192-168-0-5.ec2.internal grabbed the “AnirudhTestSchedulerLock” lock for 25 seconds.
Let's look at some code
User code: Add the below inside a @Component class.
@Scheduled(fixedDelay = 30_000)
@SchedulerLock(name = "AnirudhTestSchedulerLock", lockAtMostFor = "15s")
public void work() {
LockAssert.assertLocked();
LOG.info("The work!");
}
Add the below inside a @Configuration class.
@Bean
public LockProvider lockProvider(MongoClient mongo) {
return new MongoLockProvider(mongo.getDatabase("test"));
}
Add annotate your SpringBootApplication like this.
@EnableScheduling
@EnableSchedulerLock(defaultLockAtMostFor = "10m")
@SpringBootApplication
public class ShedlockApplication {
public static void main(String[] args) {
SpringApplication.run(ShedlockApplication.class, args);
}
}
application.properties
spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017
spring.data.mongodb.database=test
logging.level.net.javacrumbs.shedlock.core.DefaultLockingTaskExecutor=DEBUG
docker-compose.yml file?
version: "3.9"
services:
mongo:
image: mongo
#restart: always
ports:
- 27017:27017
^ docker-compose up
Finally, we are done with the setup.
Let's run it :)
Iteration1:
One JVM, sample run:
2021-08-08 16:10:51.845 DEBUG 11795 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Locked 'AnirudhTestSchedulerLock', lock will be held at most until 2021-08-08T20:11:06.841Z
2021-08-08 16:10:51.845 INFO 11795 --- [ scheduling-1] com.example.shedlock.schedulers.One : The work!
2021-08-08 16:10:51.847 DEBUG 11795 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Task finished, lock 'AnirudhTestSchedulerLock' released
Line 1 says the lock will be held at most until 20:11:06 (15 seconds).
To verify the above, add a breakpoint at “The work!” and run db.getCollection(‘shedLock’).find({}).
Once you resume the program, it will print “The work!” and lockUntil is updated to 20:10:51.845Z (less than a second).
It means anyone can grab the lock now. Beautiful.
Iteration2:
To run multiple JVMs, please check the “Allow parallel run” on “Run/Debug Configurations” on Intellij.
Run it twice and you'll notice both JVMs keep acquiring and releasing the lock. No fun.
To see only one of the JVMs holding the lock, add a Thread.sleep(12_000); after printing “The work!".
JVM1
2021-08-08 16:26:48.831 DEBUG 12099 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Locked 'AnirudhTestSchedulerLock', lock will be held at most until 2021-08-08T20:27:03.785Z
2021-08-08 16:26:48.837 INFO 12099 --- [ scheduling-1] com.example.shedlock.schedulers.One : The work!
2021-08-08 16:27:00.843 DEBUG 12099 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Task finished, lock 'AnirudhTestSchedulerLock' released
2021-08-08 16:27:30.864 DEBUG 12099 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Not executing 'AnirudhTestSchedulerLock'. It's locked.
JVM2
2021-08-08 16:26:55.727 DEBUG 12101 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Not executing 'AnirudhTestSchedulerLock'. It's locked.
2021-08-08 16:27:25.737 DEBUG 12101 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Locked 'AnirudhTestSchedulerLock', lock will be held at most until 2021-08-08T20:27:40.730Z
2021-08-08 16:27:25.741 INFO 12101 --- [ scheduling-1] com.example.shedlock.schedulers.One : The work!
2021-08-08 16:27:37.745 DEBUG 12101 --- [ scheduling-1] n.j.s.core.DefaultLockingTaskExecutor : Task finished, lock 'AnirudhTestSchedulerLock' released
JVM1 acquired the lock at 16:26:48, held it until releasing at 16:27:00.
JVM2 tried to acquire the lock at 16:26:55, but couldn't get it.
Next iteration (16:27:25) the reverse happened. Perfect.
Hence, only one JVM acquires the lock at a time.
Conclusion
We learned how to implement distributed locking across multiple JVMs using shedlock.
References
https://github.com/lukas-krecan/ShedLock
Gotchas
What happens if the lock holding JVM dies?
Mr Campbell, who cares? We have lockUntil.
Hope you enjoyed this newsletter!