CPU Throttling – Scale by restricting work

We have a web service responding to web requests. The service has a thread pool where each web request uses one operating system thread. The requests are then managed by a multi-core CPU that time-slices between the various threads using the operating system scheduler.

This example is very similar to how Tomcat (Spring Boot MVC) works out of the box when servicing requests with servlets in the Java web server space. The Java VM (v17) matches a Java Thread to an operating system thread that is then scheduled for execution by a core.

So what happens when we have a lot of requests?

Many threads here are sliced between the 4 cores. This slicing of threads where a core works on one for a while, then context switches to another thread, can scale to any level. However, there is an expense in CPU time to switch between one thread to another. This context switch is expensive as it involves both memory and CPU manipulation.

Given enough threads, the CPU cores can quickly spend a significant amount of time context switching when compared to the actual amount of time processing the request.

How do we reduce context switching?

We can trade off context switching for latency by blocking a request thread until a vCPU is available to do the work. Provided the work is largely CPU bound this may reduce the overall throughput time if the context switching has become a major use of the available vCPU resources.

For our Java spring boot based application we introduce one of the standard Executors to provide a blocking task service. We use a WorkStealingPool which is an executor that defaults the worker threads to the number of CPUs available with an unlimited queue depth.

We now move the CPU heavy process into a task that can be scheduled onto the executor by a given thread. The thread will then block on the Future returned from submitting the task – this blocking occurs until a worker thread has completed the task’s job and returned a result.

On our application, this returned a 5X improvement to average throughput times for the same work being submitted to a single microservice performing the request processing. This goes to show that in our situation the majority of CPU was being spent on context switching between requests rather than servicing the CPU intensive task for each request.

In our case this translated to 5X less CPU required and a similar reduction in our AWS EC2 costs for this service as we needed less instances provisioned to support the same load.

AWS Snap Start for faster Java Lambda

After finding Native Java Lambda to be too fragile for runtimes we investigated AWS Snap Start to speed up our cold starts for Java Lambda. While not as fast as native, Snap Start is a supported AWS Runtime mode for Lambda and it is far easier to build and deploy compared to the requirements for native lambda.

How does Snap Start Work?

Snap Start runs up your Java lambda in the initialisation phase, then takes a VM snapshot. That snapshot becomes the starting point for a cold start when the lambda initialises, rather than the startup time of your java application.

With Spring Boot this shows a large decrease in cold start time as the JVM initialisation, reflection and general image setup is happening before the first request is sent.

Snap Start is configured by saving a Version of your lambda. This version phase takes the VM snapshot and loads that instead of the standard java runtime initialisation phase. The runtime required is the offical Amazon Lambda Runtime and no custom images are required.

What are the trade offs for Snap Start?

Version Publishing needs to be added to the lambda deployment. The deployment time is longer as that image needs to be taken when the version is published.

VM shared resources may behave differently to development as they are re-hydrated before use in the cold start case. For example DB connection pools will need to fail and reconnect as they be begin at request time in a disconnected state. However see AWS RDS Proxy for this serverless use case.

As at 26th August 2023 SnapStart is limited to the x86 Architecture for Lambda runtimes.

What are the speed differences?

After warm up there was no difference between a hot JVM and the native compiled hello world program. Cold start however showed a marked difference from memory settings of 512MB and higher due to the proportional allocation of more vCPU.

Times below are in milliseconds.

Comparison of Architecture v Lambda Memory Configuration
Graph of Lambda Cold Start timings

At 1GB with have approximately 1 vCPU for the lambda runtime which makes a significant difference to the cold start times. Memory settings higher than 1vCPU had little effect.

While native is over twice as fast as SnapStart the fragility of deployment for lambda and the massive increase in build times and agent CPU requirements due to compilation was un productive for our use cases.

Snap start adds around 3 minutes to deployments to take the version snapshot (on AWS resources) which we consider acceptable compared to the build agent increase that we needed to do for native (6vCPU and 8GB). As we are back to Java and scripting our agents are back down to 2vCPU and 2GB with build times less than 10 minutes.

How do you integrate Snap Start with AWS CDK?

This is a little tricky as there are not specific CDK Function props to enable SnapStart (as at 26th August 2023). With CDK we have to fall back to a cloud formation primitive to enable snap start and then take a version

Code example from out Open Source Spring Boot framework below.

final IFunction function = new Function(this,
                                                    .description("Lambda example with Java 17")
CfnFunction cfnFunction = (CfnFunction) function.getNode().getDefaultChild();
IFunction snapstartVersion = new Version(this,
                                         LAMBDA_FUNCTION_ID + "-snap",
                                                     .description("Snapstart Version")

In CDK because Version and Function both implement IFunction, you can pass a Version to route constructs as below.

String apiId = LAMBDA_FUNCTION_ID + "-api";
HttpApi api = new HttpApi(this, apiId, HttpApiProps.builder()
                                                   .description("Public API for %s".formatted(LAMBDA_FUNCTION_ID))
HttpLambdaIntegration integration = new HttpLambdaIntegration(LAMBDA_FUNCTION_ID + "-integration",
HttpRoute build = HttpRoute.Builder.create(this, LAMBDA_FUNCTION_ID + "-route")
                                   .routeKey(HttpRouteKey.with("/" + LAMBDA_FUNCTION_ID, HttpMethod.GET))

Note in the HttpLambdaIntegration that we pass a Version rather than the Function object. This produces the Cloudformation that links the API Gateway integration to your published Snap Start version of the Java Lambda.


Native Java AWS Lambda with Graal VM

Update: 20/8/2023: After the CDK announcement that node 16 is no longer supported after September 2023 we realised that we can’t run CDK and node on Amazon Linux2 for our build agents. We upgraded our agents to AL2023 and found out the native build produces incompatible binaries due to GLIBC upgrades, and Lambda does not support AL2023 runtimes.
We have given up with this native approach due to the fragility of the platform and are investigating AWS Snapstart which now has Java 17 support.

Update: 02/9/2023: We have switched to AWS Snap Start as it appears to be a better trade off for application portability. Short builds and no more binary compatibility issues.

Native Java AWS Lambda refers to Java program that has been compiled down to native instructions so we can get faster “cold start” times on AWS Lambda deployments.

Cold start is the initial time spent in a Lambda Function when it is first deployed by AWS and run up to respond to a request. These cold start times are visible to a caller has higher latency to the first lambda request. Java applications are known for their high cold start times due to the time taken to spin up the Java Virtual Machine and the loading of various java libraries.

We built a small framework that can assemble either a AWS Lambda Java runtime zip, or a provided container implementation of a hello world function. The container provided version is an Amazon Linux 2 Lambda Runtime with a bootstrap shell script that runs our Native Java implementation.

These example lambdas are available (open source) at https://bitbucket.org/limemojito/spring-boot-framework/src/master/development-test/

Note that these timings were against the raw hello java lambda (not the spring cloud function version).

public class MethodHandler {
    public String handleRequest(String input, Context context) {
        log.info("Input: " + input);
        return "Hello World - " + input;

Native Java AWS Lambda timings

We open with a “Cold Start” – the time taken to provision the Lambda Function and run the first request. Then a single request to the hot lambda to get the pre-JIT (Just-In-Time compiler) latency. Then ten requests to warm the lambda further so we have some JIT activity. Max Memory use is also shown to get a feel system usage. We run up to 1GB memory sizing to approach 1vCPU as per various discussions online.

Note that we run the lambda at various AWS lambda memory settings as there is a direct proportional link between vCPU allocation and the amount of memory allocated to a lambda (see AWS documentation).

This first set of timings is for a Java 17 Lambda Runtime container running a zip of the hello world function. Times are in milliseconds.

Java Container1282565121024
Cold Start6464506640543514
Max Mem126152150150
Java Container Results
Native Java1282565121024
Max Mem111119119119
Native Java Results

The comparison of the times below show the large performance gains for cold start.


From our results we have a 6X performance improvement in cold starts leading to sub second performance for the initial request.

The native version shows a more consistent warm lambda behaviour due to the native lambda compilation process. Note that the execution times seem to trend for both Java and native down to sub 10ms response times.

While there is a reduction in memory usage this is of no realisable benefit as we configure a larger memory size to get more of a vCPU allocation.

However be aware that build times increased markedly due to the compilation phase (from 2 minutes to 8 for a hello world application). This compilation phase is very CPU and memory intensive so we had to increase our build agents to 6vCPU and 8GB for compiles to work.

Integrate AWS Cognito and Spring Security

How to integrate AWS Cognito and Spring Security using JSON Web Tokens (JWT), Cognito groups and mapping to Spring Security Roles. Annotations are used to secure Java methods.

The various software components of the authorisation flow.
Authorisation flow for a web request.

AWS Cognito Configuration

  1. Configure a user pool.
  2. Apply a web client
  3. Create a user with a group.

The user pool can be created from the AWS web console. The User Pool represents a collection of users with attributes, for more information see the amazon documentation.

An app client should be created that can generate JWT tokens on authentication. An example client configuration is below, and can be created from the pool settings in the Amazon web console. This client uses a simple username/password flow to generate id, access and refresh tokens on a successful auth.

Note this form of client authentication flow is not recommended for production use.

User Password Auth Client

We can now add a group so that we can bind new users to a group membership. This is added from the group tab on the user pool console.

Creating a user

We can easily create a user using the aws command line.

aws cognito-idp admin-create-user --user-pool-id us-west-2_XXXXXXXX --username hello
aws cognito-idp admin-set-user-password --user-pool-id us-west-2_XXXXXXXX --username hello --password testtestTest1! --permanent
aws cognito-idp admin-add-user-to-group --user-pool-id us-west-2_XXXXXXXX --username hello --group-name Admin 

Fetching a JWT token

The curl example below will generate a token for our hello test user. Note that you will need to adjust the URL to the region your user pool is in, and the client id as required. The client ID can be retrieved from the App Client Information page in the AWS Cognito web console.

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id NOT_A_REAL_ID --auth-parameters USERNAME=hello,PASSWORD=testtestTest1!

Example access token


If you decode the access token, you will see we have the claim cognito:groups set to an array containing the group Admin. See https://jwt.io

Spring Configuration

Our example uses Spring Boot 2.7x and the following maven dependencies:


We start by configuring a Spring Security OAuth 2.0 Resource server. This resource server represents our service and will be guarded by the AWS Cognito access token. This JWT contains the cognito claims as configured in the Cognito User Pool.

This configuration is simply to point the issuer URL (JWT iss claim) to the Cognito Issuer URL for your User Pool.

          issuer-uri: https://cognito-idp.us-west-2.amazonaws.com/us-west-2_xxxxxxxxx

The following security configuration enables Spring Security method level authorisation using annotations, and configures the Resource Server to split the Cognito Groups claim into a set of roles that can be mapped by the Spring Security Framework.

This Spring Security configuration maps a default role, “USER” to all valid tokens, plus each of the group names in the JWT claim cognito:groups is mapped a a spring role of the same name. As per spring naming conventions, each role has the name prefixed with “ROLE_”. We also allow spring boot actuator in this example to function without any authentication, which gives us a health endpoint, etc. In production you will want to bar access to these URLs.

@EnableGlobalMethodSecurity(prePostEnabled = true, securedEnabled = true, jsr250Enabled = true)
public class SecurityConfig {

    public static final String ROLE_USER = "ROLE_USER";
    public static final String CLAIM_COGNITO_GROUPS = "cognito:groups";

    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        return http
                // actuator permit all
                .authorizeRequests((authz) -> authz.antMatchers("/actuator/**")
                // configuration access is secured.
                .authorizeRequests((authz) -> authz.anyRequest().authenticated())
                // oauth authority conversion

    private void oAuthRoleConversion(OAuth2ResourceServerConfigurer<HttpSecurity> oauth2) {

    private void jwtToGrantedAuthExtractor(OAuth2ResourceServerConfigurer<HttpSecurity>.JwtConfigurer jwtConfigurer) {

    private Converter<Jwt, ? extends AbstractAuthenticationToken> grantedAuthoritiesExtractor() {
        JwtAuthenticationConverter converter = new JwtAuthenticationConverter();
        return converter;

    private Collection<GrantedAuthority> userAuthoritiesMapper(Jwt jwt) {
        return mapCognitoAuthorities((List<String>) jwt.getClaims().getOrDefault(CLAIM_COGNITO_GROUPS, Collections.<String>emptyList()));

    private List<GrantedAuthority> mapCognitoAuthorities(List<String> groups) {
        log.debug("Found cognito groups {}", groups);
        List<GrantedAuthority> mapped = new ArrayList<>();
        mapped.add(new SimpleGrantedAuthority(ROLE_USER));
        groups.stream().map(role -> new SimpleGrantedAuthority("ROLE_" + role)).forEach(mapped::add);
        log.debug("Roles: {}", mapped);
        return mapped;

A now a code example of the annotations used to secure a method. The method below, annotated by PreAuthorize, requires a group of Admin to be linked to the user calling the method. Note that the role “Admin” amps to the spring security role “ROLE_Admin” which will be sourced from the Cognito group membership of “Admin” as previously configured in our Cognito setup above.

public Mono<JobInfo<TickDataLoadRequest>> create(@RequestBody TickDataLoadRequest tickDataLoadRequest) {
   return client.getTickDataLoadClient().create(tickDataLoadRequest);

That’s it! You now have a working example for configuring cognito and Spring Security to work together. As this is based on the Authorisation header with a bearer token, it will work with minimal configuration of API Gateway, Lambda, etc.

Spring Cloud Config to AWS Parameter Store easy conversion tool

Introducing our new utility to get you from YAML to AWS parameter store.


One of the drawbacks with Spring Cloud Configuration Server is that the server needs to be running before applications can be spun up. As we have become more cloud native on AWS we’ve wanted to move to AWS centric configuration systems, but to do that we needed a path from the existing git version control system (VCS) based config server.

So what we were missing was an easy conversion to AWS Parameter Store from Spring Cloud Config.


We liked Spring Cloud Config Server for many years, as it provided the following benefits:

  • git Version control with encryption-at-rest for application config.
  • a single point of control for all applications as we could set global configurations that affected all applications deployed.
  • A very simple bootstrap.yml file for startup without having to specify a lot of configuration.

We use Spring Cloud AWS (now awspring.io) libraries in a lot of our applications, and the support for both AWS parameter store and secrets manager are now baked into a spring boot starter.

A quick experiment showed some benefits for going to AWS parameter store based config

  • configuration always available without remote hosted config server.
  • use of secureString could replace our encryption at rest with config server
  • bootstrap is even simpler with just the application name required.
  • still supports “global” spring application configuration, which we use a lot with Jackson.

We like having our application config in git, as this gives us a simple code on branch, review and merge process using bitbucket. This was the only drawback with going to AWS PS, but surely could be solved with some code.

We’re in a slow move to serverless, so any chance to remove the need for a low utilisation server gets us a step closer to no clusters.


Our code and how to use it: https://bitbucket.org/limemojito/yaml-to-param-store.

So we are pleased to announce a small Open Source java jar that allows you to convert a single or a directory of yaml spring configuration files to AWS parameter store following the path and naming convention for Spring Cloud AWS. It included support for spring profiles conversion, AWS tagging the parameters and updating changed or new values on repeated runs. The command line tool does NOT delete parameters, though the code has support for removing an application by name including all of its profiles.

We have configured our own build server to checkout the configuration server repo, and run our tool over the yaml files to keep them in sync with parameter store.

Details on usage is available on bitbucket at https://bitbucket.org/limemojito/yaml-to-param-store.

For more information on using parameter store with a boot application, please see the configuration steps using Spring Cloud AWS in your Spring Boot application.