What does a "message cache is throttled" warning mean?

As with all warning level messages, this means that AMPS detected a potential problem that can be automatically corrected, but may require operator attention if the problem persists.

This particular warning simply means that AMPS is out of resources to complete current messaging requests (and it's waited longer than 1 second for the resources to become available.) If these messages persist, or if performance degrades at the same time these warnings are logged, this can indicate that the system is overburdened and the system administrator may need to increase system capacity, or rebalance applications onto another system.

Here's more detail on what the message means:

Internally, AMPS uses message "objects" as reusable state to improve cache locality and minimize memory allocations from the OS. AMPS allocates enough message objects to fill all of the processing slots within the AMPS pipeline.

However, sometimes every slot is full because of some resource availability anomaly (not enough CPU, VM system is paging, swapping, other rogue processes swamping the system, etc.). In this case, the inbound message processor can't grab an available message object from the cache and has to wait. AMPS logs a warning to indicate that the wait occurred.

Notice that this isn't just a matter of needing to allocate more memory: this indicates that all of the available slots in the message processing pipeline are full, so even if AMPS were to allocate another message object, the processing thread would need to wait for an available slot in the pipeline.