LogoLogo
  • 60East Frequently Asked Questions
  • Release and Hotfix Policies
    • What is a "Preview" version of the AMPS Server?
    • What is the 60East Hotfix Policy?
  • Getting Support
    • What's in a minidump file? Does it include the data that's in AMPS?
    • How do I use WinSCP to upload files to the 60East SFTP support server?
    • Can everyone see things I put in support tickets?
    • How can I split large log files to upload them?
    • Is replication between different versions of AMPS supported?
  • AMPS Features and Capabilities
    • General Topics
      • What software/hardware is required to run AMPS?
      • Content filtering sounds cool, how does it work?
      • Is AMPS a point-to-point messaging product?
      • What database does AMPS use?
      • Can I publish and subscribe at the same time?
      • What CRC32 implementation does AMPS use?
      • How does AMPS support UTF-8 and Unicode?
    • Setting up Instances
      • AMPS on Apple Silicon: Running a Development Instance
      • Can I run AMPS on Windows?
      • How do I run AMPS in a container?
      • How can I have AMPS ignore SIGHUP?
    • State-of-the-World
      • What is the difference between a Topic in the SOW and the Transaction Log?
      • How does a messaging product support queries?
      • How do I subscribe to a view or an aggregate in my program?
      • How can I paginate a SOW query?
      • What happens if data changes while a SOW query is running?
      • Does AMPS return messages from the SOW in the order published?
      • How many columns/fields can I have in a SOW topic? In a view?
      • What is "Focus" in AMPS?
      • What are some ways to consume large SOW queries?
      • Do I need to list every field in a SOW topic configuration?
      • How can I get the number of records in a SOW topic?
      • How can I delete messages from a SOW topic?
      • Should I use the SOW delete action or message expiration to maintain my SOW topic?
      • How do I configure SOW storage parameters?
      • How do I get only the first 100 records from a SOW query?
      • What is the difference between historical SOW and bookmark replay?
      • Do I have to add configuration for every topic I want to publish to?
      • How does AMPS construct the SowKey for a message?
      • What happens if I change the key of a SOW topic?
    • Message Formats and Content Filtering
      • Why should I use AMPS content filtering?
      • What is the default message type for AMPS?
      • Does AMPS validate messages?
      • How does AMPS handle invalid FIX/NVFIX?
      • Can AMPS filters do case-insensitive comparisons?
      • Can I just send uninterpreted binary messages with AMPS?
      • What's BFlat?
    • Incremental Update and Change Set Delivery
      • Do I have to use delta publish to have a delta subscription?
    • Replication and High Availability
      • Does AMPS support replication?
      • Why does publishFlush() wait forever when I'm using replication?
      • Are SOW topics that are replicated guaranteed to be in sync on all instances?
      • How do I set up a failover group in my replication destination
      • How does AMPS report the specific failure in replication validation?
    • Messaging Paradigms
      • How can I do ad hoc message routing in AMPS?
      • What is a bookmark subscription?
    • Aggregation and Analytics
      • What is a View and how can it be used for aggregation?
    • Performance Questions
      • How fast is AMPS?
      • Which is faster, a regular subscription or a bookmark subscription that uses the "live" option?
  • Application Development with AMPS Clients
    • Unique Client Naming and AMPS
    • Which version of the client do I need for a specific version of AMPS?
    • How does an application pass credentials to AMPS?
    • What are the entries in a bookmark store file?
    • Is there a way to use AMPS itself as a bookmark store?
    • Should I use TCP_NODELAY for connections to AMPS?
    • Should I use Command or Message to send commands to AMPS?
    • How do I set the CorrelationId when I publish a message?
    • Why does publishFlush() wait forever when I'm using replication?
    • Can I send commands to AMPS from inside a message handler?
    • How do I use a custom Authenticator with HAClient?
    • What's the difference between CommandId, QueryId, and SubId?
    • Do I need to use an HAClient?
    • What character set is supported for CorrelationID?
    • Can I add a correlation token to the header of a message?
    • How do I change the filter on a subscription?
    • Can a client have more than one subscription at a time?
    • Can a subscriber tell who published a message?
    • Why do messages sometimes have strange data when I'm using asynchronous processing?
    • What character encoding is being used by the AMPS clients?
    • Why does my HAClient appear to hang when my server is down or the URI is incorrect?
    • How can I reconnect to my AMPS server when a disconnect occurs?
    • How can I use AMPS with a XAML application?
    • Can I enforce unique client naming without creating a transaction log file?
    • How can I tell if my program is not entitled to publish to a topic?
    • When using the Python client, will the background thread exit when the program exits?
    • Do I need more than one AMPS client object in my program?
    • Why does AMPS disconnect a client with a 'name in use' error?
    • In bookmark subscribes, why does EPOCH give me the same messages as MOST_RECENT?
  • Operations and Deployment
    • How can I manage complicated configuration files?
    • How can I find the version of my AMPS server?
    • Can I run multiple instances of AMPS on the same system?
    • What's in a minidump file? Does it include the data that's in AMPS?
    • What Does it Mean when dmesg Says AMPS Blocked for More than 120 Seconds
    • What is the 60East Hotfix Policy?
    • What software/hardware is required to run AMPS?
    • What does seconds_behind mean?
    • How do I run AMPS in a Docker container?
    • How do I use custom authentication with spark?
    • What is client reaping?
    • Is replication between different versions of AMPS supported?
    • How much bandwidth will replication require?
    • What value is normal for a specific performance counter?
    • Why is AMPS logging a memory allocation failure when the system has plenty of free memory?
    • Can I check a configuration file without restarting AMPS?
    • What's the difference between sync and async acknowledgement in AMPS replication?
    • Do I need to worry when a regular expression topic name can't be validated?
    • Can AMPS run as a User Service?
    • Can I dynamically choose which topics to replicate?
    • Should we have Hyper-threading enabled on AMPS servers?
    • Does AMPS correctly handle leap seconds?
    • How do I copy a SOW from one AMPS instance to another?
    • How can I control the size of the statistics database?
    • How can I configure AMPS to use a specific network adapter?
    • How can I limit the amount of memory AMPS consumes?
    • What if I need to move messages from one AMPS instance to another?
    • Is PCIe SSD a requirement when using a transaction log?
    • AMPS Log Messages
      • How do I get more information on an AMPS error?
      • How can I tell where a connection is coming from?
      • Does AMPS log current process limits?
      • What does it mean when AMPS has detected that it may not be running correctly?
      • How can I keep only the last week of error logs?
      • What is the difference between "execution time" and "elapsed time" in the logs?
      • What does a "potential stuck thread" message mean?
      • What does a "message cache is throttled" warning mean?
      • What does "Replication has detected an existing connection" mean? Why does AMPS close the connection?
      • Where is the list of all the event and error messages in AMPS?
      • What can cause an "unknown command received" or "parse format error"?
      • Why does AMPS disconnect a client with a 'name in use' error?
      • How do I exclude passwords in trace logging?
Powered by GitBook
On this page
  1. Operations and Deployment
  2. AMPS Log Messages

What does it mean when AMPS has detected that it may not be running correctly?

The AMPS server is highly concurrent, which means that the server itself has many threads doing work at the same time.

AMPS includes a thread monitor that keeps track of how fast AMPS threads are processing the work provided. If a thread does not report progress when expected, AMPS logs message 30-0000 to indicate that the thread has failed to report when expected. This means that it is possible that the thread may be stuck -- unable to make progress -- or an operation may be taking longer than expected. For example, an operation involving a large file may take an unexpectedly long time on a busy SAN and cause AMPS to report that the thread performing the operation is potentially stuck. Likewise, if an AMPS instance is located on a server with other processes that require large amounts of CPU, network, or storage capacity, AMPS may report a potential stuck thread when AMPS is unable to acquire enough resources to make progress in a timely manner.

If a thread continues to fail to report progress, AMPS begins to generate diagnostic minidumps to capture the current position in the AMPS code for all of the threads in the instance. Since troubleshooting a slow thread is easiest to do with more than one snapshot of the where code execution is for each thread, AMPS begins to produce diagnostic minidumps at the point at which the server detects that there may be a problem emerging but before the problem has reached a critical level.

AMPS logs error 01-0022 when AMPS generates a diagnostic minidump. AMPS logs this error at critical level since, at the time the error is logged, the thread has not reported for long enough that AMPS will shut down if the problem persists. AMPS is often able to recover from this particular error, but this typically indicates a capacity problem with the system hosting AMPS or a problem in the AMPS server.

In many cases, the server recovers from a temporarily slow thread without incident. In those cases, the AMPS server logs indicating which thread was slow may provide the most useful information on the slowdown. In cases where the server shuts down due to a thread becoming stuck, both the logs and minidump files should be provided to support.

Last updated 8 months ago