How do I copy a SOW from one AMPS instance to another?

Use the spark command line tool. In current versions of AMPS, spark is included in the AMPS distribution.

The instructions in this FAQ are useful for a one-time transfer, for example, loading sample data into a QA environment for testing. For production use, AMPS replication is the recommended approach for low-latency delivery of messages from instance to instance as described in the AMPS User's Guide.

These instructions assume that your data does not contain embedded newlines: if your data does contain newlines, we recommend writing a simple program that simply does a SOW query on one instance and publishes the results to the other instance or using the -file option in more recent versions of spark.

Copy from a SOW File

To recreate a SOW from a server that is no longer running, use amps_sow_dump to extract the records from the SOW and use spark to publish those records into the destination AMPS instance.

$ amps_sow_dump --zip-file=records.zip /path/to/sow/file.sow
$ spark publish –server new_server –topic new_topic –type message_type –file records.zip

Copying from a SOW topic in a running instance

To recreate a SOW on a server that is currently running, use the spark sow command to retrieve records from the source topic, and use the spark publish command to publish records to the destination topic.

$ spark sow -server old_server topic old_topic -type message_type 
  | spark publish –server new_server –topic new_topic –type message_type

Notice that, because this is an operation on a running instance of AMPS, any updates that are made to the SOW topic after the sow command runs will not be published to the new topic.

If the messages are binary or contain embedded delimiter characters, and the number of records can be easily stored on the system where the command is being run, you can use the -file flag to store messages to a compressed file, and then republish the messages from that file.

$ spark sow -server old_server topic old_topic -type message_type -file messages.zip
$ spark publish –server new_server –topic new_topic –type message_type  -file messages.zip

Copying the SOW File Directly

If both instances of AMPS have compatible SOW file versions (see the file version appendix in the AMPS User Guide), *and *both instances are currently shut down, it can work to copy the SOW file from one instance of AMPS to the other. This is most useful in cases where a server is migrating (so all of the data is being moved) or where the current state of the SOW file is being preserved for troubleshooting or diagnostic purposes.

The limitation of this approach is that the messages in the SOW file will not be in the transaction log of the destination server. A replay from the transaction log may not include the messages in the copied file (but may contain messages that are not in the file). Likewise, a historical SOW and subscribe could produce unexpected results. Last, but not least, if the file is damaged or removed, the destination server cannot rebuild the file from the transaction log.

It's important to be sure that both AMPS instances are shutdown when the file is copied. Copying a SOW file while AMPS is running may produce a copy that is incomplete or corrupted. Updating a SOW file while AMPS is running (for example, by overwriting the file with a copy from another server) may produce unexpected results from SOW queries, corrupt the file, or cause AMPS to exit.

For some applications, these limitations are acceptable. For other applications, the fact that the transaction log and the topic in the SOW do not contain the same information makes this approach unworkable.