Skip to main content
Sumo Logic

AWS S3 Scan Interval for Sources

 

When configuring an Amazon S3 Source, you'll set the scan interval, which defines the waiting time between scans of the objects in your S3 bucket. It's important to set an interval that is long enough to allow new files to be uploaded, but is not too short that scans are performed without any new files being available to upload.

Setting a scan interval that's too short could cause additional charges to your AWS account. When Sumo Logic scans the contents of a bucket for new files, it will perform a number of listings, which may increase with the number of objects in the bucket. Sumo Logic can't determine if the data in your S3 bucket has changed without listing each object in every scan interval.

In addition, be aware that uploading data to Sumo Logic can incur data transfer charges from AWS. You can view current pricing for list and data transferring here. To get an idea of what your charges could be, we recommend using the Simple Monthly Calculator.

Setting a scan interval that's too long can cause a delay in new files being uploaded in a timely manner. If no new files are found in a scan, the scan interval is automatically doubled, up to a maximum of 1 hour. For example, if your scan interval is set to the default of 5 minutes, after a scan is completed with no new files identified, the scan interval goes to 10 minutes. Likewise, if no new files are found in 10 minutes, the scan interval changes to 20 minutes. This continues up until the interval is set to 1 hour, which means that uploading a new file could be delayed up to 1 hour.