In a prior posting (AWS Lambda:Log Early Log often, Log EVERYTHING) I wrote about the virtues and value about having really in depth logging, especially when working with cloud services. Well today this logging saved my ASS a ton of detective work.
Little Background
I have a background job (Lambda that is called on a schedule) to create/update data cache in a DynamoDB table. Basically this job will pull data from one data source and attempt to push it as create/update/delete to our Dynamo table.
Today when I was running our application I noticed things were not loading right, in fact I had javascript errors because of null reference errors. I knew that the issue had to be in our data, but was not sure what was wrong. If I had not had a ton of logging (debug and info) I would have had to run our code locally and step though/debug code for hundreds of items of data.
However, because of in depth logging I was able to quickly go to CloudWatch and filter on a few key words and narrow hundreds/thousands of log entries down to 5. Once I had these 5 entries I was able to expand a few of those entries and found the error within seconds.
Total time to find the error was less than 5 minutes and I never opened a code editor or stepped into code.
The moral of this story, because I log everything, including data (no PII of course) I was able to quickly find the source of the error. Now to fix the code….
Till next time,
