Interlink Blog

   Inter-Networking Application Experts
 
Subscribe To
Our Blog Feed
posts - 491 comments - 91

OPNET Guest Post - All Transactions All the Time

OPNET LogoRe-Post from OPNET BLOG Industry Perspective: "All Transactions, All the Time"

How much data storage is necessary to effectively monitor and troubleshoot application performance? And, if we stored every transaction, how can we maintain cost-effective services in the cloud? We were recently faced with this question during the development of AppResponse Xpert SaaS Edition. This new product monitors end-user experience by inserting a JavaScript snippet into your website. Every hit to a monitored web page is sent to a SaaS portal where it is processed into metrics. We needed to decide what to do with these detailed records after analyzing them. Should we throw them out, sample them, or should we store and index all of these transactions?

“Problems only happen when you’re not watching” is an old troubleshooting proverb that helped drive our thinking. How often
have you looked over a co-workers shoulder as they exclaim, “This was slow 5 minutes ago, but it’s not now!” Indeed, reproducible application problems tend to be solved quickly after they’re reported. It’s the intermittent problems that linger. This is not because the underlying issues are more difficult to fix, but rather due to a lack of data in nearly every case. When we can reproduce the problem, we can also generate the detailed information to understand and resolve the issue. When we cannot reproduce it, we’re left with whatever metrics were collected by our monitoring tools. Depending on the level of detail that these tools choose to store, that data may be insufficient.

Some argue that storing this “High Definition” data is unnecessary; we disagree. Many APM products record metrics, but throw away the transactional data that those metrics are based on. If 99.9% of the transactions will never be reviewed, why bother capturing them? That would be a fair argument if it weren’t for the fact that the other 0.1% of the transactions is so extremely valuable. Rather than repeatedly identifying the same problem and trying hypothetical solutions based upon partial data, you can find the root cause and fix it correctly the first time if you have the right data to act on.

By making all of the transactions searchable, we can find answers to questions that we never thought of in advance. Each person may have unique questions about an application’s performance. Many of these will be captured by pre-calculated metrics, but not always. For example, “What are the IP addresses of readers of this blog post who are running Android 4.0 and had a DNS resolution time > 2 seconds?”

We believe IT needs easy access to all transactions all the time to maximize the value they provide to lines of business. Ultimately, this is why we made the ambitious choice to store all transactions all the time for our AppResponse Xpert SaaS solution, in line with prior design decisions we made for our APM Xpert suite. Whether that data is network packet streams over 10 Gbps networks, code-level transaction traces on dozens of virtualized systems supporting a production application, or end-user page hit records for a global commerce application hosted in “the cloud,” IT needs easy access.

Does the cost of storing so much data outweigh the benefit? Not if done properly! Many industries are grappling with Big Data challenges and the APM business is no different. We worked very hard over many years to build the proprietary NoSQL database technology that enables the OPNET APM Xpert suite to conquer the Big Data problem for APM use cases. Combined with analytics for automating the problem-solving process, the result of storing every transaction is faster answers to the hard questions.

Thanks for the post Matt and Todd! 

Be sure to Contact your Interlink Account Manager to find out how to become an OPNET Synergy Partner.

Ann

Posted by Ann Schaner on Wed, 30 May 2012 16:43:18 -0500

Feedback re: OPNET Guest Post - All Transactions All the Time

No comments posted yet...Comments are moderated.
Name
Url  
Email  (will not be published) (required)  
Comment (required)  
 (comments are moderated and will be posted shortly)