Monday, April 18, 2016

An "Ideal" Interview with the Tester for Performance Engineering Position Part 2

For part 1, click here

Interviewer: What do you do when you are asked to start performance testing on a web application. 

Candidate: First of all, I try to understand the application, its main functionalities, its architecture and technologies used.
Of course, this information will be given by developers. We will also need developers to help in the later stages of performance testing. 

After that, I normally follow 5 steps to do performance testing. 

The first step is to identify the performance test environment. 

The good rule of thumb is that our test environment should be exactly similar to the production environment. But in many organizations, this could not be the situation due to cost. So we try to create an environment which is as close to production as possible. The Test environment could be a Virtual machine in which we can increase or decrease the RAM and processing power when needed.

Some critical factors to consider are:
Network Limitations, Hardware Configurations, Load Generation Tool, Logging mechanism, Licensing constraints etc. We do need the help of Network/IT team in designing the test environment. 

The second and most important step is to identify the performance acceptance criteria. This will be provided by business analysts, product owners who understand the business side of the application. If acceptance criteria are not clearly defined, the whole performance testing activity will become haphazard and inconclusive. 

Interviewer: So can you give some examples of some performance acceptance criteria?

Candidate: Performance criteria are highly subject to the context of the web application. But I can give some examples. Normally performance characteristics include 

Response Time:
For example, Response times for all business operations during normal and peak load should not exceed 6 seconds.

For example, the system must support 25 book orders per second. 

Resource utilization: 
For example, No server should have sustained processor utilization above 80 percent under any anticipated load or No single requested report is permitted to lock more than 20 MB of RAM and 15-percent processor utilization on the Data Cube Server.

There could be any number of performance acceptance criteria, but they should all be quantifiable and correlate to user satisfaction. 

Interviewer: Good. You mention throughput, can you define what is it?

Candidate: It is the unit number of work server can handle per time unit. You can measure throughput in terms of
requests per second or
reports per year or
hits per second or
calls per day or any other number per unit time.
The higher the throughput, the higher the performance of the server is.

Interviewer: Great. So what if we don't have any idea for user expectation. What strategy should we use?

Candidate: You can just ask the user that what performance you are expecting?

Interviewer: (Laughing...) no no. I meant to say that suppose we are building a product and until now we don't have any user for that product because we have not launched that product. So how would we define the performance criteria for that?

Candidate: In that case you should follow benchmarking or baselining.

Interviewer: What is benchmarking and baselining?

Candidate: Benchmark tests are the process of comparing the performance of your system against industry standards given by other organizations. One example of benchmarking is that you can see how your competitor's application performance is. Another example is that you can read the research papers from top performance engineers and see what they propose for ideal benchmarks for your domain.

Baseline, on the other hand, is a comparison with your previous releases. You can set one particular release as your baseline. All future releases performances will be compared to that baseline. If results of any release are much degraded from the baseline that means something is wrong with the performance. if any release is performing better, you can change your baseline and set this release as your baseline. 

Interviewer: Good Answer. So what is the next step after identifying performance acceptance criteria?

Candidate: Next step is to design tests. When designing tests, we should identify key usage scenarios, determining appropriate variability across users, identifying and generating test data and specifying the metrics to be collected.  

When designing tests, our goal should be to create real-world simulations in order to get results which help stakeholders to make informed business decisions. We should consider 
Most common usage scenarios, 
Business-critical usage scenarios, 
Performance intensive business scenarios 
And High visibility usage scenarios.

It is useful to identify the metrics related to the performance acceptance criteria during test design so that the method of collecting those metrics can be integrated into the tests when implementing the test design

Interviewer: What other considerations you should follow while designing tests?

Candidate: When we design realistic test scenarios, we should incorporate realistic simulations of user delays and think times which are crucial to the accuracy of the test. Secondly, we should not allow the tool capabilities to influence our test design decisions. Better tests almost always result from designing tests on the assumption that they can be executed. After that, we should see that what tool can do. Thirdly we should involve the developers and network administrators in the process of determining which metrics are likely to add value and which method best integrates the capturing of those metrics into the test. For example, if we want to know the CPU utilization, we should take help from network administrators that how we would capture CPU utilization in our test. 

Interviewer: Good. So up till now, we have discussed 3 steps while starting performance testing. Up till 3rd step, what do you feel some biggest challenges are?

Candidate: I think the biggest challenge is to correctly identify the performance acceptance criteria. I cannot force enough is to how important this step is and how important is that for every stakeholder to involve in this step. Developers, Testers, Product Managers, Network Engineer and Performance Engineers they all should be part of this decision. 

The second biggest challenge is getting our first relatively realistic test implemented with users generally being simulated in such a way that the application under test cannot legitimately tell the difference between the simulated users and real users. This takes significantly longer time and again input from all stakeholders is very necessary for this step. 

Interviewer: Right. I agree. So what is the next step?

Candidate: Next 2 steps are remaining. Execution of the tests and Analysis of results. 

To be continued.....


  1. Please keep continuing this series. Its really great and very helpful. Cleared up some concepts too and acing the interview too....

  2. Very informative. Please post the next part.

  3. This comment has been removed by the author.

  4. I have to voice my passion for your kindness giving support to those people that should have guidance on this important matter.devops training in chennai

    oracle training in bangalore

  5. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    dotnet training in bangalore

  6. Really it was an awesome article.Very interesting to read.You have provided an nice article....Thanks for sharing..Embedded Project Center in Chennai | Embedded Project Center in Velachery

  7. Thank you very much for an informaticle blog. Really I enjoyed with the discussion and also it is very much usefull blog.
    Jmeter Training Online

  8. Good article.

    Pakistani/Bollywood Celebrities Wiki/Biography News/Gossip
    Team | Pakistani/Bollywood Celebrities Wiki/Biography News/Gossip

  9. Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.

    machine learning training in chennai
    best training insitute for machine learning
    top institutes for machine learning in chennai
    Android training in Chennai
    PMP training in chennai

  10. Thank you so much for your information,its very useful and helful to me.Keep updating and sharing. Thank you.
    RPA training in chennai | UiPath training in chennai

  11. Such a Great Article!! I learned something new from your blog. Amazing stuff. I would like to follow your blog frequently. Keep Rocking!!
    Blue Prism training in chennai | Best Blue Prism Training Institute in Chennai

  12. Wow!! Really a nice Article. Thank you so much for your efforts. Definitely, it will be helpful for others. I would like to follow your blog. Share more like this. Thanks Again.
    iot training in Chennai | Best iot Training Institute in Chennai

  13. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.
    Advanced AWS Course Interview Questions And Answers, Top 250+AWS Jobs Interviews Questions and Answers 2018
    Advanced AWS Jobs Interview questions and answers |Best Top 110 AWS Interview Question and Answers – india

  14. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    microsoft azure training in bangalore
    rpa interview questions and answers
    automation anywhere interview questions and answers
    blueprism interview questions and answers
    uipath interview questions and answers
    rpa training in bangalore

  15. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....
    Best Devops Training in pune
    Devops Training in Bangalore
    Power bi training in Chennai

  16. I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.
    Devops Training in Chennai | Devops Training Institute in Chennai

  17. Good job! Fruitful article. I like this very much. It is very useful for my research. It shows your interest in this topic very well. I hope you will post some more information about the software. Please keep sharing!!
    Blue Prism Training in Chennai
    Blue Prism Training Chennai
    Blue Prism Training in Velachery
    Blue Prism Training in OMR
    Blue Prism Training in TNagar
    Blue Prism Training in Annanagar

  18. Thank you for providing such an awesome article and it is a very useful blog for others to read.

    Oracle ICS Online Training

  19. Thanks for providing a useful article containing valuable information. start learning the best online software courses.

    Workday Online Training

  20. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

  21. Attend The Python training in bangalore From ExcelR. Practical Python training in bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Python training in bangalore.
    python training in bangalore


  22. I have express a few of the articles on your website now, and I really like your style of Python classes in pune blogging. I added it to my favorite’s blog site list and will be checking back soon…

  23. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Python Training in Electronic City

  24. Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab Python training in pune your rss feed to stay informed of any updates.


  25. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading Python classes in pune new articles. Keep up the good work!

  26. Attend The Data Analytics Course Bangalore From ExcelR. Practical Data Analytics Course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analytics Course Bangalore.
    ExcelR Data Analytics Course Bangalore

  27. I have gone through your post and I found it very helpfull. Looking forward to see more post from you. Artificial Intelligence Course

  28. Thank you so much for sharing such a valuable information...

  29. Great Article. Thank you for sharing! Really an awesome post for every one.

    IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

    JavaScript Training in Chennai

    JavaScript Training in Chennai


  30. You write this post very carefully I think, which is easily understandable to me. Not only this, but another post is also good. As a newbie, this info is really helpful for me. Thanks to you.
    Tally ERP 9 Training
    tally classes
    Tally Training institute in Chennai
    Tally course in Chennai

  31. Your willingness and intention to help us is really astonishing. Thanks a lot for such a kind and nice attitude towards the discussed issue.

  32. I really appreciate your method to explaining, I hope to see more posts from your blog. thank you!
    Digital Marketing Course In Kolkata
    Web Design Course In Kolkata
    SEO Course In Kolkata

  33. Really awesome blog!!! I finally found a great post here.I really enjoyed reading this article. Thanks for sharing valuable information.
    Data Science Course in Marathahalli
    Data Science Course Training in Bangalore