0

Lessons learned using Reactor

ColdFusion
After being a full-fledge supporter of Reactor for about the past 10 months, I have to admit that while it is without a doubt a *wonderful* tool, like anything it does have its limitations.

As many know, we rolled InstantSpot out as a MachII /Reactor project, and have been using Reactor exclusively for managing our objects and database abstraction. We built it so that all data is accessed and mutated through the reactor objects. From the very beginning of the rollout, we knew that we had some performance issues that we need to deal with, as page requests on the Spot sites themselves were taking a bit longer than we would have liked (OK, a *lot* longer)

As we started getting a little load on the system we saw some scary things. In heavy traffic periods the server just choked with the processor staying pegged out. Even just a single page request was sending the processor to 100% for several seconds. Immediately we knew we need to change some things, and finally did so over the past 3 days. Here is what we learned and what our resolution was:

First, let me be clear... Reactor was NOT the problem. However, the way that we used it was.

To put it bluntly object instantiation in Reactor is pretty expensive. Especially as the complex relationships get bigger and bigger. We found that out server was having to do *far* too much work for what we really needed out of it. For instance, when we load up a site record, Reactor does a wonderful job of transversing through the entire object hierarchy and having whatever properties and iterators ready for use. As we spec'ed out the project we thought we had accounted for all this work pretty well by using some fairly clever caching techniques. However, we found that we hadn't accounted for it well enough!

While it is really cool to be able to use the getters and setters with such ease, we came to the realization that when it comes to simply displaying output of a site, there is really not any need for all that overhead. We decided to step back and re-evaluate.

What we ended up with was a series of complex structures of data (yes structures!) that are lazily loaded rather than all loaded being up front. Additionally instead of instantiating Reactor objects and using them, we instead chose to use the Reactor query objects to return whatever recordsets we needed.

For the control panel functionality, we continue to use Reactor objects, as it makes adding/modifying/deleting data an absolutely pleasurable task, and we are no where near crisis mode on the performance in those areas.

So, the outcome?

The difference has been incredible! Now rather than a page request pegging the processor for a couple of seconds, we see it spike way under 100% for a split second. Hopefully this will be of help some of you that are considering using Reactor for your projects.

From our experience to date, I think it is still a great choice for database abstraction, just be very mindful of how you use it.

tags:
ColdFusion
Brian Rinaldi said:
 
Good post Dave. I found similar issues with Reactor is the past when I have used it both personally and at work - granted, I have not used the most recent version. However, I think you are hedging a bit here. How is this not Reactor's fault? I mean, I think Doug has put a lot of hard work into Reactor, and this isn't meant as a criticism (and I think it can have its limited uses), but that doesn't mean that it isn't Reactor's fault that you need to put into place obtuse caching mechanisms in order for Reactor to do what it was supposed to do.

This reminds me a lot of that post where I discussed an essay calling ORM the Vietnam of computer science. It discussed how you keep putting more and more resources and effort to make it work when it was originally intended to make your life easier. This doesn't sound easier. And what kind of load are we even talking? I backed off when I considered the traffic of say Star Wars or Transformers performing when under moderate load many issues cropped up. Perhaps one day with further improvement this will be moderated, but for the time being I remain open to the *idea* but unsold on the reality.
 
posted 824 days ago
Add Comment Reply to: this comment OR this thread
 
Peter J. Farrell said:
 
Metrics! Metrics! Ok, being a crazy about that...

Anyways, I was curious about the load since saying under heaviler load is very subjective. Would you mind sharing some metrics on your typical load (i.e. number of users, total requests / minute or sec, etc.) Thanks!

BTW, instantspot is very cool.

.pjf
 
posted 824 days ago
Add Comment Reply to: this comment OR this thread
 
Sami Hoda said:
 
Yes, I'd love to learn more!
 
posted 824 days ago
Add Comment Reply to: this comment OR this thread
 
 
As silly as it may sound, we really don't have any good performance analyzing tools in place at the moment. We are still a two-developers-in-their-off-hours team, and can't afford anything too fancy just yet! We were able to watch requests in real-time using the processor monitor.

We still don't have InstantSpot in its permanent home. Currently it is running in a Windows virtual machine on an Ubuntu Server. We will be moving it to the native Ubuntu platform as soon as we can. Once we get that in place we will add in some better analyzers.

As for traffic, "heavier" to us was not extraordinarily heavy. Only occassionally have we passed the multple requests per second mark. Even at times of 1 request every 5-10 seconds, we were fully aware of the problem!

Thanks for the comments guys.
 
posted 824 days ago
Add Comment Reply to: this comment OR this thread
 
SiamesePurr771 said:
 
Brain R, I couldn't agree with you more. So many developers who praise tools like Reactor don't really understand the full implications of such systems. How many people work on big systems like you and me? And how many big systems make use of Reactor?

Now I'm not attacking Reactor for small-size projects *at all*. For rapid development of small applications, Reactor can be great. But what was the old thought about using a framework? You might not need one when you start, but when your application is growing is not the time to be wishing you had a framework. Taken the opposite direction, you can get away with Reactor now, but when your application is growing, do you want to deal with refactoring it away from Reactor?

Big systems and little systems are like apples and oranges.

-Purr
 
posted 824 days ago
Add Comment Reply to: this comment OR this thread
 
Phillip Holmes said:
 
I haven't looked at reactor all that much but it sounds like a singleton pattern for your object instantiation would do you a ton of good.
 
posted 791 days ago
Add Comment Reply to: this comment OR this thread
 
 
Phillip, it goes without saying that we make use of singletons in this and every other application. In InstantSpot, any object in our system that doesn't have dynamic state is singleton. I am fairly sure there is no exception to that.
 
posted 791 days ago
Add Comment Reply to: this comment OR this thread
 
 
Phillip, here is one example of what I am talking about:
http://daveshuck.com/blog/index.cfm/2006/11/9/Fact...
 
posted 790 days ago
Add Comment Reply to: this comment OR this thread
 

Search