This last weekend I spent my time at a convetion, being in line after line after line. It got me thinking, As bored as I was in all of those lines, and as much money and time was being waisted on those lines, is there a better way?
Data is the same way, if you are building up a large queue of data, is there a better way, than to just accumulate it in a stack?
What is common at conventions? The exhibit hall and live presentation pannels.
Why not assign tickets to these people, that has a queuing time on it, for them. Then the bulk of the people in the beggining could be out doing things, while the initial line was shrinkin. Towards the end of the line, the bulk of the people could be seated while the remainder are coming in.
This results in shorter lines and less of a crowd mess.
The same thing applies to your data. Rather than dumping it into a large queue, keep it in the DB, and pull it out in batches that you are going to process. If you are using connection pools, then you even keep the time to a minimum. SQL 2005 also uses caching, now you have speed.
Another issue at the convention was to go from the main convention center, where all of the pannels and workshops were, to the exhibit area was clean and easy. But going back was a nightmare. You had to travel a mile for a quarter mile walk.
The same is with your data. Normally you do not need a queue to send your data from process to process. All it does is keep your data from being approachable when bouncing between processes.
With the convention, changing the front door, or making one door entrance only, create an exit only, and take the old entrance only door, and make it bidirectional, would have solved the problem. The bulk of the people were there for the exhibit hall, not the other services. Make that access happy and everyone is happy.
Puting this pattern to data, is simple. One input only process, one output only process, one bi-directional process. The the usecase for the data travelling determine which door to take.
And that is how an architect/analyst looks at life while on vacation. Some vacation.