May 2006 - Posts
Red-eye flights always mess me up, but it is good to be back home. It will probably take me the majority of the weekend to get back to normal, but at least I'll get to see the family today. I was kinda missin' the lot of 'em. I may have to figure out something fun to do with the boys...
Well, we've pretty much exhausted all of the scenarios we can test without moving to bigger hardware, or possibly without getting different builds from the team. We are in the process of trying to rewrite the calc script in the most optimal way possible, with the help of one of the devs who works on the the query processing engine, in the hopes that we can squeeze a little bit more performance out of the model.
Overall, we have run through a lot of cases, and reported a number of cases and issues, and have gotten a lot of information and help from the team. Unfortunately, there isn't a magic setting (or bunch of settings) that will make our scalability situation vastly better, but we have learned a lot of stuff, and we will have some things we can follow up on, and they have things they will follow up on. We will be trying to roll what we have learned back into the product where possible, and into our support processes where there are things that are more conditional.
I am pretty fried at this point, even though it is only about 3 PM on Friday. We put in somewhat long days, and I think James is getting pretty sick of us... ;) It is definitely the case that he was very hospitable and between him and the team, we have been very well taken care of. Kudos to the folks in Redmond for sharing their facilities and knowledge with us this week. A few more hours of testing and we'll be on the red-eye back to the midwest for the holiday weekend. It will be good to be home...
Well, we've been running tests for a few days, and although the results aren't as good as I hoped, we're making good progress and getting quality feedback from the team. Unfortunately, there is still a ways to go to get our stuff to a point where we have a supportable product for large applications, but progress is progress, and we're making that. I'd give more details, but we're under NDA, and I have no interest in violating that, so we'll leave it at the fact that we continue to spend a lot of time measuring and tweaking and hopefully we'll have some clear steps to go forward by the time we leave (we have several already).
On another note, the hardware here is pretty sweet. We're only playing with a couple of beefy 4-ways, but they have some pretty insane hardware available for testing large scale pure DB applications (NUMA systems with up to 32 procs). I'd love to have about 6 weeks of play time on some of this hardware. I doubt much of our software would be able to directly benefit from that power without some pretty big modifications to ensure memory accesses weren't thrashing across procs and that we weren't lock convoying all over the place, but it would still be pretty cool to see how we scaled up, and identify the areas where we stumble on killer hardware.
Well, after one full day of work, we have only been able to capture a couple of traces of runs through our test script, and spent a lot of time looking at problems totally unrelated to our product or the tests we want to run. This morning, I helped Mike get our 32-bit platform set up for our 20-user run, and we're sitting aside while the SQL/AS team debugs the 64-bit platform figuring out why the private build won't run on 64-bit right now. We're hoping to get through at least 20, 40, and 80 user tests on 2 and 4 procs on 32-bit before this afternoon. That is 5 hours of tests, so we don't have much leeway for problems... Hopefully everything goes swimmingly. 64-bit? Well, we'll see...
The cool part is this lab is set up very nicely, and our contacts here are pulling in devs from the SSAS team as necessary to make sure we are able to proceed. Right now we've got 10 machines -- all outfitted appropriately dedicated to getting these tests executed. Pretty nice setup -- I wish we had this at our disposal more often!
On another note, it is raining all the time -- as is typical for the Pacific Northwest. It is actually very pretty with all of the new growth in the trees around campus and surrounding areas. Still, I wouldn't mind seeing the sun shine during the brief moments we leave the confines of the lab to grab a coffee or a meal.
I got in to Sea-Tac on time at 9:25PM last night, but Mike's flight was a bit late, landing closer to 12:00 midnight, rather than 11:20. We finally got in the car around 12:40 and were settling in at the hotel by about 1 AM.
I woke up at 5:30 AM local time, since that would be about right at home. Unfortunately, that means I've only had 4 hours sleep, so I am bit tired. We've been in the lab since about 9, and we actually have everything running already, so that is good progress. Hopefully our results will be better than what we've seen at home.
After an abbreviated weekend, I am heading to Seattle (well, Bellevue/Redmond, actually) tonight for a week at a Scalability Lab on campus at Microsoft. Should be interesting, but I miss the kids and Jodi already. :(
I'll post when I have something interesting to say...
The video craze started by the Lazy Sunday digital short on SNL has gotten pretty far-reaching, and some of them are pretty good. Here are my favorites, with a link to the original on the SNL site, in case you missed it somehow.
Lazy Sunday -- http://www.nbc.com/nbc/Video/?c=Saturday_Night_Live/snl_1432_narnia&n=saturday_night_live
Lazy Monday -- http://www.youtube.com/watch?v=myj8woL0Kok
Lazy Muncie -- http://www.youtube.com/watch?v=8FUr4jNS1_k
Lazy Ramadi -- http://www.youtube.com/watch?v=5k3L-_Snu7k
We Drink Tea (profane) -- http://www.youtube.com/watch?v=2l1F6BmKbO0
There are lots more, and I'm sure there are new ones daily, but these are the best of the ones I've checked out...
Saturday, after hemming and hawwing for a few hours, I grabbed my stuff and hopped in the redjeep and drove to Grand Rapids (yeah, Dad, I forgot the chainsaw -- I'm sorry about that). The plan was to try to make a digital recording of the Stickerblister show Saturday night at Planet Rock in Battle Creek. I stopped at the parents' place for a quick dropoff of my bag, and headed out to meet up with Matt to carpool to the bus taking everyone down for the show.
Unfortunately, making digital recordings with a laptop requires that I have only 2 channel input, and it has to be pre-amp input -- generally a headphone output from the soundboard works well. Well, the soundboard at Planet Rock didn't have an accessible headphone jack (at least according to the sound tech), and his pre-amp out was 4 channel. I had no way to mix down the 4 channels to two, so we missed the opportunity to record :(
The show was still pretty fun, Stickerblister (after some technical issues) rocked out pretty well. The sound tech was having problems getting the mix right, so there were times when you couldn't hear vocals, or one of the guitars. Overall it still sounded pretty good, but since I know what they can sound like, it was noticeable to me.
The other bands sounded pretty good. I particularly liked Shackle Jack, who said they're having a big CD release show July 8th at Ten Bells in Grand Rapids. If you like bands like Disturbed, check these guys out. They said they're launching a website at the same time, so maybe search around in July for them.
On a less cool note, one of the girls who rode with us on the bus got really sick after about 2 hours at the club. So sick that they called an ambulance and she had to be taken to the hospital. Turns out that some asshole spiked one of her drinks with some kind of amphetamine. Her significant other and one of the guys in the band are contacting the police, although I am sure there is nothing that can be done. Whoever you are, I can only hope you wise up before someone gets really hurt.
We got all of the equipment torn down and loaded back on the bus and got back to GR around 4 AM. I was just a little tired.
Sunday, Dad and I made omelets for Mom, then I split to get home to try to help Jodi enjoy her Mother's day. I picked up some nice flowers and had the boys sign her card, and we took her out to Mac's for a nice dinner. Soft shell crabs are in season, you know... ;)
I've added the
latest pictures of the kids (Emma, Joshua and Caleb at Soccer, and the boys in their sandbox) to their photo gallery. Enjoy!
Just when you think things might calm down a little...
Today, Jodi called me at work to ask that I come home, as she was sick (flu-like symptoms, ack), and couldn't get the boys dinner, feed Emma and keep herself going. It turns out that Jodi is having some fairly bad symptoms, and Joshua is complaining of stomach pain and has a hacking cough, too. Unfortunately, today wasn't my most enjoyable day at the office either, as I was reprimanded for being 3 minutes late for a morning meeting (cripes) and then got to shuffle several people's already shuffled priorities to cover customer issues and problems with our builds. Nothing Earth-shattering, just a little added stress. Hopefully, I can get the work stuff straightened out tomorrow, and maybe with a little luck Jodi and Joshua won't come down with full-blown sickness (if they do, it is a near guarantee that Caleb and Emma, and probably me, will get sick, too). Wah! I know. I am whining. Get over it -- this is my forum! ;)
Anyway, as of right now, I finally have the boys sleeping -- 9 PM is a record for the last few days. In Grand Rapids, Friday night, Caleb finally had a breakdown with the whole attention from Mommy thing changing with the baby, and didn't get to sleep until almost 11 PM. Saturday night wasn't much better, and last night, they were up until almost 10. This isn't good for the little ones. For most of the last year, we've been good about getting them to bed by about 7 and they usually are settled by 8ish. 11 is a big jump, and if you have kids you know they mostly get up at the same time, regardless of when they go to sleep -- resulting in grumpy, whiny, difficult behavior. It's our job to make sure they don't get to that point and this weekend was a seriously stressful parenting exercise for us. Hopefully, it was just a blip -- the kids are really good, and I hope we didn't set a precedent with a couple of days of trying bedtimes.
I'm off to read a little and decompress. I need to be in early tomorrow for a customer con call, and I can't afford to be late (see first paragraph).
I have been pretty bad about posting anything lately, mostly because I've been pretty busy. My mom stayed with us for a week and helped out with the kids, and this past week Jodi was on her own Monday and Tuesday, then her parents came for Wednesday-Friday. Yesterday we packed everyone up and drove to Grand Rapids for Elizabeth's first communion today. After the ceremony and reception after, Dave and I are going to take my parents golfing at the relatively new Mines golf course, near my parents' house. Should be interesting. I've got 101 strokes over 18 holes so far this year, and 1 30-ball bucket at the range. Not much warmup for the high-pressure match of the year ;).
Anyway, I had better get myself ready, apparently the church is going to be jam packed, so we have to go early to get seats, and with 3 under the age of 4, I don't think SRO is going to cut it. We sent my parents ahead to hold some seats for us, but we don't want them to get mugged for holding them for us...
In this video, the guy rips the Bush administration apart, with Bush in attendance. It is kind of uncomfortable, but also pretty funny.