By now, you’re probably getting an outstanding idea at what goes into the improvement of a groundbreaking smartphone. In part one we followed the construction of the Motorola Atrix 4G , discussing the method of the way AT&T decided upon a design and collaborated with Moto to get the ball rolling. Now, we conclude the series by covering the extreme testing process, getting the device prepared for launch and updating the phones after they’re released.
Testing
Every facet of the phone’s development is critical, but ensuring the device actually does what it’s imagined to do is understandably mandatory. AT&T praises itself for having one of the most highest standards in testing, though it’s impossible for us to confirm without having the ability to compare notes with the opposite US carriers (Sprint, Verizon and T-Mobile, we await your valued input anytime you ought to divulge this knowledge).
The lab cycle
In spite of everything of the collaboration from side to side between AT&T’s and Motorola’s product teams, doesn’t it feel just like the Evora must be a elegant product by now? Nope, not even close. It was September 2010 when the telephone reached the testing labs. Making it to this stage was no small feat in and of itself, however the device still had an extended road sooner than it. Give it some thought this fashion: AT&T desired to have the product able to ship in time for the vacations, and Black Friday was a trifling two months away. The LapDock hadn’t even been sent to testing yet at this point, which works to indicate how much was left at the team’s plate before the telephone could be able to hit shelves. We all know what it’s desire to have an impossible deadline, for you to relate.
The carrier wants its test devices to be completely defined, with each of the specs as almost final as possible — and it requires that the OEM gets its handsets validated by a 3rd party to make certain all of AT&T’s quality metrics are met.
After reaching the labs, the Evora needed to endure thousands upon thousands of test cases. AT&T has the facility to imitate and simulate virtually any sort of network condition or environment that it could possibly recall to mind, and runs the device through they all to peer if it might reproduce any issues. A standard test lab contains 1,200 servers and about 100 cell sites, which enables them to simulate specific cities, frequencies and technologies. Have to see how the Evora operates on 1900MHz HSPA+ in Portland, OR? No problem. What about 850MHz UMTS service in Baltimore? Easy as pie. But why would the signal be any different in these cities? Because each individual market typically uses a selected infrastructure provider: AT&T contracts companies equivalent to Ericsson and Alcatel-Lucent to construct out the carrier’s network, but every one is selected to take action for a given city or market. The range of providers adds to the complexity of the network and its operation, and it’s as much as the testing lab to mimick every possible scenario a customer may face — irrespective of how unlikely — to ensure the telephone isn’t going to behave wonky.
The test labs contain 1,200 servers and about 100 cell sites, which enables them to simulate specific cities, frequencies and technologies.
Every simulation’s run in a copper cage — no bigger than your run-of-the-mill walk-in closet — that’s able to blocking out all outside signals to be able to not interfere with the tests (or leave them out within the open to be discovered by curious folks, for that matter). These labs also run tests on firmware updates, radiation, audio quality and any accessory that gets sold in AT&T retail stores. They’ve installed a bug tracking system that both companies’ teams have access to; bugs are assigned a severity level as a style of prioritizing phone issues that must be resolved, and the method to shut out each individual bug is quite detailed and closely tracked.
Severity level one is anything that negates service, and absolutely none are acceptable in a last product. These kinds of issues aren’t simply bearing on such things as dropped calls or network frustrations, though that certainly comes into play within the labs. Anything that forestalls service or possible use of the telephone is slotted into the highest echelon of priority. To present us an example of something that’s considered service-negating, Dante showed us probably the most Evora units used during testing and had us go into the net browser. Once there, he asked us to aim pressing the house key — and nothing happened. The shortcoming to correctly use the entire buttons needs to be a top-priority fix, naturally.
Severity level two issues are those who are seriously frustrating to the user, and affect their delight in the device: copy and paste doesn’t work properly, the telephone won’t mean you can switch calls or even the speakerphone doesn’t work quite in addition to it is going to. Too many infractions in this level will pause the phone’s progress and it can’t continue until they’re fixed, though one or two penalties could be pushed through if the OEM commits to fixing it once possible. Good to listen to that some of these issues are treated before reaching final approval; could you imagine owning a phone that wouldn’t assist you to switch or merge calls?
Finally, a degree three issue is really an improvement ticket — a small bug that AT&T would favor fixed in a post-launch maintenance release, and is something to be addressed in future products to make certain it doesn’t occur again.
Usability testing
The usability tests for the Evora began in October, this means that they were run together the device was still within the lab. Each product goes through a slough of usability tests to aid the 2 companies learn more about how people interact with it. The teams watch multiple people as they use the device to peer in the event that they can use it easily and comfortably. Can testers navigate through a definite menu structure quickly enough? Are they tripping up somewhere at the phone? If anything is located that adversely affects the user experience, the teams attempt to incorporate new things into the telephone to make certain it doesn’t keep happening. “If we glance at incorporating best practices and improving user experience and take out extra steps, the client becomes satisfied and becomes recommenders,” Chris told us. As mentioned earlier, we’ve a tough time believing that testers were glad with MotoBlur, but a minimum of the UI has improved because the Atrix was launched.
There are some areas of the user experience wherein just a few people appear to trip up on. When these issues arise, the team begins to take a look at other avenues to maintain the priority. As an example, is it something which may be addressed by adding a small blurb into the Quickstart guide that incorporates the telephone? If not, that exact case — and others love it — is worked through in a single of the Evora’s post-launch maintenance releases and incorporated into any future Motorola phone with similar features.
Lapdock and Webtop
Motorola’s unique Webtop environment and accompanying laptop dock were born alongside Evora and given the codename Virgil to distinguish its development. It was going to be a brand spanking new product, an idea that no-one had really tackled head-on before. And never just that, it was the right fit for AT&T and Motorola’s vision of a groundbreaking device that might completely change the industry. Since it was such an unknown, both companies were going to be taking a big risk by bringing it to market, for the reason that costs concerned with developing the laptop dock and the entire Webtop experience were astronomical. How wouldn’t it be used? What direction should we take the vision? What value will this be to customers? Are you able to get this thing to do what you would like it to with no need a PhD?
AT&T invited CIOs from ten different companies to fulfill together under an NDA to produce feedback at the LapDock.
Because the LapDock was primarily aimed at the business professional, AT&T needed feedback from legit sources to get it right. On September 15th, the carrier invited CIOs from ten different companies to satisfy together under a Non-disclosure Agreement (NDA). These executives had no idea what they were going to work out; it could’ve been a knowledge card, for all they knew. The disclosing consisted of some carefully crafted demos, because the Lapdock was so early in its development that not everything was working properly. However the mission was a hit: the belief of incorporating Citrix functionality was cited on the meeting, let alone a couple of other additions that were either included within the first-generation models or pushed to later versions (details on those other additions weren’t given). Citrix made it into the ultimate first-gen product, but barely made the cut.
The LapDock made its way into the labs near the tip of September, not even two weeks after the CIO meeting. Again, the device met AT&T’s rigid stability requirements, however it still had a lot of bugs to determine. The lab would find issues and send the hassle tickets over to Motorola, which might then be addressed by spinning new software updates to the labs and repeating the cycle again and again.
Virgil was ready for usability testing in October, nonetheless it was still the sort of secretive process up to now that not one of the internal teams even knew about it — it had only been on an exclusive “want to know” basis thus far. So, NDAs were issued to the usability testers, that’s a gorgeous burdensome deal. Why? Not only anyone will also be given access to something with the sort of high level of confidentiality; Dante needed to plow through a vetting process to make your mind up who can be offered the NDA, and the lucky contestants were required to read and acknowledge it before from now on progress might be made at the project.
The testers were faced with a slightly tough challenge when Virgil showed up. Here was a wholly new product that they’d never seen before, which meant that an entire set of test cases needed to be thought up. With this type of tight deadline to make this work, Dante admits that he wasn’t the hottest guy on this planet for some time. “But,” he said, “we needed to do what we needed to do.”
Prepping for launch
By the point October and November rolled around, the teams knew Evora and Virgil weren’t going to be ready in time for the holiday season, but at least a CES deadline was somewhat feasible. Launching a game-changer at the largest consumer electronics show in the world would give the carrier a boatload of publicity, not to mention the momentum AT&T needed going into the device’s release (after all, we did deem it to be the best smartphone of the show). It still wasn’t going to be easy though: it was red alert from this time forward, and nobody got much sleep the rest of the way — we can envision several of them asleep at their desks, empty coffee mugs in hand. There were some huge last-minute changes that needed to be made before making the final launch preparations.
Hardware changes and delays
There’s a huge reason AT&T doesn’t like to make last-minute hardware changes on any of its devices unless it’s an absolute must: it involves a hard tradeoff. Even the slightest adjustment usually translates into six to eight weeks, since the OEM has to first manufacture the product and push it through the full test cycle another time. Any hardware or firmware change requires the complete litany of tests all over again, because even the smallest of adjustments can break something else on the device that’d previously worked perfectly fine. Ah, the fun life a software tester leads.
Delays like this become even more costly when you consider how compressed the lifespan of a phone has become. The Atrix is a great example of this, since its sequel launched not even nine months later. Taking six weeks out of a product’s nine-month lifecycle to fix issues or make changes means there’s that much less time to sell the device before the market changes and the phone becomes obsolete. Thus, lost revenue opportunity — not hilarity — ensues. With such a groundbreaking device on the line, further delays were simply unacceptable.
Two key factors can cause delays. First, there’s a serious quality issue that prevents the phone from reaching Technical Acceptance (final software certification) and the problem isn’t being solved by new builds. Second, the market changes and AT&T sees a need to incorporate a different feature into the phone to make it as successful as possible. “Sometimes we decide [together with our OEM partner] that it just has to be done in order to push the product forward,” Dante said.
Of course, tiny wrinkles happen all the time in device launches, and teams are used to dealing with these types of unforeseen circumstances that these obstacles hardly ever affect the timing of the handset’s release. Just because the screen protectors designed to fit the Atrix 4G don’t function properly, doesn’t mean the train stops moving. So many things move along in parallel, and if one item gets completed en route to launch, more resources are reallocated to strengthen the other parts of the phone’s development.
But the Evora project had a couple hardware-related concerns that were more significant than mere wrinkles, neither of which we’re able to speak on. However, Motorola and AT&T made excellent use of the opportunity to update other components in the phone as well; in such a dynamic market, prices can come down swiftly and market trends can move incredibly fast. One of the big changes AT&T wanted to make was the inclusion of HSPA+ — the Evora was originally designed without the next-gen tech in the works.
Leaving an HSPA+ radio out of the Evora’s original design seemed like a monstrous oversight.
Looking back to the first quarter of last year, Verizon was busy 0 making preparations 0 to launch its LTE network across the same time the Atrix was supposed to come out, and T-Mobile was already 1 rolling out 1 21Mbps HSPA+ service. Leaving an HSPA+ radio out of the Evora’s design seems like a monstrous oversight, right? We know that AT&T had originally planned to simply 2 leapfrog 2 the tech on its way to deploying LTE, but just a week before the Evora was greenlit, Ralph de la Vega declared that his company intended to build out 3.5G sometime in 2011. There was more than sufficient time to switch tracks at this stage in the development process… or so it seemed.
We have a theory on why HSPA+ was left out until the last minute. The phone was originally expected to launch during the 2010 holiday season, ahead of the time that the next-gen tech would blanket the great lands of this nation. However, once AT&T knew the Atrix would be delayed into 2011 and the timeframe for HSPA+ deployment became much more clear, it likely made more sense to add the radio. After all, Motorola was adding a few things to the spec list already, so why not make sure the upcoming flagship device had all of its other components completely up-to-date as well? It’s a good thing, too: it’s hard to imagine the Atrix, a smartphone marketed as the carrier’s top-notch flagship, being released without no less than 3 14.4Mbps 3 connectivity, let alone anything faster — not this year, certainly.
To the relief of both companies, Motorola was able to refresh the hardware rather fast, delivering the new build to the labs in the middle of November. And the testing cycle started from scratch. It was becoming clear thon the device would, worst-case, be ready to announce at CES.
Naming the device
What’s in a name? In this industry, everything. When titles like Inspire, Revolution and Transfix reign supreme, it’s just way too easy to mock 4 phone names 4 on a regular basis, and we doubt we’re going to stop anytime soon. We get it, though — the value companies take in finding a marketable name that’s easy to remember is obviously priceless. Evora is just a codename, of course, so how did it turn into Atrix and why?
In general, the carrier and OEM both have a hand in a phone’s name, but AT&T prefers to be in the driver’s seat. The team generates a list of possible names to take to their legal department, which performs a trademark search and whittles it down to roughly one-tenth of the size — and usually it’s the worst from the list (all the best ones, we presume, are already taken). If they can’t decide on a name, the process repeats until they stumble upon something good.
The team generates a list of possible names to their legal department, which performs a trademark search and whittles the selection down.
The Evora was ready to obtain an official name in November, and it had a couple top contenders that had passed legal muster: Glory and Catapult. Dante’s team felt that Glory just didn’t mesh well with the features they were trying to perpetuate, and Catapult just sounded too… medieval. But everyone loved “Atrix.” Since it’s a bleeding-edge phone for the tech enthusiast, it should have a futuristic name. Not only that — starting with the letter A is always good, there aren’t too many syllables and the X at the end adds an element of coolness. Making up words can definitely be cool, but it’s a gamble. Dante educated us on the danger of diving into the realm of made-up names too often:
One of the nice things about a made-up word is that your message is the assigned value to it… the problem is, if you do it too often, you start sounding like you’re coming up with baby talk or gobbledy-gook and it no longer makes sense. There’s a fine line there.
Pricing
We weren’t able to glean a lot of information from Dante or Chris on how the pricing of phones is determined. The process is incredibly complex and depends on a number of factors, such as component pricing (which, due to the dynamic industry, is always changing), the carrier’s target segment and, as you’d expect, some influence from the OEM. Sometimes pricing can even be affected by either partner’s willingness to promote the device (more on that later).
The carrier’s decision to begin selling the Atrix 4G at a $200 price point wasn’t anything that warranted shock value. The cost of Motorola’s LapDock accessory, however, was. We loved the idea of a laptop dock with Webtop built-in, but it certainly wasn’t worth $500. AT&T was stuck between a rock and a hard place here: the carrier wasn’t oblivious to the fact that the device was highly priced, but a much different business model applies to unsubsidized accessories. Says Dante:
“When you’re working with an OEM who’s used to carriers using one model and that’s how they price, and then they price to us the same way to something we can’t apply that model to, it doesn’t work out very well.”
In other words, without attaching a contract to accessories, AT&T was constrained to keep the LapDock at standard retail pricing, though it was willing to take a hit on profit by bundling the phone and device together in one purchase.
Motorola spared no expense in making the LapDock, which not only included a stellar battery for the form factor, a durable magnesium enclosure and individual metallic keys, but also incurred significant R&D costs associated with a first-gen device and a brand new ecosystem. We’re witnessing some validation of that, as the next generation of the LapDock — namely, the 5 100 5 and 6 500 6 — is coming to market at a significantly lower cost.
Controlling leaks
Leaks. They’re a fact of life for the tech media, the proliferation of which we participate in daily. Rumors of the latest and greatest (heck, even the blasé) handsets roam free across the internet like a pack of wild buffalo, whether they’re true or not. As we learned in our meeting with Chris and Dante, leaks like these frustrate product managers to no end. It stings, but why? According to Chris:
We have an investment. If there’s a leak, it can have a detrimental impact on the perception, or when there’s a misquote and an assumption is made, then Dante has to work twice or three times as hard to try and convince people that’s not what the product was, it never was that, but the decision has already been made, the court of opinion has already been flawed.”
Dante went on to explain that each phone AT&T launches has a unique story to tell, since they’re meant to reach different segments of the market and hit various groups of people based on what they care about. It’s all about projecting the company’s vision to the customer. Leaked phones tell no tales, because media organizations report on what components they feature and what they look like, rather than the “story” the carrier’s hoping to portray. This especially speaks volumes to any device that isn’t considered state of the art and may get negative points for not having the best specs on the market. “They can never set the right expectation,” he said.
“We’ve never personally been involved with a purposeful leak. It’s not in our best interest. One thing can go right while 99 things can go wrong.”
Since product managers like Chris and Dante are forced to go into damage control mode as a result of harmful leaks, we asked if they ever seed a counter-leak on purpose to resolve the problem. They told us:
We’ve never personally been involved with a purposeful leak… it’s not in our best interest. It’s like one thing can go right while 99 things go wrong. We don’t know how that makes sense.
The managers went on to say that no matter how something gets leaked, the information goes through the telephone game: no matter what you say, it’ll be different once it gets to the other end, and the perception of the truth is altered. Granted, we didn’t see a horrible alteration of the Atrix 4G when it was leaked as 7 the Olympus 7 , but that isn’t the case with every phone. We’ll give kudos to Dante and his team, however, for keeping the LapDock virtually leak-free.
Promoting and marketing the device
This part of the process is when the claims come out to play: ” 8 the world’s most powerful phone 8 ,” “the world’s fastest phone” and other similar marketing terminology is pushed through AT&T’s legal team to make sure the company remains free from the threat of any possible lawsuits.
It’s also the time when the product managers determine if the Atrix 4G will be a hero device. Ultimately, a phone is a hero candidate when it either moves the ball forward from a feature standpoint or offers some type of value proposition — the Impulse 4G is an excellent example of a hero that gets a lot of TV time for this very reason. The carrier will also negotiate with vendors to promote devices, which in turn can lower the price.
0 “Even though we do a lot of phones, it’s not really a cookie cutter process,” Dante explains. “New things become important… in general, we have the baseline for each launch. The idea is always to build on that. With each launch we’re working with OEMs on new ways to promote it.” Samsung and AT&T teamed up to sponsor the most recent Keith Urban US tour, which involves the superstar shooting video with an Infuse 4G. The opportunities to market phones are incredibly diverse, which forces the carrier to get creative.
First Article Inspection
Just as the Atrix spent time getting tested in the labs, pre-production units are seeded to a decent number of field testers to try out in real-life situations. But as these aren’t the final units coming off the actual production lines, AT&T wanted to make sure that devices destined to wind up in customers’ hands are as good — if not better — than what’s been tested already. This is where Final Article inspection (FAI) comes in.
To obtain the coveted FAI status, Motorola sent AT&T a few hundred devices from the final Atrix production line to go through the entire suite of tests one last time. Sounds so sentimental, doesn’t it? Once the phone passed, the vendor got the green light to begin flooding all of the available distribution channels — a process that took around two weeks.
Know why AT&T will, more often than not, announce that a device is heading to stores “in 9 the coming weeks 9 ?” FAI may occur two weeks before the phone actually gets shipped out; if something goes wrong and it doesn’t receive that approval, everything the PR reps promised is no longer true and everyone has egg on their face.
1 The carrier may elect to announce a phone at CES or CTIA but prohibit the device from being turned on, handled or photographed — much to our chagrin, of course. This happens because those handsets haven’t yet reached FAI and officially aren’t finalized. AT&T gets nervous when it comes to negative first impressions and hands-ons because the software may be pre-production quality. We’ll offer an example: the Samsung Infuse 4G (shown above) was announced alongside the Atrix 4G at CES 2011, but media 0 wasn’t allowed 0 to touch it or take pictures of its back. The phone wouldn’t officially launch until May — roughly four months later — and was nowhere near achieving FAI. It was definitely not the carrier’s style to announce new phones so early, but AT&T’s reason for doing so was to further emphasize that it was making some huge leaps to embrace Android. Still, the event sent out mixed signals and the message wasn’t clearly received, which ultimately caused much more damage than AT&T had intended.
Preparing all channels
2 Now that it has the official name and made its way through the proper legal channels, the Atrix is ready to get a final review on the packaging and materials. The box, accessories, Quickstart guide, and so on are all given the thumbs-up here. Which accessories are included in the box is typically related to keeping the device cost as reasonable as possible, while accentuating certain customer experiences at the same time — such as HDMI cables or stereo headphones, in some cases.
The last few weeks before a device’s launch are the most hectic. It typically involves achieving FAI for the final go-ahead, getting all of the marketing materials ready to go, training employees, communicating the proper information to PR reps so they can talk about the device intelligently to the press, distributing review units and shipping out the Atrix to retail, third party and eCom channels. So many elements work together simultaneously to make sure everything is aligned correctly and the launch will be a success.
Launch day and beyond
Team Dante’s hard work didn’t stop the moment the Atrix 4G was released; far from it, in fact. The device was solid and in great working condition, but there were wrinkles that still needed to be ironed out after its official outing. So now what?
Rapid response
The first day of sales, February 22nd, was all hands on deck and a war room was set up for rapid response (we can’t help but picture a standard red rotary phone in the middle of a conference room), so any snafus that show up could be worked out: if the SKU didn’t ring up in the register at the corporate stores, for instance, it needed to be resolved immediately.
From that point on — day of launch and beyond — Dante was constantly on the lookout for any feedback he could get. Media reviews, social networking (Twitter, for instance) and word of mouth are very important to determining what went wrong and how to correct it in this device and in future models. This is where maintenance releases and bug fix updates become crucial.
Firmware updates
3 People begin using the phone in ways the product managers never imagined. Feedback on bugs and other issues gets back to the team. The next version of the OS comes out. All of these scenarios occur with every device, which means Dante and Chris need to be on the ball in cranking out updates and 1 maintenance releases 1 . As often is the case, these refreshes take a while to push through. Five moons had passed before the 2 phone was ready 2 to get Gingerbread. The main reason for this is that placing new firmware on existing hardware involves much more than just the click of a button. Chris told us:
Just like in the labs [the first time around], when new software is introduced we have to go through the full gauntlet of tests from this end to that end all over again. New software can always introduce new bugs in areas that were fine before, and the last thing you want to do is take a person who’s happy with their phone, and they update it and it doesn’t work as well anymore. People think with upgrades that they’re getting more. You don’t want to break anything. That’s why we go through the same rigorous testing in these updates, we don’t want to have a negative impact on a customer experience.
“People think that they’re getting more [with upgrades]. You don’t want to break anything. That’s why we go through the same rigorous testing in these updates.”
There’s a fine line between ensuring a good upgrade experience without actually making it worse for the user. But because of this, updates have a tendency to take longer to roll out. Android devices, for instance, appear to be the most difficult. Google announces Gingerbread, OEMs finally get their hands on the source, they push it through their own development cycle to build their own UI onto it (such as MotoBlur, in the case of the Atrix), and then it has to undergo QA (Quality Assurance) tests to ensure the firmware update is good enough to get into AT&T’s labs — and that’s when carrier testing can finally begin.
Sometimes, as the managers claimed was the case with the Samsung Captivate, new versions of the OS can take longer to push out on older hardware. According to Chris:
Dealing with devices that are mature in their lifecycle, getting software with new features that are built for newer hardware, means even more grueling regression tests to make sure it doesn’t break the phone or any other aspect of the product. Anything that doesn’t meet the criteria has to go through the process all over again.
Where’s the 4G uploading?
4 Prior to the Atrix’s release, Dante’s team already had items on the list for a post-launch maintenance fix. One of these, if you may recall, was the enabling of 3 HSUPA 3 speeds. Here was a device advertised as having 4G capabilities, but only 4 HSDPA 4 — the download portion of the next-gen network — actually came enabled on the Atrix when it launched (as well as the HTC Inspire 4G, launched in the same timeframe).
We were left utterly confused as to why this was the case — was it the biggest oversight on earth, or was AT&T leaving it out on purpose? No matter the answer, the carrier either looks stupid or mean. We asked Dante, who insists it was the latter, but not because the carrier wanted to be a jerk. Instead, it was a result of AT&T’s last-minute addition of HSPA+ back in November; additional time was apparently needed to push it out.
We thought, what can we make happen in time? That’s how we ended up where we did. We got our MR out as fast as we possibly could to address it, but my rationale was — as long as you can download your videos at a good clip, you don’t sit there and wait while you’re uploading something… we wanted it in there. But we had to get the product out there; we didn’t want to delay any further.
It’s understandable that software changes can take a fair amount of time to implement correctly. But if the uploading capability had been done in concert with HSDPA, how much longer would it have been delayed?
Wrap-up
5 Knowing the journey helps us come to a greater appreciation of the destination. It’s not an easy job to come out with a smartphone lineup that pleases everyone, and we certainly got a sense of the pressure Dante’s team felt in getting a landmark phone ready and in shipshape condition. The Motorola Atrix 4G wasn’t a perfect handset, of course, but it reached its intended goal: it was supposed to be a game-changer, and it’s hard to deny that it was. And don’t get us wrong, we still — and always will — have our concerns about the extraordinary amount of influence carriers hold over phone manufacturers throughout this process, but we willnot refute the truth that Dante and Chris gave us a brand new perspective on what actually goes into making a masterpiece.
The winners of the 2011 Engadget Awards — Readers’ Choice
NPD: Apple grabs over 1 / 4 of the mobile PC business in Q4 2011 (including iPads), HP tops with laptops
