Setting up a software QA process

I recently dropped in my 2 cents in a discussion about between Ludovic and SpriteCloud about QA 101. I am now working at Longtail Video performing a bunch of roles, including sales, support and QA (Quality Assurance) of our Bits on the Run online video platform. I knew a lot about online video because of my previous job running transcoding operations at Joost, but I was only aware of some common best practices; I didn’t know much about the day-to-day business of running a QA process.

Getting started

The first couple of releases things were mostly a matter of familiarizing myself with the software and trying to click everything on each page and thus hoping to catch all bugs. A lot of time was spent trying to think up as much invalid input as possible and see if that would break stuff. Of course there was already a sane development/staging/production environment set up so software could be tested in a safe environment before being deployed to production.

Initial signs of organization

The easiest way to get started organizing software quality assurance is by compiling checklists. For each page you should define all operations that should succeed. Do still remember that invalid input should be rejected and, (this is a lesson that I learnt the hard way) do realize that it is quite possible that software breaks badly when you try to feed it UTF-8 characters. I keep all of my checklists in Google Spreadsheets, as that way it is very easy to share how the QA process is coming along. Also conditional markup makes all failed test cases nice and red.

This will probably not work for everyone, but during a test run we do not ticket all the bugs that are found. At first we just have a shared Google spreadsheet where we track all the bugs that are found during the QA process. Most bugs (especially typos and smaller logic bugs) are just fixed straight away and tested again once fixed. Only issues that are tagged for a future release or bugs that would require major rework are ticketed.

Also, in a fairly early stage we split up the test process in two parts, as we have two systems that work largely independent from each other.

1. The base software (contentserver, API, databases, encoders)
2. The dashboard (which runs completely on top of the API)

Thus most of our releases are also staggered releases, where first the base system is released and sanity checked, and after that we release and check the dashboard.

Automation

We don’t have a lot of automation, but I have done a little bit of programming to make the tests of the content server go quicker. Essentially I have built a bunch of PHP pages that I walk through before each release. For each release I need a video key of a newly uploaded video, so this means that I have built one config file that needs to be updated before each test run and after that I can click through the pages relatively quickly. Switching between staging and production is a matter of commenting out a couple of lines in the config file. The pages are really simple and look more or less like the image below:

At some point it might be nice to automate the tests of the dashboard too with a Behavior Driven Development (BDD) solution like Cucumber/Selenium-webdriver but for now we have not deemed it worth the development time yet.

Test process

For a typical release I do a couple of steps to test and verify the software:

• I start by gathering all the tickets for this release. I make a spreadsheet that contains all changes/tickets and how they should be tested. This also doubles as the test plan. There are a couple of possible test solutions for each ticket:
  • Add this to one of the existing test plans
  • Nothing fundamental changes, the existing test plans should cover this
  • This is covered by unit tests, just run the normal test plans
• For any new features or radical changes I will spend a lot of time on just using it so I understand how it really works and is supposed to work and will also try to feed it as much invalid data etc to see if that breaks anything.
• When this is done I execute the (updated) test plans.
• When everything looks sane I give a green light for deployment
• After release I do a sanity check with a limited test plan (that can be executed quickly) to verify that nothing essential broke in production.

Making things work for you

In the end I do not think there is one correct way to test software. Especially if you’re working in a smaller team, you will just have to figure out a process that works for you. No matter how much you test, there will always be bugs that slip through, but there are two things that are key to an effective QA process:

1. Catch all major bugs that break the core functionality of your product
2. Whenever you receive a bug report from end-users, verify if this warrants a new test case in your test plan and figure out what you can do to catch similar bugs in the future. (aka learn from your mistakes)

Why I want Joost to succeed

Last week we had a lot of meetings about what we need to do with regard to content ingestion and distribution at Joost, and it has been interesting, very interesting. On Wednesday we had a break out session, and we managed to think up a lot, both things that we will be able to do in the short term, and longer term stuff.

One of the main things where we returned to was who are we doing it for, and this is where we can make the biggest difference. If you look at things like the CBS audience network, big content is rapidly commoditizing. In a way doing Big Content is easy. You put the right content online, have it exclusive, and people will flock to it, because they love it.

What we can do too, and why I personally hope that we will succeed is that we can be a mechanism that enables small, back-of-the-shed-operations, to put their professionally produced content online and make a decent buck from it. I won’t divulge many details about our new product yet, but stay tuned, it’s going to be a great ride!

*Edit*

Okay, this got picked up by the Joostteam blog. Just for clarification: big and small is not an or/or preposition. We still sign on big deals (for example NHK is a huge content owner in Japan that hadn’t done any distribution deals over the internet up to now), and we will continue to do just that, it’s just that if we do it right, we can also help the smaller companies to start making a living of their hard work.

Past midnight thoughts

I was in my bed, couldn’t sleep and was thinking, so what does one do?

What I did?

1. Write a rather thorny email that contains constructive criticism too.
2. Throw together the responsibilities for my new job within Joost. Hope I made sense, get feedback soon.

Thedailywtf posts a non-WTF story

Like many other IT people I’m an avid reader of thedailywtf, a site about all kinds of things that go wrong in and around IT in companies, and where the only right response is: WTF!

Today I read a post that was not WTF at all, and something that I can really buy. Long story short: Law and accounting have been using the Cravath System from the beginning of the 1900′s

“Here’s how the “Cravath System” works. Bring lots of new employees in, team them up with mentors, provide real work to do, and give them a choice: either get lots of great experience and get out, or work hard for a higher-up position. Whenever you hear someone aspire to “make partner”, he’s undoubtedly working at a firm that’s adopted this model.”

When employees leave an employer, because they have reached their value apex, the point where they aren’t bringing anything new anymore, they don’t become ex-employees, but instead they become alumni, who will gladly refer their friends to their previous employer, if they think that that they need the same kind of experience as they have had. I can relate to this story, and for what it’s worth, I’m an OGD alumnus myself, so at least they did it right.

On work (as far as I’m concerned non-WTF), Joost really wants to hire more people, and they have a whole list up somewhere on the website somewhere, but around me some people are hiring too: We really need someone who can do more or less the same work as I am doing currently, which is making sure content gets online. So what do we need? Computer skills like any college student has, good communication skills, accurate style of working, autonomy, and a drive to get things done right, even with imperfect systems.

Also, a colleague dropped by my desk, and asked if I know someone who has basic scripting, basic database (SELECT and UPDATE), and basic linux knowledge (ls counts) who wants to learn a lot about scripting and converting values to information, to do scripting and business information work. I haven’t worked under her, but from what I’ve heard she is a nice boss to work under, and when you’re done you’ll have gained invaluable knowledge about information processing that will change your resume forever. :-)

So, if you are in need of cool work, ping me, and I’ll set you up :-).

Here to stay

Today the two months probational period at Joost have passed, so even though it was pretty clear after a couple of weeks that I would stay, it does feel very good that I’m really there to stay. I’m still very much enjoying my work at Joost, it is very varied, I get more responsibilities than even before, but I think that I can handle them as well as anybody humanly can. I do notice that the extra responsibilities can bite you in your ass though, so, last thursday, when I lost sight of priorities and fucked up a bit, I was suddenly working to past 10 pm to put out fires.

The really cool thing of working in a startup is that there are no best solutions or working procedures. Almost every procedure has been created, and part of the work we are doing right now is inventing solutions and procedures that kill our current jobs. Ain’t life great?

At the moment of writing I’m sitting in the train, being tired, and enjoying the view. The sun is setting and I’m seeing clouds and all kinds of blue and pink. I just viewed a movie Steve (my boss) lent me, it is Danish, cruel and bloody and reminded me of Happy Tree Friends (we have it on the platform too, if you like cruel animation with sweet furry animals, go watch it).

Joost now also has an official blogger, who watches and recommends stuff. While I think it is a great thing that people create buzz and put things in the spotlight, I won’t read her entries anymore, her writing is too superficial, bubblish, and maybe girly for me. I will try to eat some of my dog food, and check out if there is anything good on the platform, I really intend to go look at some Movieola and Raindance features, it could very well be quite fun for a movie buff like me. ;-)

Without spilling any numbers, I am really thrilled about the amount of content that we are beginning to serve, both he technical side, as well as the momentum I see building up, thrill and amaze me, and we are getting more and more content that I really really want to watch (but I can’t watch most of it unless I move to the USA :( ).

It’s alive!!!

It’s been ages since I have blogged, so it’s clearly time to breathe some life in my weblog again, before it dies of asphyxiation.

Let’s see, what is there to write about…

Holiday

I’ve been to Scotland, and I made it there and back again. Unfortunately the road was sometimes a bit dull to walk on (by no means as interesting as the West Highland Way), but sometimes the views were very good. I have made pictures too, will upload them on my flickr page soonish. Unfortunately I haven concluded that my camera doesn’t really suffice. Some of the pictures are quite interesting, but overall the autofocus of my camera plainly sucks.

Work (work work work work work)

I work at Joost for almost two months now, and there is no indication that the end of the trial period (in four days) will be the end of my career there, so that is good. Since the last time I’ve blogged, a lot has happened. When I wrote last time I was spending a lot of time living the dream (i.e. watching tv and getting payed) but now real work has started. and it’s getting more and more interesting.

Let’s start at the day just before I got on holiday, when I spent one day on the “Second Annual Joost Company Meeting.” Unfortunately I already had booked my flight before I even had my first interview, so I missed the later days, but I must say that it was fairly interesting. There were some talks (very confidential of course), but they gave me quite a few new insights of where Joost wanted to go and to what degree systems will have to scale.

After my holiday I had my trial by fire, the two persons that usually did the loin’s share of the content ingestion process were on holiday. So. Besides my regular activities I also became responsible for the complete content ingestion process (and I can tell that by looking a few times at the process before your holiday you don’t get much routine). Fortunately one of the transcoding operators really stepped up to her new responsibilities, and managed a mighty bit of the workflow on her own. Besides that, I also became responsible for sample processing, so my days were pretty much swamped, but I managed more or less to push most of the content in time (barring engineering issues). I have learnt that it’s very important that everybody assigns tasks to the right person; it is very hard to keep track off stuff if you don’t get e-mail notifications that it’s your turn to do something now.

Now the others are back, work is getting a lot less busy again. I don’t know how happy I am with that though, I tend to thrive under pressure.

Houses

The hunt for houses is still going slow. I am putting myself under pressure though, try to find some more motivation…

Oh, and there is fun stuff too :-) I’ve had dinner with colleagues several times now and that is good fun, talks, beer and all ;-). During the holiday I have re-read Lord Foul’s Bane (by Stephen R. Donaldson). When I read it the first time I was clearly too young for that book, I hated it the first time, but by now I have learnt to see and appreciate how good and deep this book is when you compare it too the crap that most fantasy writers seem to produce.

I also picked up a copy of the Nebula Awards Showcase 2006, with short stories. I haven’t finished it yet, but especially the first story, The Green Leopard Plage, by Walter Jon Williams was exceptionally good. It was a story about gen-tech for the good of mankind and what would happen in a world where work is no longer required to feed oneself, which would make labour the base commodity to make the world go round. It also provides some interesting insights in the economy of starvation that is used in too many (mostly African) conflicts.

First week

Claudia was complaining that she hasn’t heard from me for too long, and she has a point, so I’ll write down a bit about what’s going on. First of all I must say that I really like it at Joost. Most of this week I have been doing the basics, doing new things once they come up. This week I spent a lot of time with the quality control operators, but next week I will install the software on the company laptop, and work at my future regular working place.

The coolest thing I heard today is that we get a company dog, a colleague will adopt a six-year-old labrador that sought a new house by way of the in-company mailing list. Second thing I heard is that our effort is validated by Microsoft now. The press claims that they will make a Joost killer, just like they would make an iPod killer I suppose. That means that the press sees Joost as the leading company when it comes to p2p television. I wish them luck (and will sign up for an account to see what the competition is doing ;-)).

Tuesday I looked at a house, and though it looked pretty nice on the internet, I won’t buy it. Not because it is very bad (it might even be quite acceptable), but the living room is just too small. Enough space fot my couches and dining table is all I ask for, so it ain’t that much. I wonder how the current inhabiters do it; they have a big, pretty nice kitchen, but have to eat their wonderful creations sitting on a couch (watching TV I suppose). I must say that I really underestimated the house seeking thing, so far I hate it. I’m also open for good advice, so if any of you knows a house for an affordable price near Leiden (preferably Leiden or The Hague), I’d love to hear from you.

And now it is official

Yesterday I sent in my letter of resignation, today I mailed the signed contract, so now there is no way back. When I signed the contract I knew at last the exact job title I would get, it’s quite a mouth full: content digestion coordinator/librarian. As for what I’ll do exactly: I don’t know, and I am not sure in what degree I am at liberty to tell, but, as far as I can tell, I’ll do a lot of things around the process where content enters the system.

I have become a librarian after all…

I got it!

The long wait is over. I got the counteroffer from Joost today, and it was a litte more than I was about to ask before my father told me to raise that amount, so at least I was realistic. Well, all of this is just the financial bla bla. what matters is that I have said yes to a job I really look forward to doing in a company where I really want to work. I have had a job description of sorts, but time will tell what I’ll do exactly and if I can tell you without having to kill you.

I will be a bit sad to leave OGD though. OGD was a great, fun employer, that picked me up at a moment that I was desperate that I wouldn’t be able to get any fun work, being without a degree and all. Unfortunately the first project I got at OGD was the best as well, since it was a programming project, and I think that programming is just plain fun. After that I still was pretty happy with OGD for most of the time. It is a great club of people that is very fun to hang out with. Unfortunately I didn’t get to work with them, but in IT departments of companies, where I always felt a bit of an outsider.

That, combined with the fact that that I am not that thrilled about the prospect of becoming a windows system administrator for the rest of my life, prompted me to grab this chance rather eagerly. Staying with OGD would probably have been fine, but now I get this chance, I realized, that I was settling for less. When I applied for OGD it just happened, while the last month has been one of tension and nervousness, because I was so excited at the prospect.

So, this is me, settling for more. I hope it’ll work out just as great as it looks.

Patience my young Padawan

Claudia (the former flatmate variant, not the sister variant) had to teach me something about patience. How odd is that?

There are things I’m happy about though. Today The Essential Blender arrived in the mail, time to play with Blender again. And I saw on the OGD intranet that I’ll have to drink beer with OGD next thursday, life is tough ;-)

And the new batteries for my camera have been dispatched, so for once an order will arrive on the day I am home, not that my neighbours mind it very much, but still.