Max Wenzin

When you push to production multiple times a day, the stakes are high — and manual rollbacks slow.  Here’s my experiences from deploying services using Canary Deployments for an important backend service for a large global streaming service. Why Canary Deployments? We adopted Canary Deployment for a couple of reasons. We had one or two

Don’t. Don’t do Pull Requests. But if you have to, continue reading. Have you ever been ambushed by a Pull Request? It’s happened a lot to me. There’s a Pull Request on some software that you and your team are responsible for, and before you start reading all the changes, you’ve no idea what it

Max – Tell me briefly what your team is up to @ Discovery?

I’m one member of a devteam which takes care of external API’s. We build services for streaming TV (OTT), for example user management, authentication, user entitlements and playback services.

We started this effort to enable a more flexible way to assign entitlements to users, for example through affiliates (like cable TV companies and mobile phone operators). Before, you had to be a subscriber at Discovery to be able to view any content. Being a direct subscriber is called the DTC model, ”Direct To Consumer”. But a large part of the business in some markets still comes from the so called “TV Everywhere” model where the customers come in via affiliates.

In this post, I will use an example to show how you can:

• Write a small microservice with a REST API in Java, using the Dropwizard framework
• Containerize the application using Google JIB
• Deploy the application to AWS using Ansible
  • With a robust, clustered and self-healing setup

Continue reading

This article is the third in a series about system testing:

1. Dockerized testing vs end-to-end testing
2. How to setup Dockerized testing
3. Readable Java system tests with good old JUnit

JUnit is poorly named. Given the name, people tend to think that it should only be used to write Java unit tests. And then people feel a bit hesitant about writing their integration tests with JUnit too. When they start with system tests, they often think they need another driver for their tests. Sure, maybe using another abstraction layer and a custom domain specific language (BDD), you can make the tests more readable for a non-programmer. That often comes at the cost of making the tests less readable for the programmers. And if we are honest, who’s going to read the tests the most? Perhaps just naming test classes and methods well and writing readable code can suffice?

Continue reading

This article is the second in a series about system testing:

1. Dockerized testing vs end-to-end testing
2. How to setup Dockerized testing
3. Readable Java system tests with good old JUnit

You may want to read Dockerized testing vs end-to-end testing first.

How to setup Dockerized testing?

A prerequisite for running your tests against a dockerized environment is of course that the service you want to test is packaged as a Docker image. Also, all of the services which the System Under Test (SUT) depends on needs to be dockerized. If you don’t have that, your environment is not fully dockerized and you will suffer the consequences of having to test against external services.

If your SUT is packaged as a Docker image and has no dependencies on other services, you can just start it using “docker run” and run your tests against it. However, usually services have one or more dependencies to other services, such as its own database, a queue and perhaps a external RESTish service that returns nasty XML. You want each of these services in their own container. To spin up a bunch of docker containers, you can use docker-compose.

Continue reading

I recently rewrote the end-to-end tests for a service. This brought up the question of what we should test in the end-to-end (e2e) tests vs our Dockerized tests.

This article is the first in a series about system testing:

1. Dockerized testing vs end-to-end testing
2. How to setup Dockerized testing
3. Readable Java system tests with good old JUnit

e2e tests vs dockerized tests

Since the original e2e tests were written, we (my team) have also introduced another type of tests. We call them “dockerized system tests”. These are similar to the e2e tests. They also test the connections between services in the system, and how the system works as a whole. One key difference between e2e tests and dockerized tests is that the dockerized tests do not test the system when it is deployed the way we deploy it in staging and production environments. Our dockerized tests bring up all services in a single machine, and we usually don’t set up redundancy, clustering, load balancers and such things.

Continue reading

How should you structure your Ansible variables in playbooks calling cloud modules? Ansible has extensive support for variables at different levels (there are 21 different levels!).

The inventory group_vars and host_vars offer enough flexibility for many use cases. However, when you are setting up cloud infrastructure, you don’t really have any hosts yet. Maybe you are setting up load balancers, instances, security groups and such things. Perhaps you want to setup the same things in several different environments (like “staging” and “production”). You want to re-use the same playbook, but adapt what it does by using different variable values in different environments.

In this post, I will demonstrate a way to structure Ansible variables in a playbook which pretends to setup a AWS AutoScalingGroup. I will show how you can separate the configuration for different environments by using a inventory directory. As a bonus, I will give you a neat trick to automatically load a extra variable file for each environment. We will use this to load a separate secrets file which is encrypted with git-crypt.

Continue reading

When you use Ansible to provision hosts, running a task on a set of hosts in parallel is not a problem. You simply use the `serial` keyword in a playbook. See: http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size What about when you want to run a task several times in parallel, either on each host, or on a single host? A

You have probably read about “at scale” implementations, activity based offices, globally distributed teams, SAFe, Agile transformations and outsourcing. Beware. Danger can be lurking beneath the surface of these popular phenomena.

Continue reading

When the BBC published their “No CEO” piece where Crisp is featured with an article and a 4 minute video, there were a lot of reactions. Friends cheered on Facebook. Colleagues gave a thumbs up on LinkedIn. The article was featured on Hacker News and Slashdot. Here are our reflections on some of the comments we found.

Continue reading

 In my team, we are working on improving real-time performance for our main service. The goal is to have response times below 100 ms in the 95th percentile and below 200 ms in the 99th percentile for certain database volumes and request frequencies.

We don’t know what will be needed to reach this goal. We have some ideas, but we don’t know which one, or which ones will do the trick. We call these ideas “experiments”.

We can estimate each experiment, but we don’t know how many we will need to do to reach the goal.

This is the story of how we apply the scientific method to working with performance improvements.

Continue reading

Since I have found the Ansible documentation to be lacking, and StackOverflow insufficient in this matter, I feel the need to share how you can append to a list using Ansible.

I’ve created a demonstration playbook and published it on GitHub. See: https://github.com/betrcode/ansible-append-list

Continue reading

The www.crisp.se website is based on WordPress, with a custom Crisp theme.

This is the story about how we’ve developed our custom WordPress theme, how we’ve made it easy for any Crisper and external consultants to work on the theme, how we’ve setup version control, continuous delivery, staging and production environments on Amazon Web Services (AWS). And how all of this is setup with absolutely no automated tests whatsoever.

Continue reading

Should the Scrum Master role be full time or part time? What if there is not enough Scrum Master work to do? Can the Scrum Master also do other work in the team? Can the Scrum Master be Scrum Master for several teams?

There was a debate about this and Scrum Alliance created the Scrum Master Manifesto in 2011.

Even though this has been debated by many minds before, I still get asked what my views are on this topic.

I’ve done all kinds of variations on this role. I’ve been a dedicated Scrum Master for a single team. I have done the SM role and a developer role at the same time. I’ve been a Scrum Master for several teams at the same time. These alternatives have their own advantages and challenges. In this post I intend to describe my view and recommendations.

Continue reading

We get a lot of questions about how Crisp works and why, especially from other consultants looking to create something similar. After many years of experimenting we’ve converged on a model that works well, basically the sweet spot between being an independent consultant and being an employee. So we decided to open source it. In January 2015, at

The Gist

• ToDo has a flow. It knows about cycle times and about being DONE. Trello does not.
• ToDo has Planning Poker Estimates. Trello does not have any estimates.
• ToDo has automatic burn up charts. Trello does not.
• ToDo has swim lanes which groups cards by your dimensions. Trello does not.
• ToDo has Work-In-Progress limits. Trello does not.
• ToDo has upgrade possibilities to the full tool set of Projectplace. Trello has a bunch of plugins from different vendors of various quality.

Already convinced? Sign up for ToDo by Projectplace! Want to know more? Read on.

Continue reading

Building a well-functioning software delivery team is complicated. There are many factors to consider. Current team (if any), needed skills, available people, company politics etc.

There are some fundamentals that often (but not always) seem to work.

My fundamental principles for teams

• Static
• Cross-functional
• 5-9 people
• Co-located
• Dedicated team members (belong to only one team)

I find these principles to be a useful basis for discussion, when helping managers configure their teams.

The principles are goals, and one must realize that all cannot be achieved all of the time, nor instantly.

Continue reading

As a visitor to the Crisp Blog, you may have noticed that we’ve launched a new web site on www.crisp.se?

Apart from a new design, we’ve also changed CMS from imCMS by imCode to open source WordPress.

Continue reading

I am currently working as a Scrum Master for multiple teams at Projectplace in Stockholm, Sweden. One of those teams is the Mobile Team. They are developing Action Boards for both iOS (iPad) and Android platforms. These Action Boards are also available in the Customer Preview of the Projectplace web service. Both Web Team and Mobile Team share the same API’s. The iPad app is planned to be released in 2-3 Sprints from now.
This case study can be written from many perspectives, but in this article I am going to focus on how we are working with the challenges of having a distributed Scrum team.

Continue reading

I have been reading Gamestorming lately and found that I’ve practiced many of the games and followed most of the principles of game design already. However, there are a lot of new things in that book that I’d like to try. Today I tried drawing a Visual Agenda for the first time. I think it was well