How We Helped Predictable Revenue Scale

A few weeks ago I had the opportunity to talk to Preston St. Pierre, CTO of Predictable Revenue. We worked with Preston and his team for 2 years to help them scale their software development.

Ernesto: First of all, thanks for taking the time to talk to me. I have a few questions about our past engagement. Let's start with this one: What was the main problem you were having when you hired us?

Preston: I suppose the main problem was lack of developer hours. We needed somebody that was flexible, on time, that we could hire for more hours or less hours if we had the budget.

But the part that we didn't realize that we needed was that there was no real structure to our development, being just Francesco and I. We'd discuss the problem, decide what to do, and that was kind of it. You gave us a number of processes and principles that we used to start our growth towards a more traditional organization for engineering. It didn't affect us much at the time, but it sure helped us later on as we added more developers.

Ernesto: That's great to hear. And how did that impact the business for Predictable Revenue?

Preston: Ombu Labs was a significant part of our development team for over two years. You guys were a core part of our team, and therefore you helped shape our product. I couldn't put a specific number value on the impact or anything, but I definitely would say that your expertise brought a lot to the table and that it positively influenced our system architecture.

For example, using Amazon's SQS to communicate with our sender nodes via message queues; we might have went that way otherwise, but I think that was your idea. I didn't have that in mind. So it's possible we could be using that now only because of you.

Ernesto: That was a very interesting project. Thanks for the kind words! I appreciate that. I actually have a question about that but I'll leave it for later. You said a little bit about the value we added to the team. Anything in particular that you valued from working with us?

Preston: We work with a number of contractors, but the thing that set the really good ones apart was how they, instead of just doing the job they were told to do, they would actually figure out what we needed and try to find ways to solve our problems as though they were members of the team. And that's exactly what you guys did. Whereas the bad contractors, they would accept the terms that we gave them, and they would give us what we asked for. It was kind of like we were holding their hand the whole time, and it was not really worth it to hire them at that point.

But you guys were much different. You took the initiative to figure out where there were problems, where updates/maintenance were needed, then went out and did the work. That was very useful.

Ernesto: I think one thing that I valued from our engagement is that there was value in disagreeing with you every now and then. And not just disagreeing for the sake of disagreeing, but proposing other ideas or other ways to implement a solution, then having a discussion about it, and then presenting the arguments and eventually going with one way or the other.

Preston: Definitely. And while I can't guarantee that everybody's going to appreciate that like I do, I know I'm not always right. Sometimes I needed to be challenged.

Ernesto: So let's talk a little about how things were after we ended our engagement, how did we impact your day-to-day engineering operations?

Preston: I think it circles back to what I mentioned before, which is that you guys largely helped put in place at least a framework of processes that we were able to build upon later. Kind of sent us in the direction of using some sort of organizational structure to our development as opposed to simply discussing what needed to be done and doing it, which was the previous way we did it. I hesitate to repeat my previous answer, but it's the only one I can think of.

I should mention, your push for pair programming has been very influential on our team. We do a lot of it.

Ernesto: I'm glad to hear that. We are big fans of pair programming. How are guys doing it now? Are you doing it like just when it's needed or do you have scheduled pair programming sessions?

Preston: We generally do as needed, but also if there's a special situation where somebody who has knowledge needs to share with somebody who doesn't. We'll get them to pair up together while working on a feature just to transfer that knowledge. Additionally, now we have office days. Every Wednesday people come into the office. There is usually a lot of programming going on at that time since we're all sitting around together anyway.

Ernesto: Cool. So the last question is quite technical and it's about the cluster of server nodes we helped implement and if you could tell me a little bit about the problem we were having before that solution.

Preston: There were two problems, actually. One was that we were connecting to Gmail from the same IP address for all of our clients and that presented a security risk. If Gmail ever decided to ban us, they could quite easily do so by simply blocking our IP address. Or simply silently routing our messages just to spam or whatever it is they chose to do. So there's the security aspect. And then there was another more simple aspect, which was throughput. We were sending through Gmail and it imposed a limitation on how many messages you could send per minute by simply holding the connection open. So you had to wait around three seconds per message. Accordingly, the throughput was relatively low. We couldn't send for all of our clients in the timeframe that we wanted to.

Ernesto: So what value did we add there in terms of the solution, this cluster of sender nodes?

Preston: Well, specifically, the problem that you solved that I wouldn't have had a solution for was that of using queues to communicate between the sender nodes and the API server. The one-way communication from the sender nodes to the API, that would seem pretty easy to make an API call. But the other way, it was like are they going to connect to the database, or how exactly is this going to work? And then of course it made a whole lot of sense if you simply use SQS and they have a separate system waiting with the messages all there.

In retrospect, I should have been looking at it going, "Hey, this is obvious." But I didn't have that idea and you guys contributed that.

Ernesto: It was really cool to work on that. I think it's one of the coolest projects we worked on in the past three years or so.

Thanks for your time and your feedback! I really appreciate you taking the time to do this.

Preston: No worries. Hope it helps.

Read more »

AWS S3 Policies for Carrierwave

When you create IAM credentials and policies for your app, you should make sure that they have access to the resources that they need and not more than that!.

This way, if anyone gets access to those credentials, the impact of this leak is reduced to the resources associated with them (and not all the buckets in your S3 account)

Read more »

'Flaky' tests: a short story

One of the hardest failing tests to debug are those which fail randomly, also known as "flaky" tests. You write your test cases, you run the tests in your environment (in random order), and see them all pass. Afterwards, you push your code, your CI server runs them and one test fails.

This is not an uncommon scenario, and one too common when using integration tests which use JS, with Capybara-Webkit or Selenium. But if your failing test doesn't communicate with an external API, doesn't use JS, and passes locally, it can be a bit nerve-wracking.

After you have identified the failing test, and it still passes after running it locally, one way to figure out why it's failing is running its context multiple times.

To automate this process a bit, I like to use the following command:

Read more »

How to run multiple dependent builds on Circle CI

We all know the importance of having a CI tool integrated in your project. It allows you to run your entire test suite every time you want to merge a set of changes. If you have a core project and many projects that depend on it, you want to run the tests for the core project and the dependent projects at the same time. This article explains how you can do it with Circle CI.

Read more »

How to test a React app using capybara-webkit

I have been using the capybara-webkit gem for a while now since I first tried it out after making the switch from Capybara + Selenium.

Using capybara-webkit speeds up my Selenium tests due to its headless nature, and it's very straightforward. However, I had some trouble testing a Rails based React app.

In this post, I will explain how I worked around the issues that came up when trying to use capybara-webkit with React.

Read more »

Announcing AfipBill

If you live in Argentina and you ever use AFIP, you should already know that their platform is not the best in terms of user friendliness. We wanted to integrate OmbuShop with AFIP (using their API) in order to generate and print the bills for each seller. Unfortunately, there is no way to do this because the API doesn't generate a printable version (PDF) of the bill.

Read more »

How to use any gem in the Rails production console

How many times did you come across a great gem you wanted to try out in a production console, like benchmark-ips or awesome-print?

Be it for performance or for readability, sometimes it's nice to be able to try out something new quickly without going through a pull request + deployment process. This is possible by modifying the $LOAD_PATH Ruby global variable and requiring the gem manually.

Read more »

Present? vs Any? vs Exists?

When working on a Rails project, you may have seen present? calls on ActiveRecord relationships. This might feel natural, mostly because present? exists on all objects via ActiveSupport, so you expect the relationship to respond to it, but it's actually not a very good idea. If all we want to do is check if the scope returns any results from the database, there are better ways than using present?.

Read more »

Brief look at RSpec's formatting options

A few weeks ago, I noticed weird output in the RSpec test suite (~4000 tests) for a Rails application:

.............................................................................................unknown OID 353414: failed to recognize type of '<field>'. It will be treated as String  ...........................................................................................................................................

This Rails app uses a PostgreSQL database. After some Googling, it turns out that this is a warning from PostgreSQL. When the database doesn't recognize the type to use for a column, it casts to string by default.

Read more »

Tips for Writing Fast Rails: Part 1

Rails is a powerful framework. You can write a lot of features in a short period of time. In the process you can easily write code that performs poorly.

At Ombu Labs we like to maintain Ruby on Rails applications. In the process of maintaining them, adding features and fixing bugs, we like to improve the code and its performance (because we are good boy scouts!)

Here are some tips based on our experience.

Prefer where instead of select

When you are performing a lot of calculations, you should load as little as possible into memory. Always prefer a SQL query vs. an object's method call.

Read more »

Spy vs Double vs Instance Double

When writing tests for services, you may sometimes want to use mock objects instead of real objects. In case you're using ActiveRecord and real objects, your tests may hit the database and slow down your suite. The latest release of the rspec-mocks library bundled with RSpec 3 includes at least three different ways to implement a mock object.

Let's discuss some of the differences between a spy, a double and an instance_double. First, the spy:

[1] pry(main)> require 'rspec/mocks/standalone'
=> true
[2] pry(main)> user_spy = spy(User)
=> #<Double User>
[3] pry(main)> spy.whatever_method
=> #<Double (anonymous)>
Read more »

The Need for bin/start

Getting started with a new project should be as simple as possible, even for someone who is not technical. As a maintainer, you must make sure that anyone can clone your project and get it up and running in a few minutes.

After you clone a project, you should follow two steps:

  1. Setup
  2. Start
Read more »

Tips for upgrading from Rails 3.2 to 4.0

There are already quite a few guides in the wild to help with the upgrade of Rails 3.2 to Rails 4.0. The official Rails guide for upgrading from Rails 3.2 to 4.0 is very thorough. With the recent release of Rails 5.0, apps currently in production running Rails 3.2 should probably be updated to any stable Rails 4 release as soon as possible.

There is even an e-book about upgrading from Rails 3 to 4, which serves as a useful guide to make this upgrade easier, and also helps understand the advantages & disadvantages of this new (soon to be old) version.

However, if you're using any non-standard gems, you're mostly on your own. Some gems stopped being maintained before Rails 4 was released, as was the case with CanCan, a well known authorization library. After many open pull requests were left unmerged, CanCanCan was released. It is a community driven effort to have a semi-official fork of CanCan. It serves as a drop-in replacement for people who want to use CanCan after upgrading to Rails 4.

Read more »