Beta Testing WordPress Plugins with WP RollbackThe real-world plight of the WordPress plugin developer cannot be understated. With WordPress powering now 25% of the entire internet and new hosting platforms birthing at a shocking rate, testing plugins under every possible environment and configuration is very literally impossible.

When plugin developers face that reality dead-on, they often are left with one of two responses:

  1. Can’t win ’em all… oh well!
  2. How can I get more beta testers?

#2 is the right response, but it’s fraught with complexity. Part of my job at WordImpress is specifically to get as many beta-testers to test our plugins and provide us with actionable information as possible. It’s no small task. You have to find people willing to test it, who can test it in a safe environment (meaning NOT a live site); testers who can provide you with enough detail to reproduce the problem or who are wizard’s enough to provide a fix. If you have users who are desperate for your next update, but you can’t get enough beta testers, you’re stuck with the reality of pushing your code live with quite a bit of uncertainty.

In light of all that, we have a proposition for plugin developers which might help this scenario become a lot easier without changing your workflow significantly at all. But first, some context.

The Yoast SEO Backstory

Recently, we noticed a significant uptick in reviews for our WP-Rollback plugin that we released earlier this year. They each were 5 star reviews and they each were mentioning the Yoast SEO plugin.  Here’s an example:

WPRollback-Review-Must_have_plugin

We love Joost de Valk and all that the Yoast Team produces. In fact, when we noticed that some of our Give customers were having trouble because of Yoast SEO, we jumped on Github to see how we could help. Luckily someone had already provided the exact fix that we needed and that fix was rolled out relatively quickly.

Joost put together a really succinct article providing some context to this major update. There’s several points in there that are worth mentioning here:

“We knew right when we hit the publish button that there’d be bugs”

This is that reality that ALL plugin developers are faced with. It’s not because of laziness or lack of testing. I mean, they hired a new developer in March to start implementing some of these new features. That’s 9 months of development from a not-so-small team of Pro’s. Read their post on that here.

Despite all the development, Joost admits that the amount of people actually testing the new code was small. So he next asks:

“But how can we get more people to test our betas and release candidates more extensively?  We’d love your ideas.”

This is the rub. I mean — seriously — if a WordPress shop that sustains 10’s of thousands of websites has trouble getting quality testers and feedback, how can us up-and-coming shops get the feedback we need? A member in our Advanced WordPress Facebook group (which Devin, Joost, and I are each admins of) replied to Joost’s plea for more beta testing with this:

 If you gave me an easy way to opt into tests – say on a staging site, I get a beta update banner in a fancy green color asking whether I want to opt in to test the new version – I would definitely test.

I chatted with Joost about all this and realized that honestly, WP-Rollback — though intended for rolling BACK — could be the solution. We haven’t done this with our own plugins (yet!), but conceptually I thought: “Why not just add a beta version to your plugin repo for WP Rollback to install?” Joost loved the idea, so I tested it and it works perfectly. Here’s how to do it:

Setting Up Your WordPress Plugin for Beta Testers

If you have a plugin on the WordPress Plugin Directory already then you’re already familiar with SVN and tags. All you need to do is create a new tag that clearly indicates that it’s a beta version. Here’s how I did this with my ImageLens plugin:

1) Create a tag folder for the new version

ImageLens is currently in version 2.0.1. I want to direct my users to beta test version 2.1. So I created a folder called tags/2.1-beta

2) Copy the trunk version or add your beta copy to that folder

Next you need to add your beta version to the folder. Pretty simple.

3) Update the readme.txt and PHP version in your beta copy

Lastly, you’ll want to update the “Stable tag:” in your readme.txt and your PHP version tag in your main plugin file to match your beta version folder name. Make sure you are only changing this in the new beta folder. Because you haven’t changed the Stable tag of your “trunk” version, the version that is live on the repo and available for users to download doesn’t change at all.

4) Install and Use WP Rollback

We covered this in a previous post, so check that out first.

WP Rollback is able to list the available plugin versions according to your tag folders in SVN. So now that you have a new tag that clearly shows that your plugin has a beta, you can direct your users to install that via WP Rollback. Here’s what that looks like for my ImageLens plugin:

imagelens-rollback2

Once your user chooses that version and hits “Rollback” — viola! They are now a beta tester for you. Naturally, there’s a lot of possibility with this. You could add unique code in your beta versions that alerts your user that they are working with the beta — like custom admin notices. Or you could add a “Beta Feedback” tab to your settings page to allow your users to provide detailed feedback to you via a form in that tab and even have it automatically send you the system info in the submission.

The great thing about this is that you don’t have to go shipping zip files to users, just direct them here, and if they have trouble with the beta, they can just rollback to the current stable version. Of course, we put alerts ALL OVER our docs and within the plugin saying you should never be rolling back (or forward) without proper backups anyway.

But I Want to Develop and Track Issues on Github

I knew you’d ask. Again, I haven’t tested this out yet, but here’s what I’m thinking:

Use a script like Mike Jolley’s Github-to-WordPress to deploy a BRANCH of your plugin to your SVN tag. This allows you to continue to develop on Github, and still have your beta version stay static. According to Mike, it would take a little customization of the script to make it happen, but it’s certainly possible.

Once you have fixes and updates to the beta that you want further testing on, you can create a new tag in SVN called 2.1-beta RC1, and push that from a new branch. That way, you can still encourage developers to create issues on Github and even link directly to lines of code from that branch, while still giving a much wider audience the ability to easily and safely test your beta versions.

Just Beta Test It!

Generally, I believe the reason plugins aren’t beta tested enough is simply because developers aren’t running successful beta campaigns. It’s not that people don’t want to test. It’s that they don’t know how to, and aren’t encouraged to or both. I’ll follow up on this concept more in a future post, but it’s my belief that WP Rollback can help make sure that as many people as you need can access your beta version safely if you are willing to take the time to ask them to.

 

Matt is Head of Support and Community Outreach at WordImpress.com. He's the author of many free WordPress plugins, a popular blogger at his website, an admin of the Advanced WordPress Facebook group, co-organizer of the San Diego WordPress Meetup, and a frequent WordCamp speaker and attender.

Follow Matt:

Leave a Reply