Show HN: Spotbot, a Slack bot that monitors your website for visual bugs | Hacker News Reader

Show HN: Spotbot, a Slack bot that monitors your website for visual bugs

spotbot.qa

179 pointsjmtulloss59 comments

Hi Hacker News! We've been working hard on Spotbot for the past couple months and are looking forward to your feedback on what we've built. @mgc and I will be around all day to answer any questions.

Justin,

People cannot try free, your servers are responding with 503s. I tried to email you from your contact link on website (hi@spotbot.qa) and was undeliverable.

Well the email thing is pretty embarrassing, just fixed. I wrote "hi@spotbot.qa" and set up an email for "hello@spotbot.qa". Face palm.

We are experiencing a lot of load (or we were until we went down). We scaled up and things seem to be working better now.

"Pool is destroyed"

No longer destroyed!

The site I'm currently maintaining doesn't change very often, but it's approx. 1,500 pages if spidered, plus a large search index. I'm looking at the Fancy plan, but we've got multiple subdomains or internal versions of the site. How are these covered if it's not all under one domain? Are sub-domains part of the same domain? :)

Is it possible to integrate with a CI system for on-demand runs, perhaps by supplying a list of changed pages as a hint on where to expect changes and where not? The idea being that to lower your costs, a subset of pages could be run on-demand?

We don't currently have CI integration, but it's coming up fast on the roadmap. What system do you use?

We're still hammering out the pricing plans. If you email me at justin@spotbot.qa we can set you up with something that makes sense.

No specific CI in mind, maybe just Github integration or a webhook to start. I'm currently looking at options to modernize and add tests to a rather manual, SVN-based process.

I'll email for sure, I'm interested to see how this will work. Thanks! :)

I understand slack integration for some tools, but is this slackbot-as-a-product thing going to become the new fad? What if I want to use your tool without using slack?

slackbot-monitoring-without-slack-as-a-product

Coming soon to Spotbot :)

I was thinking the same thing. I'm interested in trying it out, but do I really feel like setting up slack for a side project when I'm the only one that works on it?

You can have it just post to you in Slack if you don't want to bother other people.

What's your preferred notification system?

No I mean a personal side-project that I have no slack set up for because I'm literally the only person that works on it. Not just the only person on a project in a company.

Anyway, email would probably be fine for me.

We plan on supporting other workflows soon, we just started with Slack cause that's what we use.

Next stop is probably email. Any other integration you'd like to see?

I've worked with many systems like this and know that notifications are too easy for individuals to ignore. Your primary action should be to open an issue in a bug tracking system, and let that system trigger the email/instant notification. This helps teams get a better picture of bug volume/impact/area over time, allows for prioritization of work and ensures issues don't get lost during major bug storms.

Also, being able to link a bug conclusively to a changeset, build or a deployment ID would be very valuable. Rather than building lots of deep CI integrations, you could do this by allow users to configure your monitor to look for version numbers in server hints (headers, hidden fields, status pages, etc).

How about an RSS feed?

I tried setting it up to post to my Slackbot but hesitated at the permissions (giving away everyone's email in particular, when I just want to notify myself right now.)

I feel like a shitty tirekicker for complaining, but I really would rather just get email alerts and then optionally integrate with Slack directly or via webhook/Zapier. The service feels a bit inaccessible.

Good to know, thanks.

We would also prefer not to ask for that permission. In fact, the only reason we do it is be able to call Slack's `users.info` API on your user. Which... we're only doing in order to be able to show a profile picture and your real name. Those things are nice, but not must haves.

I'll make a note to get rid of that permission!

Oh, yes, if that's all then I support typing in my name myself after account creation. :) Funny how such things have to go with APIs sometimes.

Right now, it's starting to look like Slack is becoming the new email.

I used to suffer from having 100 emails to shift through everyday. Now I just have to shift through 100 Slack channels!

Agreed - I was ready to sign up until I realized it was slack alerts, not emails.

I'm wondering if it's related to Slack providing seed money for startups [1].

[1] http://fortune.com/2015/12/15/slack-app-investment-fund/

Nope, although we do have a good relationship with Slack.

We started with Slack cause it's really easy to get started with and a lot of people, including ourselves, use Slack as the central repository of what's going on in the organization.

We realize that it's not for everybody though, more options will be coming soon.

Yep, same here FYI. I was bummed to see slack-only signup, and stopped because I didn't want to share my teammates' profiles.

I can see this working for static pages, not sure about pages that require input, user login, or display constantly changing data.

We're really excited about tackling those problems.

@mgc and I founded Spotbot with the thought that models trained with ML techniques can be immensely useful to developers (novel, I know). So much of coding, debugging, and firefighting is pattern matching, and machines are getting really good at pattern matching. That's how we're going to tackle the user login and dynamic data issues in the long term.

We have some short term solutions in the works as well.

For dynamic content, we'll have a way to ignore changes that are expected and don't mean anything significant changed.

For login, we may implement a token API that will allow us to log in with a bit of work on the developer's side. We can also script logins without too much effort if we are provided with test account credentials.

Thanks for giving me an opening to talk about all this :)

My app is a social network. A social network means that the two most important pages (global stream and user stream) continuously mutate because there is always new user-generated content.

Now, humans are very good at catching bugs there; if a the name of the user goes above the avatar picture next to it, humans will see it immediately.

But how can a tool like Spotbot not to be confused by all the normal changes in user-generated content, and to just spot structural differences?

Right now we're not great at it. However, the way Spotbot works means that we will be decent at it soon.

We do a hash of the structure, so we know exactly what kind of changes there are. For something like your page, the image sources, the text, the names, the votes, and other content change all the time. Soon we'll have a way to ignore those changes while still recognizing that other changes need to be vetted.

So, for instance, if the padding between the avatar and the user's status changes, we'll still catch that even though we're ignoring the avatar source and the text content.

Soon!

Thanks! I'm a little hesitant because it sounds a little like magic, but I'm happy to help with testing it, should you need testers. Any place I can subscribe for updates?

Couldn't you point it at a test environment that would be loaded up with non-changing content so that UI only changes could be tested against the same data always?

You could create a page which uses as much of the real production code paths as possible, but which returns the same "stories" every time. It takes a little dev work, of course, but it would almost certainly be worth it.

Super excited to try this — as an early start-up (and the founder in charge of UI), I KNOW our site has UI issues but I don't have the time to do full inspections.

What kind of "changes need attention" will Spotbot tell me about?

Right now it checks your site every 24 hours, and we try to alert on things that might actually need a person to verify (ie, not animations or videos).

We're in the process of building moderation tools so that you can refine what's important to you, and we're really excited about building training a model to be able to highlight changes that are probably broken. We're starting at square one though :)

I can't really say as to whether this is due to load, or some underlying problem, but it seems that - currently - if you just wait around for the system to get to your requested (free) screenshot, it ticks down to 4 Screenshots ahead of yours, and... Nothing.

Not a single one of the boxes seem to update anymore. They all just say 4 screenshots ahead of yours. It seems, I spoke a bit too soon. Something is happening now. I guess my cynicism got the better of me when they all hit 4 and just... Stopped :)

Shameless plug: http://shoov.io/

Looks pretty cool!

I like that the tool itself is open source.

We found that pixel diffs were pretty unwieldy for a lot of different scenarios. How do you deal with, say, a div inserted at the top that pushes all the content on the page down 50px?

Shoov uses webdriverCSS under the hood, so you can simply remove/ hide/ exclude that div.

We use Percy.io for this and can't recommend it enough. It integrates with your CI workflow, which makes a whole lot more sense to me than waiting until production. (Or to be fair, staging.)

Percy is awesome, no doubt. We're a little different than Percy though.

We don't require any code. Setup is just entering a URL. Percy doesn't really require code either, but it relies on you having already written integration tests.

We do structural diffing, not pixel diffing. Pixel diffing can be challenging to view as a developer since innocuous changes can cause massive visual pixel diffs. With structural diffing we can have a better idea of exactly what changed and highlight those regions.

We're definitely going to be integrating with CI workflows and non-production sites as well :)

this is cool, but I just wrote something like this at work using hubot and some node libs, and there are a couple extra options you can setup.

if they let me I'll release it OSS, but it was super easy to write, and it pulls diffs from the github project when sees a change.

I was actually working on a reverse of that, so if it sees a repo get pushed to it will let you know if the site changed or if it didn't.

hubot is so cool, but coffee script is just gross. meh.

other than that, it's been incredibly useful.

Interesting, but this needs to be integrated into CI, not Slack, to be taken seriously by teams larger than just a few people. And arguably, I tend to set up CI of any serious project before it goes to production.

What's your CI system of choice?

I really like Circle CI, but I've used (and enjoyed) most of the other services as well.

Hey It seems that spotbot does not cover for delayed page transistion effects in CSS. I have multiple objects appearing in row and pictures show currently each page in different stage and not fully loaded page :(

Other than that, awesome :)

Thanks!

We do try to deal with CSS transitions but I don't think we have every situation covered yet. What page were you trying it on?

I tried it with our product and it took a snapshot before all of the (wall time intensive) api calls finished. Is there a way to configure it to wait a few seconds?

Not currently, but this is a common request.

What would you prefer:

- A javascript object being injected into your page that you could call to indicate the page was done

- A setting within the UI that would allow you to configure a wait time

Instead of injecting something into the page, what about having the snapshotter running an is-it-done-yet callback periodically to see whether it returns true? That would allow some basic checks like "is this particular DOM element visible yet" or "how many entries are in this list" without needing to modify the production code in the page.

I don't really care about all of the calls, but I'd like to say "here's a regex pattern describing a uri. once any connection with uri pattern closes (within 10 seconds), you can render."

Yeah, this isn't great for websockets/long polling or other persistent methods, but it's fine for simple ajax-y calls.

This looks very useful! I'm wondering if it's possible to see a timeline of changes, like the history of changes for the last 30 days?

It is, although it's not the best UX at the moment. In the details about any page that you're tracking, the past snapshot runs will be listed on the left side. You can go to any point in time, view it, and compare it to the previous time something actually changed.

Got it! I can see that being helpful when looking at other site metrics.

What if my website isn't published yet? Is it possible to run this locally for a locally deployed website/webapp.

Why would I choose this over Browserstack?

We actually use Browserstack for some of our devices.

Spotbot monitors your site over time, so it'll alert you when things change. With Browserstack, you have to manually run your site in all the devices you care about each and every time you want to check something. Or you can write a Selenium test, but that's a lot more painful than just entering a URL and letting us take care of it for you!

Got it, thanks!

How will this work with a split testing tool like Optimizely?

Great question.

If you're willing to do a little work on your side, you can craft a URL that will guarantee that the user ends up in a particular bucket.

One of the features that we're thinking about is doing visual diffing on A/B test variations. Then you can page through your variations and quickly see how they're different from the control, or if they're broken on any browsers.

To do that we would need a deeper integration, either with the user or with the A/B service. I think it'd be pretty cool though!

Reply on news.ycombinator.com