Easily Parsing Failed Cucumber Scenarios from the JSON Report

Every so often, I have to deal with failing Cucumber tests. And even with the best visualisations (as per my article Prettier HTML Reports for Cucumber-JVM), it can still be a pain to pick through the reporting to work out what's failing, and why.

So I decided instead I would script the parsing of the Cucumber JSON report, which can help me more easily determine what's wrong.

#!/usr/bin/env ruby
require 'json'

report = JSON.parse(ARGF.read)
failed = []

report.each do |feature|
  feature['elements'].each do |scenario|
    next unless %w(scenario background).include? scenario['type']

    if scenario.key? 'before'
      scenario['before'].each do |step|
        if %w(failed undefined).include? step['result']['status']
          f = {
            feature: feature['name'],
            scenario: scenario['name'],
            step: 'Before hook',
            failure: step['result']['error_message']
          }

          f[:step] += ": #{step['match']['location']}" if step.key? 'match'

          failed << f
          break
        end
      end
    end

    scenario['steps'].each do |step|
      next unless step.key? 'result'
      if 'failed' == step['result']['status']
        failed << {
          feature: feature['name'],
          scenario: scenario['name'],
          step: step['keyword'] + step['name'],
          failure: step['result']['error_message']
        }
        break
      end
    end
  end
end

failed.each do |failure|
  puts "#{failure[:feature]}: #{failure[:scenario]}: #{failure[:step]}:"
  puts failure[:failure]
  puts ''
end

This can then be run, i.e. as:

# i.e. https://github.com/damianszczepanik/cucumber-reporting/raw/master/src/test/resources/json/sample.json
$ ruby cucumber-failures.rb sample.json
Second feature: Account may not have sufficient funds: Before hook: MachineFactory.wait():

Second feature: Account may not have sufficient funds: And the card is valid:

However, these steps don't have any errors to show, so the output is empty. But if we do have errors thrown that get caught by the Cucumber run, we would see i.e.

List: When I append to a list, it appends: Then the list has 1 item in it:
java.lang.RuntimeException: BOO
	at me.jvt.hacking.Steps.theListHasItemsInIt(Steps.java:41)
	at ✽.the list has 1 item in it(features/List.feature:6)

Written by Jamie Tanna's profile image Jamie Tanna on , and last updated on .

Content for this article is shared under the terms of the Creative Commons Attribution Non Commercial Share Alike 4.0 International, and code is shared under the Apache License 2.0.

Tags

This post was filed under articles.

Related Posts

Other posts you may be interested in:

Interactions with this post

Interactions with this post

Below you can find the interactions that this page has had using WebMention.

Have you written a response to this post? Let me know the URL:

Do you not have a website set up with WebMention capabilities? You can use Comment Parade.