Who tests the contract tests?

Articles
1

July 9, 2018 (updated April 25, 2020) Integrated Tests Are a Scam

This is a companion discussion topic for the original entry at http://blog.thecodewhisperer.com/permalink/who-tests-the-contract-tests
2

I think the idea is great and I saw the presentation. In fact I was keen to try in out straight away on a java project with JUnit+JMock. The package that I was trying contract test on was about conversion of data. I had several data converter implementations that has common generic interface Converter<original, converted="">. What I did is set up an Abstract JUnit test for my Converter interface to test all the common bounds of functionality. After that for each of the concrete implementations of the converter I have created a JUnit test that is inherited from the AbstractConverterTest.

So in basics the AbstractConverter had all the tests but the creation of the instance was delegated to the inheriting test through abstract method getInstance(). This worked FANTASTIC - I could see how some of the converters that I had do not confirm to the contracts.

My first impression of the Contract test was "This is quite an overhead to make it in this way", but my little investigation really showed that it pays off.

The only downside that I have faced is that abstracting the getInstance method, does not always play nice with mocked objects, since you have to set the expectations for them. I know that this is kind of JMock specifics but do you have any best practices advise on that?

3

Thank you for your comment, Denis. I think I can help you.

Can you upload some of your code here? If so, then I can see your example and give you more specific advice. Please go to http://www.thecodewhisperer... to register and post an item under "I Need Help!" We can work through your example there, and maybe other readers will have suggestions.

4

Many years later, https://github.com/psyho/bogus is trying to test the contract tests.

5

Hi Joe,

I've just watched your Integrated Tests Are A Scam presentation. Do you have references to any talks, presentations, articles which would cover in details (ideally with code samples) the main difficulty with contract testing? It's not obvious how to build a practice that would help keeping server and client code tests in sync. I assume that some gap from two sides of contract tests is inevitable during the development life cycle. Code coverage tools are useless in detection and measuring this gap, right? It's also impossible to imagine at this point a tool that might help with this issue. Do we have to rely on some habit, process, practice to minimize the gap?

P.S. Thank you for a very useful and bright talk!

6

Hi, Igor. I'm not sure which version of the presentation you've seen. Look through http://blog.thecodewhispere... and perhaps that'll help you with more details, but in short, to keep server and client code tests in agreement requires discipline. As of early 2016, nobody has produced a tool to automate this, and I don't think such a tool is possible.

I use a simple system: I know the correspondence and I look for it. It's exactly the same to me as checking that comments and code match, then fixing them when they don't, often by removing the comment. :) A stub in the tests for the layer above corresponds to assertEquals() in a test for the layer below and an expectation in the tests for the layer above corresponds to an action in a test for the layer below. That's it.

Especially if different teams write the server and client, then you simply have to talk to each other. You can't get around it. As the client, I'd use the contract tests as a classical acceptance test suite for the server: if you don't pass these tests, then I can't use you. When the tests fail, we have to negotiate which side changes their behavior. :)

A gap is inevitable only in the sense that we're imperfect beings.

I hope this helps.

7

This is a very weak test, too, because it only check the syntax of the contract, and not the semantics. The compile-time type checker runs these tests for us when we work in languages like Java/C#/Haskell that have them.

Don't get me wrong: it's better than nothing, but it's still weak.

Good news: I find that the more I practise this approach, the more my design tends towards modules that depend on very simple contracts in other modules, reducing the risk.

8

I have completely rewritten this article, in case the new version interests those of you who have commented on it in the past.

9

Fantastic rewrite!

10

Thank you for sharing this great article. I really like the paint throwing analogy.

11

I wish I could remember how I thought of it. It's one of the rare independent contributions I've made to the field. :)

12

I just loved that "Paint A Wall" example. Awesome and thanks for sharing this!