This reminds me of the concept of the conflict resolution diagram from the theory of constraints. You have a seemingly mutually exclusive conflict (more features vs. more refactoring), that can be broken by both parties realizing they have the same goal. Thank you for your well written article!
1 reply
Great writeup. I've always been on the 'programmer' side of the fence and it's very easy to overlook the fact that business and the the programmers want the same thing in the end.
Nicely put. Great things happen when we listen to what others means rather than what they say, and then respond to that => Get a shared understanding of the problem and solutions will come by themselves.
Rather than talking about compromise/equilibrium of refactoring vs features, I use another reasoning to make the code base better aligned with business. The reasoning is short (and simplified) : For a given feature, a good programmer will produce better code in less time than a bad programmer (that will have to stabilise a more complex solution). A programmer is good mostly because he has trained. So if a company wants better code, then better create a training environment than invest loads of time for refactoring.
I've found much less resistance to this than other approaches I've tried. I also think that life would be easier if we were better at demonstrating the business value of good code/architecture/infrastructure etc. That's what I'm working on right now.
1 reply
Merci, Johan, de tes commentaires.
Unfortunately, I only have a model and no hard evidence for the claim that refactoring reduces volatility in the cost of new features. Measuring this stuff creates all kinds of problems, because of the sheer number of confounding factors in the research. I also have only an intuition that the cost of refactoring doesn't dominate the costs due to unhealthy design. I think the difference shows up most in support costs, but you can't measure support costs that you don't incur. :) I've seen organisations spend 70% of their development budget fixing problems, and I think that we can design ourselves away from that problem, but I haven't got the permission to run a controled-ish study-ish with that hypothesis.
Not only that, but we have to be careful about measuring the learning curve, just one of the many confounding issues that makes researching this topic difficult.
Please, please, please share your results with us as you go.
1 reply
Yes. Normally I use conflict resolution diagrams to look for obvious contradictions as a wedge to encourage people to change their behavior. In this case, we have unexpected alignment! I love it.
Nicely framed J.B. I think one dimension to this struggle is time and making the right overall decisions within a given time span. Within a frame of weeks or months the decisions lean towards the 'business position' and over years towards the 'developer position'. So, I think a dialog around is more constructive if is framed within a mutually agreed time span.
1 reply
You're talking about evidence and measures. I guess you interpreted my last sentence as something of the sort "I'm working on a way of measuring value of good code/refactoring".
What I meant was that I'm working on sharing my "insights" on the value of it. Btw it was not measures that convinced me so there are other ways. Together with Remy Sanlaville we've developed a 20 min code demonstration for business users. It's still a bit crude but worked great on a few beta testers. I'm dying to share more details about it to get some feedback, but right now I really have to get back to preparing a workshop for tomorrow :)
1 reply
Aha. Well, whatever we can do to make this concept clearer interests me. When you have energy, please share.
Indeed. I think that a perceived difference of goals and objectives lies at the heart of this pendulum swing. We agree on pretty fundamental things; let's use that to our mutual advantage.
I don't think we need to measure the cost of refactoring, so much as the cost of *not* refactoring.
Nicely put!
>> The Programmers need to agree that the Business has a legitimate need to deliver more features, that everyone’s livelihood depends on this, and that trying to ask for more than the Programmers can deliver forms part of the Business’s “job”, as it were.
My key lesson: Business and Programmers are in the same boat.
Do they share a common goal? (Do they really want the same thing?)
In this one aspect, yes: the Programmers want to deliver a steadier stream of valuable features, although they probably focus on the value of "it works"; the Business wants to receive a steadier stream of valuable features, although they probably focus on the value of "it excites the customers".
In my opinion the article translates to is bureaucracy makes life harder for the programmers.....
1 reply
“If only you’d stop with this refactoring nonsense! The economy has to grow. We have to grow. The board of directors yells at the vice presidents, who yell at the third-line managers, who yell at the second-line managers, who yell at us. More features! More features! They only want more features! Not only can they not afford for you Programmers to slow down, but they need you to speed up! I need you to speed up! What do you go faster?!”
- This part basically means Business does not understand(maybe does not want to) the needs and concerns of the programmer. But a programmer only wants what is better for the business, often business takes this negatively thinking programmers want to slow down become a parasite for the business.
Programming as a profession isn't a like manual labour. 9 women can't produce 1 baby in a month. This is were the upper management of the programmers should come in try to fix the gap as you say. This rarely happens quite like that in the real world. The communication becomes bureaucratic nightmare and eventually programmers give in to what the business want NOW.
1 reply
I don't see clearly what you're criticising here. I think you're criticising something, but your initial comment sounded like you wanted to criticise me or the article, but now I just don't know. :)
The Business's job is to ask for too much. The Technicians' job is to deliver as much as possible. If we agree that we're planning for 3 months, then we don't have to refactor, but then we have to throw everything away after 3 months. Often, the Business doesn't get this. But also, if we agree that we're planning for a 3 months, then we don't have to refactor. Often, the Programmers don't get this. They misunderstand each other; they disagree on the effects of refactoring or not refactoring.
Both sides get it wrong at different times.
As you say, though, when the Business "gets it wrong", they win the game, because ultimately the Programmers want their money -- except, of course, that the Programmers might be able to find another Business with whom they align better, and can ask for *their* money.
I still don't see the bureaucracy. If the Business and the Technicians won't talk about this situation, then it never gets better, but I don't see a bureaucracy.
I live in the real world, by the way.
1 reply
I don't really like neither craftsman nor artisans. We're engineers. Well, not all of them, but I am. The root cause here is that we are often directed to problem with aggressive randomness, pretending agile is a way to skip requirement gathering, specification writing and all testing but a little ui testing here and there.
If you need creativity to program, then of course features can take anywhere from zero to years to complete. With proper analysis, you usually find yourself in a position where adhering to requirements constrain you so much that there is only one obvious implementation to follow.
"A well posed problem contains the seed of its own solution" - cit. I can't remember from whom, which is doubly true in software engineering, because 99% of the time we are not conducting research in new fields, just using off the shelf components.
Now the question is, why we don't push back to proper engineering practices? Of course business love to use journeyman, as it cut costs, and that waters down the voice of us engineers when we scream that we are not psychic and agile doesn't mean interactively guessing business requirements.
Most of the time it is not the creative crafting that kills deadlines, it has more to do with the fact that deadlines are set arbitrarily, without having any idea of how many guesses we have to deploy before matching the unknown business requirements.
On my first leadership role, I got a six month development plan down and it was completed within a two day error margin. It took two weeks to write, because it came with cutting and assembly instructions. Because we designed the interface properly between components before developing them, having more upfront design and analysis and zero integration costs afterwards.
Wake up, developers! Push back the urge of pretending our job is fuzzy in nature! Some of it might be, but most of uncertainty in today development world is generated by the managers themselves, forcing fuzziness upon us and pretending clear cut software in return.
We are not apprentices, journeyman and masters. We are engineers, mathematicians, scientists.
2 replies
"First, the Business can easily justify their position: they need profit, and profit comes from sales, and sales come from features, and features come from, among other people, the Programmers. Without more profit, nobody keeps a job. Clearly, the Business has this one right."
Can you explain this? As written, it seems wrong to me. Unless the business just took a bunch of external funding and is operating in the red temporarily, why do we need *more* profit just for us to *keep* our jobs? The reason business generally wants more features is because they want to *grow* into new markets, and new markets require new features. If we already have paying customers, they will continue to pay (and in my experience, get even happier!) if we do nothing other than fix bugs and improve performance and reliability.
(I've had managers try to justify feature-chasing by telling me that customers expect new features, but I've never heard of any customer leaving because of this. We've definitely had customers leave because of stupid bugs, or because we didn't fix some basic performance issues, though.)
This is the fundamental schism I've always seen between business and programmers. Programmers want to take something good(-but-sloppy) and make it great. Business wants to take something good(-but-sloppy) and get it in the hands of a lot more people. This is frustrating to programmers because nothing can ever be truly finished. Imagine the programmers asking the business people to abandon leads that are 90% of the way through the sales pipeline because they think that's good enough. Even if it weren't a financial loser, it would surely feel very professionally unfulfilling.
1 reply
I think he's referring to the game of telephone that happens between (in your stated scenario) VP > 3rd-line manager > 2nd-line manager > business person > programmer.
The hierarchy of management and the distance between "decision-makers" and "do-ers" is the bureaucracy that's being referred to.
The fact that the decision-makers and the do-ers don't have a direct line of communication makes any sort of arbitration difficult.
This, specifically, makes life more difficult for the programmer because of the balance of power - the VP can always fire a programmer but the reverse is not true. So a programmer simply has to do what he/she's told and is typically unable to even argue his/her case to the decision-maker (they can complain to the middle manager, but the middle manager has very little incentive to escalate the complaints).
At least, this is how I read initial statement.
So I don't think the initial statement was a criticism of yourself or the article, but rather the typical business case where decision-makers and do-ers do not have direct communication and the power is balanced almost entirely in the hands of the decision-maker.
1 reply
First of all, the vast majority of programmers I've worked with over the years have not had engineering degrees or engineering certifications. In many jurisdictions, you're not even legally allowed to advertise your work as "engineering" without that. If you are, then good for you, but you're probably not talking about the 99% of us who are at CMMI Level 2 or below.
Second, you lump "engineers, mathematicians, scientists" in the same bucket, but these are completely different professions. Engineers have a lot more in common with craftsmen than they do mathematicians. John Nash would have been a *terrible* engineer. Thankfully, he realized this in college and switched from engineering (like his father) to mathematics (where he made huge contributions to the field)
Third, you say "With proper analysis, you usually find yourself in a position where adhering to requirements constrain you so much that there is only one obvious implementation", which may be technically true, but only helpful if "proper analysis" were not also part of the programmer's job. I have never in my career been given a complete, unambiguous specification -- as you say, managers themselves are forcing fuzziness on us. We could make the "coding" part of our jobs deterministic if we limit ourselves to only writing code after the specification is perfectly complete, but since Business doesn't care about this distinction, there's nothing to be gained by eliminating parallelism here.
You seem to associate "engineering" with reliable scheduling, but I don't see it. Every day I can turn on the news and hear about some engineering project that is behind schedule and over budget. Seattle's tunnel project was originally supposed to be finished by fall 2014. Now at 9 months in, they're only 10% of the way done, and currently stuck, with current estimates saying they should be able to start digging again in March 2015. I've been on some bad software projects, but none so bad that they had to take 10 months off from coding just to fix the compiler.
Finally, you claim that "99% of the time we are not conducting research in new fields, just using off the shelf components", which does not match my experience at all. I'm sure there are places where all they're doing is mashing Lego together, but I've never seen the inside of a company like that. Every company worth its salt today seems to have Github repos full of recent projects, with their own patches -- "off the shelf components" that didn't exist 3 years ago, and were never designed to work in the ways they're being used today. The more innovative ones (like Apple, Google, and Amazon) even have their own compilers, operating systems, and databases.
Some generalisations. Bear with me. Either a business sustains itself by its profits or participates in welfare/patronage. Yes, we hope to have periods where we have sufficient reserves to withstand low profit periods, but eventually, either (1) profit becomes the bottleneck to sustain the business or (2) a benevolent force sustains the business irrespective of its profit situation. I assumed (1), because in the case of (2), it doesn't matter what else we do as long as we appease the benevolent force that funds our work. In the welfare/patronage situation, all bets are off; therefore, I assume that we don't find ourself in that situation.
Yes, in some situations, "just keep it working" will mostly satisfy a customer base, for a while. It might satisfy enough of your customer base for more than long enough to satisfy your personal needs for money. On average, however, customers want more in the long term.
One aside: I did assume in this whole article that we find ourselves in the context I stated, because of how frequently I see it. In those situations that don't match this context, this particular opinion might have as little as no value.
Yes, I have remained a customer of some services out of inertia or genuine satisfaction. This happens, for example, when it costs so little to remain a customer that I don't need to optimise that part of my life. I find value in remaining a customer on autopilot: just keep everything working, don't change anything I use, and so on. Eventually, however--and it might take years--a competitor offers me something better-enough to get me to consider switching. Or perhaps my financial situation changes and it now becomes worth the effort to reconsider my choice. Or perhaps a friend tells me how much happier e is with another service, and convinces me to switch. Complacency represents a risk in this environment. (A risk doesn't necessarily become a problem.)
(Incidentally, I think about improving performance like adding features. "We can now handle load that we couldn't handle before." or "We can handle customers that we couldn't handle before." or "Customers can trust assets to us that they couldn't trust before.")
And yes, in spite of the unexpected alignment that I've tried to highlight here, we might see more perfectionists among programmers than among business people, and my observations here have no real bearing on that.
Thank you for trying to clarify. I think I agree. I still have no idea how to my article translates to any of that, nor why the original commenter brought it up.
My experience of business requirements vs developer requirements are a bit more simplistic. I have found the business types spend far too long brainstorming new features and then telling clients about them. The client immediately says "Yes, we need that now!!!" and then the developers are expected to get the new feature integrated ASAP. The problem is that they may not do a great job integrating the feature due to time constraints resulting in various fragmentation issues in the long run. I've had this conversation many times "It's working, but the back end is ugly. I'll have to tidy it up ASAP", "Can we deliver it to a customer right now?", "Err yes, but...", "Don't care it's good enough".
I think the key to preventing developer/business conflict is having a road map. That way the business types can get the features they want and the developers can think ahead about new features they will have to add.
Most business/programmer conflicts I have encountered are nearly always due to the sales people overselling the product. My old IT director, a formidable man who took no nonsense, announced that we would not add features unless they were in the road map. The CEO overruled him so nothing changed.
1 reply
I, too, have experienced the problem of the business overpromising. I see it as baked into system: we seek perpetual growth, which we assume comes from perpetually-rising profit, which we assume comes from more sales, which we assume comes from more features. I don't believe in this worldview, but neither do I benefit much from pretending that my employer's sales force doesn't believe in it. If they do, then I find myself in the situation I describe in this article.
This worldview necessarily eventually makes the feature delivery system a bottleneck. If it didn't, then we would never have had books like _The Goal_. That sets the scene for my words here: in this situation, the business people and technical people think that they want different things, but they have a strong common goal: reducing the volatility in the cost of adding features, which we believe will help us get off the merry-go-round of overpromising and underdelivering. We do run the risk, however, of a new imbalance: maybe the business uses overpromising as a business model. (I guess that you mean this in particular.) This means that the business has made peace with burning out the feature delivery system and everyone who works in it. If you find yourself there, then I hope you can get out. I teach people how to do that.
I hope that once the business people and the technical people see this unexpected alignment, that they'll have better conversations about when to release features, and so on. Maybe we *can* release a feature before the programmers put their stamp of approval on it, as long as our customers understand the risk. The programmers can help the sales force understand the risk, and even work directly with the customer to find ways to mitigate the risk. I don't want programmers to become the bottleneck to releasing features! I'd like to avoid building up an inventory of features for which customers would gladly now pay money! (On average; we can think of exceptions to this rule, but I hope you get what I mean.)
I wouldn't call having a road map "the key". I see it as a tool to agree on what to expect, but like any plan, if we believe in it too strongly, then we end up building features after we've long forgot why we need to build them. I agree that we need a common understanding of where we want to go, and a road map can help with that, as long as we don't turn it into a five-year line-item-level plan. (It happens.)
This leads to a much more complicated discussion, and I don't think I can address it adequately in a comment text field. :)
Interesting POV, to which I partially disagree.
I was recently working with a mathematician gone developer, and his skills in organizing code into separate layers and reusable components were quite poor.
A single data point, but I think this capability comes largely from experience, not from a mathematical mindset.
After encouraging the whole dev team to document more their code, insisting on the "why-not-how" and spend-time-properly-naming-stuff parts, I was still quite dissatisfied, but gained some insight when I found out that when even they had to explain the algorithms as they were thinking them, it was still a big mess. Corner-cases not explored, using name A for something which is actually a B, etc...
The capability to see bugs before they happen is imho a thing of experience.
You probably remember the day you realized that storing all dates as timestamps in UTC is the only sensible way, and that storing them in localized-timezone format or truncated-date-such-as-a-day is calling for trouble. Or when it dawned on you that assigning different meaning to the NULL, FALSE and 0 values of a variable was not a good idea. Etc... (just a couple of random examples of things which are good coding practice but hard to teach and check with automated tools).
But, more importantly, the capability to see business requirements changes before they arrive is really the holy graal, much more than the propension for good architecture. And again, that takes experience, it is not hard science.
If you are the genius-technical-program-leader, all my respect.
But more often than not overspecification does not lead to better projects. That it does not work is proven: otherwise we would all be doing waterfall in 2014.
The problem with specifications is that they always lie, because the end user is not an engineer
. his ideas are vague and contradicting
. he has no clue if a small change he asks for has a huge impact on development or not
. he can not even see the better ways to do things he asks for which are in fact possible
Also the developer is bound to the spec, not to the user needs, so
. he can not question the need for any feature or propose changes
. he has no vision of the big picture
. he ends up coding something that will clearly be thrown away in the next day
In most projects, the more you give importance to the spec, the more you end up putting diving walls, and reducing communication. Between end user and programmer you now have business-requirements-analysts, project manager, scrum master, maybe more. It ends up a bit like when you were young and played cord-telephone.
True, the more pressure you put on churning out features upfront, the more tech debt you incur, and the bigger the need for refactoring later. But pressure is not the only thing.
Communication is the key, and the more your project makes it flow, the better.
"Don't blame the Business for this: ..." - exactly! All blame should go on the programmers. This maybe sounds sarcastic but really, I'm serious.
Why do the business folks even know what refactoring is? Who mentioned the word "refactoring" to them?
Do I know procedures my mechanic applies when fixing my car? Or my doctor when fixing my body?
There is a similar topic about testing and TDD. "I asked my manager should I write tests." Why, oh why in the name of sanity would you ask that???
Programmers who don't refactor (or write tests) simply aren't being professional. They don't need permission from their managers to write code which ensures their employer's future in the IT market - its implied!
Anyway, great article. Although there is one thing I would have liked - the practice of continuous refactoring (i.e. you shouldn't do stop-the-world-refactoring, but should refactor a little in each story/task) is IMHO not made explicit enough. A link to Ron Jeffries' awesome article on the subject would be great: https://ronjeffries.com/xpr...
1 reply
I think I understand the impulse behind your comment, and in general I agree, but I feel even better when I can explain my work to the stakeholders in a language that they understand.
Doctors already tell you what they plan to do and even some details of how they plan to do it. I feel better when a mechanic can explain to me what they intend to do, because then I can judge their competence without waiting for them to butcher my car. The people renovating my house needed to explain some of what they intended to do, because otherwise how would I know that things were going OK? I say all this because I don't think we should always equate "professional" with "opaque". If you can't explain some aspect of what you're doing, then why do you do it and why should I pay for it?
Once we establish more trust, certainly I will delegate more of this responsibility to you. I won't ask as many questions. But where does this trust come from?
"Programmers who don't refactor (or write tests) simply aren't being professional. They don't need permission from their managers to write code which ensures their employer's future in the IT market - its implied!" I certainly don't agree with the first sentence; if someone can produce a steady stream of features without refactoring and tests, then they should do it. I agree in principle with the second sentence, but I refer you back to the examples earlier in this comment: when we have established trust, then the relationship becomes easier, but where does the trust come from?
Regarding the practice of continuous refactoring, it can't hurt to make that more explicit, so I'll look for a way to do that. Thank you for the suggestion.
1 reply
I agree that some doctors and mechanics explain what they do to patients/customers. However, as a customer, I have no ability to use this information to assess the professional's quality.
It is another matter if they explain that I might experience some pain walking for a week, or that the new brakes need a couple of days of use to properly function. Also of course, I will appreciate if I'm asked to choose between this feature or another.
But software refactoring doesn't fall into that category. The transparency you (and I) wish to have is early customer feedback and is usually implemented with CI/CD. Tied to that is your car-butchering example: We do agile software, not cars - I can butcher your software because a) the amount of butchering is very small and b) you'll give me feedback on it in a couple of day at the most. With software, I can revert/fix the butchering very easily.
To me, refactoring or not (tests or not, code review or not, pairing or not, ...) is like if a doctor would ask me which scalpel to use for the surgery. Or which surgical technique (probably more to the point). I don't care. More importantly - I'm not qualified to make that choice (even if I do care for some obscure reason)!
I'm not qualified to make that choice because I'm not er *peer*. Now, a peer would be able to make the choice, or assess whether er choice was bad or good. A customer can only assess the external factors: functionality, duration of the work, etc. A customer cannot infer that by doing refactoring er software will be better or arrive faster, nor should e attempt to. E simply is not qualified.
Actually, you said a similar thing yourself in your reply: "if someone can produce a steady stream of features without refactoring and tests, then they should do it"
Which brings me to the second part.
I admit that my "Programmers who don't refactor" sentence was a bit harsh. I agree that a programmer is equally professional if e produces a steady stream of features with or without tests or refactoring.
But...
Less importantly: How many of such programmers have you met?
More importantly: What about others on er team? Do all members have such discipline, technical skill, and ability to communicate with code?
"However, as a customer, I have no ability to use this information to assess the professional's quality." I do. Well, almost. Specifically, I can use metainformation to assess the professional's intentions. Occasionally, this will fail, because some people can say "the right things" and do terrible work, but if this happens commonly in some context (some industries, some geographical locations), then I will probably learn about that in advance.
I don't compare "refactoring or not" to "which scalpel do I use for surgery". That compares more to "which IDE do I use" or "which programming language do I use" or "which database management system do I use". The scalpel is a tool; but refactoring is a technique.
Different surgeons use different techniques. Not all surgeons can use all available techniques. Not all surgeons even know about all available techniques. Some surgeons specialize in a technique. I consider refactoring (really, evolutionary design in general) in this category.
Bigger, we have differences in approach. Some doctors (not surgeons) do not understand about low-carbohydrate diets; some doctors know about them and have wrong impressions about the results; some doctors know about them and argue about the evolving understanding of the underlying biochemistry. I know this: I ate 95% fewer carbohydrates and lost 70 kg in four years. I know this: when I eat more carbohydrates, I gain weight, and when I eat fewer carbohydrates, I lose weight. Unfortunately, this means that I can't benefit from a doctor who believes that eating low in carbohydrates "doesn't work". I consider evolutionary design in this category.
So once again, even if my Customer can't judge how refactoring affects the quality of the result, if I say "We don't refactor here", and in the Customer generally understands how refactoring generally affects one's capacity to deliver features, then I find it reasonable for the Customer to feel worried about working with us. We have to work harder to earn that Customer's trust. How could it be any other way? The Customer might still invest in working with us, but then we need to retain that trust by delivering in a way that satisfies the Customer. We simply might start from a place of slightly less trust than another group that shows the Customer that they approach design in a way that they already believe creates good results. Again, how could it be any other way?
The questions that you ask in the end really interest me. I have a biased view of the correlation between evolutionary design and producing sustainable designs, because I have been teaching evolutionary design for 15 years now, and so I see more people learning evolutionary design and practising it. I know one significant data point: a bunch of well-trained, highly-skilled programmers who don't practise TDD any more, because they have built an environment in which they don't need it AND they have already learned "enough" from practising TDD, so they have those benefits without the practice. (Search the web for "programmer anarchy" to learn more about them.) I know some of these programmers personally, so I also have some bias in their favor. It tells me that such people can exist.
So I cannot remember meeting a programmer who produces sustainably-maintainable designs who does not practise TDD, but that reflects my life choices more than it reflects the world. Because of this, I can't really answer the second question.