Learning and groking recursion

I’ve started learning functional programming in July 2016. Actually I wanted to learn Elm because it looked to me like a powerful web development language (and I’d head some good things about Elm from very experiences Haskell programmers. One word that drew my attention was : the code written in Elm is very elegant.

I “practice” Elm only when I am at home, in my free time (which basically means, when the 3 young kids are asleep, at night).

I soon discovered the Functional Programming world, which was not really my first intention. But I thought, well, let’s go for it, let’s be “mainstream” J

That’s where things got a little more complicated than I had expected. As many will say, it is a change of paradigm, for people like me, who did some PHP, SQL, ASP 3.0 or a little “copy/paste JavaScript” before.

When this change of paradigm was done in my head, things became really smooth and fun.  But then I decided to dive into recursion (I had not really notice that a List.map IS actually a recursion).

Things again became complicated. I was told to read some books to grasp the “recursion” thing, but it was hard. I started reading “The little schemer”, I understood this very simple and fun book, so I thought it was ok.

Then I decided to actually write a recursive function of my own, and I fell lost. I didn’t understand what I was not doing that was in the book. I decided to stop looking at the book, and to try, and try and try, again and again, with very different exercises that I invented. It was hard, the solution was never easy to find, sometimes I felt like I was an expert in recursion, and sometimes I felt like I would never be able to do it. I thought it was a good thing that I was not paid to program.

Guess what? Yesterday I began reading Evan Czaplicki’s book Functional Programming in Elm. And he describes, not exactly in the same way, the pain that he had when he was a student, learning and practicing recursion. Wow, how good it felt to hear that.

Especially when he compares it to leaning the For loop, and it reminded me that, yes, at the beginning it was hard for me to grasp the For loop. But I had forgotten that I had to learn it long ago. Now it is feels so obvious.

In this book, Evan also gives some advice about how to understand recursion when you try to apply it. It looks like some “dumb” advice, coming from an expert in this field. But guess what? That was the same tricks that I had found myself. Which means that what we think is just something that we use because we are not experts, is also used by experts! How good it is to hear that also!

One last thing. Evan says that we the existing functions of Elm, we don’t really have, most of the time, to invent our own recursive functions in Elm. Was I wasting my time trying to learn that? Trying to re-do the functions of the Elm List library? Was I the only one on earth trying to do that?

Guess what… Later in the book, Evan writes that it is a very good thing to learn recursion, and the best way to do it is, for instance, trying to write the recursive functions from the Elm List library.

What I learned :

  • Don’t be afraid of what you try.
  • Prefer practicing again and again than reading books.
  • Consider you learned something not when you understood what was written in a book, but when you can actually do it.
  • Invent your own exercices.
  • Try to dig deeper than the surface.
  • Be your own master.

Software complexity cannot be avoided

Introduction : various thoughts on development, programming, creating.

Building a software is creating automated reactions to predefined user behaviours. Users have unpredictable and unexpected behaviours. So we must no think projects, but products: a product is never be finished, we have to make it live by adapting or creating new automated responses to new predefined user actions.

A software is like an helium balloon: it can be (re)shaped, it can explode with a single needle, and if you let it go, you loose control!

Design thinking  must anticipate testing but not get ahead with it. The risk is that we get lost  before event testing.

Pressured by time we rush to writing code syntax without preliminary design thinking. Confused thoughts will not result in effective coding.

How do we automate tasks that will have to respond to various behaviours, users, inputs…? That is the inherent uncertainty of programming.

Making errors is an absolute right. Testing is a prerequisite. Correcting is an absolute duty.

Refactoring is a good practice. It should not require the agreement of the managers.

Extensive, thorough  and exhaustive testing and adjustments can save a project that had a bad start.

If you’re tired of making wasteful estimates for your boss, wait untill he gets bored and become the boss; it worked for him.


 

Softwares are part of a vast eco-system from which the different parts are all created and maintained by humans: computers, programming languages, networks, protocols… Each of these is a sort of small universe that is constantly modified, improved, and extended by humans.

Their rules, limits and constraints are mostly a result of human rules, limits and constraints. This is one of the reasons of the complexity of software programming.

The programmer’s job is to gather a considerable knowledge and then constantly increase and update it, so that it can support his thought on how to find solutions to some problems that he discovers when the different tasks of a project are assigned to him.

This means that when he receives those tasks, the real complexity of his job begins: he must understand the task to do, find a way to do this task, and implement the solution.

I am of course talking of the complex tasks, the ones that he has never done before and that represent a sort of “mathematical” problem to solve. And these tasks are frequent in every software project.

We understand here that even when the team has sliced the project in many small tasks, the complexity still exists and has not been truly explored: a task is mainly seen by the project team as an output. It consists in the mental visualization of the result of an action, and not in the description of how the action should be done. If it was the case, the programmer would have nothing to do.

So a task is not the how-to. And the programmer’s job is not only to determine the how-to, but first to understand the what: what does this task mean in terms of programmatic problem. How to find a programmatic solution to that problem, and how to implement it (which is the easiest part).

The whole programmer’s life consists in finding solutions, but the tasks that can have a true impact on the schedule are the ones for which the programmer really has to work hard to find a solution. The degree of complexity does not depend on the size of the task (the “size” in terms of, for example, the number of data to be processed). A very “short” task on a sprint board could require a hard-to-discover algorithm to be properly addressed.

To find the solution the programmer uses different tools in the same time:

  •          his technical knowledge, which can never be exhaustive
  •          the specific research that he can make to improve his knowledge on this specific subject
  •          the discussions that he can have with pairs, that could have hints or even have had to solve a similar problem before
  •          his personal reflexion, intuition, intelligence, judgement…

From all this, it can result that once he has found and applied a suitable solution, the programmer has in the same time improved his knowledge, and another solution, more optimized,  pops in to his mind.

I have already seen this situation were the developer indicates the task as Done, and shortly afterwards asks for a new time-box to implement a better solution. In most cases, it is refused by the Project manager, because of schedule constraints. And of course it has resulted in generating frustration for the developer.

Refusing to implement this new solution is, in my opinion, opposed to the continuous improvement Agile principle: continuous improvement is not continuous learning. It means improving the product, continuously. So finding a better solution and not applying it is not continuous improvement.

We now see that the estimation of the length of the programming part of a software project cannot be accurate, even when the Team has divided the project into several small tasks. Because this linear slicing does not prefigure the non-linear programming activity that will be necessary for the realization of the tasks.

The tasks are mainly a list of problems for the programmer, and when the tasks are defined, the programming science begins.

So, even the best agile practitioners will never be able to reduce the programming complexity of a project. That is why even in the best Agile world, software estimation will always stay a guess.

Dealing with software uncertainty

Software engineering should not be compared with any engineering process, because it leads to harmful shortcuts. There are many analogies that are drawn with the automobile industry or the building sector.

But software engineering is a very particular industry: it is the kingdom of the invisible. Users do not see the code behind the software and the developers cannot always visualize clearly the interactions between various elements:

  • the different components of the application
  • all the configuration settings on the server
  • the many ways the people will try to use the software
  • the malicious tactics of hackers
  • the caches

There is never a clear picture. It seems invisible, very obscure, and the potentiality of problems seem infinite… it looks like Space!

If the developers cannot have this clear picture, it is even more obscure for all the other stakeholders: project manager, business analyst, customer…

And there is this dichotomy between the product that you can see – web pages or screens – and the reality of the construction behind, which is the code, the databases, the server…The user sees only the visible side of the software. It is as when looking at someone very tanned and smiling: you can never know but she perhaps has a serious illness, that even she could not be aware of.

We must also add the initial complexity of not knowing exactly how one will build the product, because each software project is unique, and is in nature a prototype. The product that the programmer is building is a research, a test, and will be made of plenty of things that have to be invented by this programmer. By invented, I mean something that the programmer himself has never done before.

And the context of the product will never be similar for two products, so we really are talking about inventions: softwares are like human beings, you can never find two humans that are identical.

You cannot observe the complexity of the software you are building: you can only explore it, like an astronomer explores the Universe. Of course they draw maps or the different constellations, calculate distances, but every day they discover new planets, new phenomenons in the Universe.

I dare say that software complexity is quite identical to that. If not, how can we explain the gigantic disasters of some software projects?

Being aware of this, our experience should have taught us to be humble when involved in a new project, and to accept that we will build a prototype, perhaps a beta, that will eventually become a stable product.

In fact our reaction is often the contrary of humility: in reaction to our fear of uncertainty and darkness, we become arrogant. So we try to build very precise and rigid roadmaps, schedules, features lists. By doing this, we think we can constraint the developer, force him to make what we want him to do. It is like writing operating instructions of a new model of washing machine before the machine is built.

We should think differently: if we agree that the developer will first build a prototype, we should let him this liberty: the freedom to try the best solution to build this prototype.

Software programming is close to an art, but it is different: you have to create a melody, but you also have to guess the notes.

 

My little plea for #NoEstimates (Liberty, Democracy and #NoEstimates)

This letter was written by a developer and was addressed to all software project managers. This is purely fictional and any resemblance to actual individuals or events is not coincidental. Please note that this (fictional) developer is French. Here is the text of the letter, as I have deciphered it.

Liberty, Democracy and #NoEstimates

Dear Project Managers,

Have you already heard about Democracy? I think so, but strangely, this does not seem to apply in the workplace. In democracy, everyone has the right to give his opinion. Decisions are taken by a majority of the voters. When you, Project Managers, come and ask the developers in your team for an estimate and they answer that they cannot estimate this stuff, shouldn’t you listen to them?

Do you know how we vote in a democracy? Yes you know that: in a referendum, you have three different possible votes: Yes, No or… no vote. It is the same thing for polls. You have 3 possible answers to a question: Yes, No or I don’t know.

In the workplace, it doesn’t seem to be the same. If a Project Manager asks a developer if she can estimate a task and the developer answers “I don’t know”, this answer is rarely taken into consideration or accepted. If the developer answers “No I cannot estimate this, I do not have enough information, or enough knowledge about this particular thing” it is even worse: it is as if he refuses to collaborate. When you do not really have the right to answer “No” or “I don’t know”, is that democracy in the workplace? When you have to answer “yes” and you don’t really know how to provide an accurate estimate, isn’t it some… abuse of power?

When you want to hire a developer, do you tell her during the interview what are the real rules. You know the rules that will be imposed to the developer… don’t you? Yes you know because you make the rules. Let me just remind those rules that were established:

  1. If I – Project Manager – ask you – developer- to do something that you cannot do (like estimating a task that you tell me you cannot estimate), you must do it. I don’t know how you will make it, but it does not matter to me.
  2. If I ask you to do something that you are not sure you can do, you must also do it. So if you tell me that you are not sure about what you are estimating, and how to estimate, I don’t really care. That’s your business, not my business.

I do not fully disagree with this second rule, but let me add that if I am asked to estimate a task that I am not sure I can estimate, I have the right to fail, that is to say I have the right to provide an estimate that will not be really accurate.

When you come to ask me for an estimate, and I tell you that I need more details to do so, why are you unable to give me those details? Why are you guys – Project Managers, Business Analysts, Customers Ambassadors, Products Owners or whatever you call them – paid for? Why do you ask me to complete a task, when you are not able to complete your task – gathering enough details about the requirement – that would help me do my own task? Why do I have the impression that it is not really a Democracy when you ask me to do something and you do not do you own work?

Also, when you ask me for an estimate that I am not really able to provide but that I provide anyway, why do I sometimes have the impression that the result was sort of suggested to me… when you tell me, we have to deliver this in two weeks, do you think we can do it… the question seems to imply the answer… Is that not a sort of abuse of power again? Particularly when I later begin to understand that if I don’t deliver the task in the estimated time, I will be judged responsible for that.

You see, I am asked to do a task (estimating) that will imply another task (doing the task that I estimated) and I will be held responsible for both tasks: the estimate, and the development of the estimated task. That is to say that I will take the full responsibility for two tasks that require very different qualities: event if I am a very good developer, I am perhaps not as good in estimating things. But if my estimate is not accurate, that means that I deliver later than expected. So it is as if I fail in the estimate – not accurate –  and in the programing – presumably delivered late, because not in line with the estimate.

It is the same if the programming takes longer than expected: I will be held responsible for the presumably bad estimate, and presumably late delivery of the program. And moreover, the whole project will be at risk, and I often feel that I will have to endorse the full responsibility for that.

Dear Project Managers, as you see, talking about estimates, for me developer, is more talking about Democracy at work. Or dictatorship.