Ggthjy

You seem to be confusing two things:

• Them working any amount of hours to meet unexpected or unplanned issues.




• Them being responsible and providing quality work in a predictable way.

Ownership is not about the team working the whole night to fit your promises to customers. Ownership is about knowing what's in the code, how it works, having a plan and being able to tell you how and when things will be done. Ownership is developers making the right decisions so the code works correclty not just tonight, but in the years to come.

Sorry if this is a bit rough, but I've had too many managers tell me variations off your post. More often than not it boils to:

• lack of clear mandate


• changing requirements


• short term focus


• constant urgency

Would you please elaborate, in the question on what you, as a manager, did to prepare those releases, empower your team, and how you listened to their feedback? Then we can talk about ownership.

Even when I told him I needed it now, he said he had something else to do and sneaked off when I was not there.

Today, there is a critical bug and this senior guy said the same thing again - "I can't finish it today. I have a meeting with friends and I have to go." then he sneaked out while I was talking to my manager.

In both of these examples, you refer to him as sneaking off, but by your own words he told you that he wasn't going to do this work and then didn't do it. Sneaking off implies he's being deceptive or dishonest, but it sounds like he's being transparent, and you ought to recognize that. I've worked with people who say they'll handle things and then disappear, and those people deserve to be fired. Someone who informs you of their bandwidth and then follows through is different entirely. This person's integrity isn't an issue; he is only unemployable if his results aren't sufficient.

Last week, I told them in a team meeting that I expect higher level of ownership. If they promise something, they should do it. If there is a critical bug, they must fix it even if they have to stay late.

This is a reasonable statement and a level of ownership that senior engineers should generally accept with some caveats:

1. Critical bugs must actually be critical. For example, in my own career I have stayed late to fix "critical" bugs that were then not deployed into production for two months. In those cases, it was a manager freaking out about something and wanting it now instead of actually a critical bug. Of course, there have been actually critical issues as well.


2. Staffing levels must be generally sufficient. Meeting release dates and fixing issues are important, but if we are always late because we have 3 people doing 4+ people's work, that's a different situation.

Is there an alternative way to deal with issues like this?

Some development methodologies have built-in ways to manage these issues. In Agile development, for example, sprints are ways of promising what work will be delivered. It also includes built-in ways of measuring velocity (the amount of work being accomplished) and usually goes along with software (JIRA is the most popular one I believe) that makes whether or not a team or individuals are meeting those goals. In agile development, if you need to change course mid-sprint - like take time to fix a critical bug - it reflects that you're changing the scope inherently. Normally, you take things out in order to add whatever it is that must be added. This process makes it really easy to evaluate whether "I can't get to it today" is because he's working hard on other important goals or that he is just being difficult.

IMO, it's a fantastic method of software development that unfortunately is almost never done correctly.

UPDATE: in response to the question edits, this bug is absolutely critical in nature (at most companies it'd be called a showstopper instead of critical) and should be fixed immediately. I would follow the technique I described above - taking things off of his plate in exchange for him working on it now.

It sounds like this project has been a mess and very stressful for everyone involved, but a bug that makes it so 90% of users can't log in is worth staying late for. You need to assess whether or not this employee has completely checked out (in which case you have to help him move onto other employment) or if the project has just worn him down and he needs a break.

In my office we use to quote the following:

“Poor planning on your part does not necessitate an emergency on mine.”

In my experience developers often are motivated to help with a problem that appeared because of a mistake on their side or something unforeseen.

But all to often issues arise that are not only unsurprising but predicted.
Before you decide to give your developer an ultimatum and likely make him look for a new job, you should ask yourself the following:

• Have you done enough to avoid "critical" bugs in the first place? Did you give developers enough time to implement testing, code reviews, refactorings and monitoring?




• Are you making sure that new features get activated when there is enough time to fix them? (as opposed to late in the evening or on a Friday).




• If critical bugs are common, are you paying enough for overtime or on-call duty?




• Did the developers you want to have ownership, "own" the release process? Would they able to stop a feature release, if they think it was buggy?




• Are your deadlines realistic and agreed on with the dev team?

If all of the questions can be answered with a clear "yes", then you might have to let go of your senior developer.

If any of the answers is "No" or "I am not sure", then I would start looking for the problem in management and fix these problems first.

You claim lack of ownership by the team. Everything your developers build is owned by the company, not them. When you say that your employees should "own" the results of their work, does it also mean that they will receive the profits that those results make for the company? If it doesn't mean that, they don't truly own the work and you can not ask ownership from them.

If there is a critical bug, they must fix it even if they have to stay late.

Your solution to fixing critical problems by making your people stay late is convenient for the company and the employees pay the price. Again, that would be OK if they also get a share of the profits. Do they?

In this particular example, the bug prevents 90+% of users from logging in into the system. On average, this happens once a month this year while it happened twice last year.

When this happens so often and you don't install organizational procedures to reduce the impact of those errors, it is you as an organization that is at fault.

Actually, your current approach to fixing "critical" problems and your contemplation of firing your employee could be considered a sign for a dysfunctional organization. Your employee's behavior might be his way to react to that. Your update on the original question with a list of what you think you are doing right (as opposed to thinking what you might be doing wrong) also shows that you might have an issue accepting that you as a manager are a part of the problem.

There are a lot of things management can do to improve quality and reduce urgency before you ask employees to stay late:

1. No matter how well you think that you have focus on quality, the results show that you haven't. You have to seriously improve the quality of your development process, which could mean measures like reviews, inspections, pair programming, increased testing, redesign of critical components, improved architecture and design etc. You better start analyzing the organizational issues that cause those problems instead of writing down the list of measures you have already implemented. Obviously, they are not working.


2. Why does your employee have to stay late to fix the error? Can you do your releases in the early morning to give your developers the entire working day to fix issues?


3. Have you thought about using feature toggles or other measures to quickly revert to the previous version of the feature to give your team time to fix the problem?


4. You can not blame your employees for having plans for the evening when issues pop up on short notice. You can install a system of stand-by duty on days of critical releases. Then people know beforehand that they might have to stay late and can prepare accordingly.

Working in software this is very common.

You treat your people as professionals. You're talking ownership but then giving demands that a 'critical' bug must be fixed NOW.

Is the bug actually 'critical'?
Is it the result of unclear requirements?
Our old friend 'scope-creep'?

In each of these you (as the manager) need to manage expectations. Not every bug is 'critical'. Requirements can suck. Project scope changes.

Instead of demanding they drop everything for something 'critical' work with your teams to when it will be fixed. Then hold them to this estimate.

I've been putting 'critical' in quotes because after 30+ years in this field (yikes I'm old) this term is very misused. Everything can not be 'critical'.

With the updated question, it is now clear that you are trying to fix the wrong problem. The senior engineer's behavior is a symptom of a fundamentally broken software development process and/or dysfunctional company.

If you have critical bugs getting into production every month, then you have at least one of the following problems:

• Incompetent engineers


• Unmaintainable code base


• Inadequate testing

Given how much manpower you have at your disposal (20 engineers is a LOT of resources), it's likely a combination of all three.

My guess is that the senior engineer is fed up with the constant firefighting, and rightfully so.

You need to dig deeper and fix the underlying problems that are creating the need for people to continually work late. Convincing one engineer to work late more often is not going to help the big picture.

Now, what to do about it...

Step 1: Figure out why testing is not catching these critical bugs

The first thing you absolutely need to do is stop these critical bugs from ever reaching production. Every bug that reaches production is a failure in the testing process.

Go back over every critical bug that was discovered in production and determine exactly why it was not caught in testing. Add more automated test coverage, manual test coverage, or testing resources as necessary.

Step 2: Determine the root cause of every critical bug

For every critical bug, find out:

• Who created the bug


• When the bug was created


• Why the code was being modified


• Where the bug was introduced in the code

By doing this analysis, you will discover some patterns. Maybe there is one or two developers who keep introducing these bugs. Perhaps there is one code module that is very difficult to modify without causing problems. Or it's possible that the code as a whole is very difficult to with.

I want to make one additional point. Rushing out a bug fix often leads to technical debt. If your senior developer is questioning how it will be tested tonight then that is a good question that a senior developer should be asking! I’ve worked at places where urgency is prioritized over quality and this has had negative long term consequences. Ultimately, your team will have reduced capacity because it is always fighting fires.

It sounds like you have a huge testing problem. You ask why does everyone not drop all outside commitments to put out a fire but the real question is why are there fires starting every month?

Do you have any QA/Testing? Why did they not find that the first and most basic step (logging in) does not work. How did something that does not work at all get pushed to production.

Also why is your response to users not being able to log in to get everyone to stay late rushing "critical" fixes instead of having a system admin revert the update and the update can be attempted again later after the issues have been fixed.

"How are we going to test that tonight?" This is the correct response. When there is a critical issue and you are being pressured to fix it right now how will developers set aside time to properly review the changes are correct/high quality and how is QA meant to check that everything else is still working after the change. It sounds like you are also asking for these changes at the end of the day where everyone is tired and their thinking ability is at its lowest making it even more likely other issues will sneak in to this critical fix.

When are people most productive? When is the team most able to handle critical bugs?
There have been studies that answer said questions to when humans are best able to handle certain tasks.

You have a critical bug, and you want, a) Sr. to switch mental gears, b) Pick up a new "critical" task, c) work "till whenever" to fix it. And you expect this critical patch to work? Honestly, what do you expect for the product, the team, the team members if your wants were satisified?

Let go of your ego, and your irrational beliefs.

The term you are looking for is Discretionary Effort not Ownership.

I am assuming that your employees are meeting their contractual obligations (otherwise your course of action is clear).

You have no right to expect discretionary effort. That is what it is by definition. Fundamentally this is not something that you can speak to them about and expect a change. You are likely to get the opposite response. They are under no obligation to give it. Threats about firing them are likely to have such an overwhelming poor response, as well as being illegal.

I don't have any good suggestions on how you can improve things. The very fact that you can rely on Discretionary Effort by some of your people suggests to me the culture is not necessarily broken.

Fixing this will take time, so instead, I can offer stop-gap measures:

Fix the bus-factor of 1

Why can only a single employee resolve this issue?

Have an on-call roster

According to reimbursement agreed upon with individual employees, not what you think it is worth.

Roll out updates at better times

It may not be possible, but rolling things out at better times can increase the chance for someone to assist.

The worth of your software is a function of how well it is supported, so you shouldn't use Discretionary Effort as a crutch. If you want your software to be supported to a level, you need to ensure you have things in place to ensure it.

So, you expect your employees to give up their social and/or family lives at the drop of a hat in order to fix problems?

Are they really all that critical?

Managers always seem to think that everything is critical because saying no is hard. This is a strong potential reason why your lead dev is pushing back. They are trying to protect their boundaries because you won't. And they are trying to protect their team's boundaries because you won't.

If they truly are all that critical, then what is going wrong that allows these issues to happen?

If your product quality is that bad, then you need to move over and let your developers devise a plan to get the product back on track. Poor quality isn't just about bugs. Poor quality derails predictability. If you are consistently going off plan because your quality is this bad, then fix your quality. And you don't fix it by asking developers to do it in their personal time. If that is the expectation you set, then you are telling your developers the business does not care about quality and therefore does not value predictability. If you do not value predictability, then stop complaining.

If they truly are all critical, then why don't you plan an on-call rotation?

Not only does this protect employees' personal time and protect the business's needs, it also creates incentive for developers to fix the systemic problems that are causing them to fire fight so much. (maybe you need more or better tests, maybe you have broken legacy code, etc.)

Why don't you stay late and fix things?

You're complaining that somebody doesn't step up to work through the night to fix a problem. Why don't you work through the night to fix it? I think you'll find the same conclusions as your team lead.

Your behavior

You have threatened to fire your employees for not doing something which you yourself refuse to do. You are complaining this happens a lot, yet you have not planned for it with an on-call rotation or by repaying technical debt.

Reading your list of steps to plan a release, what stands out to me is the frequent use of "I told them to..." and the granularity of planning all the way down to function names. You plan out minor details that are easily changeable, but won't plan a support process for your product.

This is 100% your problem.

Your team

It sounds to me like you have a bunch of smart, honest, professionals who know how to make good software, but their manager likes to dictate to them how to do their job and when the manager's approach causes a problem, force them to work more hours.

Have you stepped back and asked your team how to get less critical bugs? Have you asked your team how they think they should handle responsibility for unexpected critical issues?

Your team lead is right to push back on your expectations. And I'm glad to hear that he is encouraging his team to say no to things. He is trying to protect the team because you aren't.

In my time as a team lead, I can tell you that one of the hardest but most important lessons is learning how to say no. Maybe you can learn something from this employee of yours.

You can't force someone to do overtime (depends on country and potentially extreme circumstances as exeption in laws)

If the person can't or won't do it, it is your responsibility to find a willing employee or to hire external help if the task is vital and needs immediate attention.

Ask an employment lawyer in your jurisdiction to clarify.

As for ownership and following through with assigned or promised tasks, you have your disciplinary arsenal all the way up to ending employment contracts.

Also, what Sefe said...

Answering the updated question:

Your big problem is not a lack of ownership. This rather seems to be a sympton of deeper, underlying problem: The fact that your Development Process seems to be substantially broken.

In theory your process (i.e. automated tests & test coverage, planning in sprints, no sudden requirement changes) should prevent most of the issues you see.

By your own statement, you ran into multiple "Showstopper" issues with the program even when deployed to the customer and some of them even being regressions. The sprints are not finished on time (tests are part of the sprint). And even when written the test do not provide sufficient coverage to catch those bugs. You also said you already had a "stay late" situation (for which you gave them days off afterwards).

You need to discover what is going wrong. Only by finding and resolving the underlying issue you can hope to fix things.

If 90% of users cannot log in, and users are not able to make purchases ( i.e. sales are being lost ) you need to revert the update to the previous working version immediately. Waiting for your developers to troubleshoot and fix the bug can take much longer and cause more of a negative user impact than simply reverting to a previous version.

More importantly, your developers are less likely to want to continue working for you if they are forced to perform overtime work when there is a better solution available. If you value your employees you should respect their time outside of work.

Answering the updated question,

It appears good working practice has been followed:

1. sprints


2. feature lock down close to deployment


3. time estimation from developers


4. automated tests

I partially agree there is a positive culture of fixing critical bugs in overtime.
However the culture also needs to reflect no code is perfect, unless you spend a lot of money on it, there's an age old storey saying that NASA spent around $1,000 per line of code!. I won't comment to much on the cultural side, as that has already been covered by others, instead some methodology suggestions:

A team structure in fashion at the moment is feature Squads who own end-to-end delivery and operational responsibility for isolated vertical they write. If it goes wrong, they are the ones woken in the night, however I'd be wary in introducing this in a legacy environment as it will heed distaste if a team didn't have the opportunity to introduce the quality they desire from the get-go. And traditional Ops roles with contracts/pay to suit are likely more open to on-call way of working.

A better idea would be to introduce the idea of QA champions within a team and the "3 amigos" approach i.e. product, developer and QA member all write the Feature Spec (Behaviour Driven Design) together. This ensures that "how a QA member would break it" is accounted for from the beginning and it should be in just enough detail i.e. a specification by example manner. The QA member doesn't need to be the person writing the automation, but they should code review it. As people have mentioned above the writer of the code shouldn't be solely responsible for accrediting it's quality and introducing a 3rd party as early in the process as possible is a positive move.

Perhaps also the Production environment and release management needs enhancing. "Blue/Green deployments" and "Testing in Production" are common practices for gradually rolling out changes to a wider and wider audience only as metrics prove themselves. Ideally your staging environment should catch critical bugs, but there's always something different about production, therefore it should never be a big-bang release.

Judging from the timeframe although you are using a typical release cadence, you may wish to consider releasing more often. More frequent smaller releases can lead to less risk if coupled with good test and release automation. This can be paired with feature switches so that features can be switched on in full when the composing user stories are complete.

I read your update. Your problem is that you shipped a broken product. That’s it. That’s what you need to work on - don’t ship broken products.

Your complaint is ridiculous. YOU decided to ship a product when it was broken. That’s where the buck stops, with you.

You then made two mistakes: First, it seems that the timing of your release was such that you were told about it at 4pm. Release half an hour before developers arrive at work. Very simple solution, it’s your job to know this. Second, there seems to be no motivation for the employee to work overtime. Paying for overtime usually works quite well. An environment where you allow broken code to ship regularly doesn’t.

In light of the updated question, I suggest that while you may have an employee attitude problem on your hands, what you really have is a software quality problem:

1. There is no quality assurance process independent of the developers who are writing these features. Developers are notoriously bad at testing their own code, largely because they naturally start with the assumption that anything they wrote does and should work the way they intended it to, as opposed to how the end user expects it to.




2. Whatever testing environment you have isn’t adequate for reproducing problems identified by your customers. You identified bugs that prevented login at several stages in the process, but the final check did not identify this problem.




3. The senior developer is willing to ignore critical issues that actually are critical. “Dude, my friends and I were going to do something cool tonight” is not the kind of reaction to “nobody can log in therefore our business is losing 100% of this products’ possible revenue that we use to pay your salary” you want to see. If this is how he reacts to an emergency, how do you think he deals with minor stuff? With more attention to detail or less?

There is a specific part of your updated story that raises many more questions for me and I think you need to investigate in detail: At one point in the process, the team said they wrote all of the tests and there were no issues. Then you say later, it was discovered that not all of the tests were written after all, that the login bug was not identified, and that you devoted additional time to writing those tests and fixing the login issue, which later was demonstrated in production to be not fixed.

1. When you said the tests were not written after all, why were they not written? Was it that the case in question was an oversight, or did the team misrepresent the completion of their work? The former is reasonable to expect will happen occasionally, the latter is unacceptable behavior that you need to make clear will not be tolerated in the future (unless it was a misunderstanding on your part).




2. Once the tests were written, how do you know that these tests are correct? Did you review the tests? Clearly they don’t work; they all passed but the problem that was explicitly tested for and said to be fixed still made it to production. It is not unknown for unscrupulous people to write tests such that they always pass, especially if there is deadline pressure. If you are not capable of reviewing the tests yourself, you should find someone technical on another team to review the tests for you.

There’s also something else you wrote, that the number of critical bugs is now up to once a month, but before it was more like once or twice a year. Why is that? Have you done any investigation into why that is now happening? Did the product or team change significantly in that time period? This sounds like quality is slipping.

Here’s what I think you should do:

1. Hire a good, experienced QA tester and subject all of this team’s work to independent QA testing (you should do this throughout your company, but this team in particular needs it because quality is slipping).




2. Review the test environment and compare with production to ensure that the test environment still reflects production.




3. Review tests and code your team is writing for quality and correctness on a regular basis and in more detail than you are currently doing.

You might have an attitude problem on your team with the senior guy, but you definitely have a quality problem. Attitude problems are hard to fix, but quality problems are easier to fix, and have the added bonus of rendering the attitude of one guy on the team irrelevant if you do it right.

Being a senior team member myself, and having to deal with such situations time and again, I can only tell that a guy whom you can always rely on, needs some motivation as well. The guy is surely honest and has his right to his personal life after office but he will stay and fix bugs if there's a better motivation for him, or if you manage to create a friendly environment for him.

Think of it the other way around: what if the guy gets sick or leaves the job or has some emergency and he has to leave the office for a time. Currently his time with his friends seems to be a problem for you but he can have all the legitimate needs to leave the office even during work hours (and spending tim with friends after office is a legitimate need as well).

A very common managerial attitude is to always through it on the guy who can do it. Now this varies person to person but being an employee, this can lead to frustration and such situation emerges when employee starts saying no to you.

So in my opinion you need to do two things:

1:) You should create a platform of learning for junior staff, and gradually delegate things so that in time of critical need you don't have to look at the same guy each time.

2:) As long as within office hours the guy is doing what he is supposed to do and meeting his KPIs, he is fine and needs to be dealt with in a friendly manner.

It sounds to me that he simply places a higher value on being away from the office than in it.

There are many reasons why, and a good manager would not simply assume the worst. Perhaps he's having personal issues at home, be it a relationship, sickness, whatever other stresses.

Perhaps he is growing dissatisfied with the job and simply doesn't care anymore.

There are many reasons why he may be acting as he does. I'd suggest seeing if there is something that you'd want to give extra grace for before you rip into him too much, or discipline him.