Aren’t those almost always race condition bugs? The debugger slows execution, so the bug won’t appear when debugging.
Turned out that the bug ocurred randomly. The first tries I just had the “luck” that it only happened when the breakpoints were on.
Fixed it by now btw.
bug ocurred randomly.
Fixed it by now btw.
someone’s not sharing the actual root cause.
I’m new to Go and wanted to copy some text-data from a stream into the outputstream of the HTTP response.
I was copying the data to and from a []byte with a single Read() and Write() call and expexted everything to be copied as the buffer is always the size of the while data.
Turns out Read() sometimes fills the whole buffer and sometimes don’t.
Now I’m using io.Copy().
I had a bug like that today . A system showed 404, but about 50% of the time. Turns out I had two vhosts with the same name, and it hit them roughly evenly 😃
Had a similar thing at work not long ago.
A newly deployed version of a component in our system was only partially working, and the failures seemed to be random. It’s a distributed system, so the error could be in many places. After reading the logs for a while I realized that only some messages were coming through (via a message queue) to this component, which made no sense. The old version (on a different server) had been stopped, I had verified it myself days earlier.
Turns out that the server with the old version had been rebooted in the meantime, therefore the old component had started running again, and was listening to the same message queue! So it was fairly random which one actually received each message in the queue 😂
Problem solved by stopping the old container again and removing it completely so it wouldn’t start again at the next boot.
And that’s where Release with debug symbols comes in. Definitely harder to track down what’s going on when it skips 10 lines of code in one step though.
Usually my code ends up the other way though, because debug mode has extra assertions to catch things like uninitialized memory or access-after-free (I think specifically MSVC sets memory to 0xcdcdcdcd on free in debug mode).
Perfect, now you just have to wrap your program inside a debugger in production!
One of our customers does that. It happened multiple times already that one dev fixed an issue in production, and the next regular deployment overwrote everything.
But fortunately, it’s just critical infrastructure and nothing important.
This is where printf debugging really shines, ironically.
I once had a racing condition that got tipped over by the debugger. So similar behavior to what’s in the meme, but the code started working once I put in the print calls as well. I think I ended up just leaving the print calls, because I suck at async programming
Yeah, I was going to mention race conditions as soon as I saw the parent comment. Though I’d guess most cases where the debugger “fixes” the issue while print statements don’t are also race conditions, just the race isn’t tight enough that that extra IO time changes the result.
Best way to be thorough with concurrency testing IMO involves using synchronization to deliberately check the results of each potential race going either way. Of course, this is an exponential problem if you really want to be thorough (like some races could be based on thread 1 getting one specific instruction in between two specific instructions in thread 2, or maybe a race involves more than 2 threads, which would make it exponentially grow the exponential problem).
But a trick for print statement debugging race conditions is to keep your message short. Even better if you can just send a dword to some fast logger asynchronously (though be careful to not introduce more race conditions with this!).
This is one of the reasons why concurrency is hard even for those who understand it well.
Honestly, this is why I tell developers that work with/for me to build in logging, day one. Not only will you always have clarity in every environment, but you won’t run into cases where adding logging later makes races/deadlocks “go away mysteriously.” A lot of the time, attaching a debugger to stuff in production isn’t going to fly, so “printf debugging” like this is truly your best bet.
To do this right, look into logging modules/libraries that support filtering, lazy evaluation, contexts, and JSON output for perfect SEIM compatibility (enterprise stuff like Splunk or ELK).
Just run your prod env in debug mode! Problem solved.
Lol my workplace ships Angular in debug mode. Don’t worry though, the whole page kills itself if a dubious third-party library detects the console is open. Very secure and not brittle at all! Please send help
Blink-blink-blink. Blink. Blink. Blink. Blink-blink-blink.
No, I don’t have something in my eyes, I swear I’m fine looks nervously at boss.
You can imagine how many node projects there are running in production with npm run. I have encountered js/ts/node devs that don’t even know that you should like, build your project, with npm build and then ship and serve the bundle.
Sound like a critical race condition or bad memory access (this latter only in languages with pointers).
Since it’s HTTP(S) and judging by the average developer experience in the domain of multi-threading I’ve seen even for people doing stuff that naturally tends to involve multiple threads (such as networked access by multiple simultaneous clients), my bet is the former.
PS: Yeah, I know it’s a joke, but I made the serious point anyways because it might be useful for somebody.
